2024-03-19T00:09:29.506 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-19T00:09:29.506 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} --archive /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250 --name pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250/orig.config.yaml 2024-03-19T00:09:29.562 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T00:09:29.647 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250 branch: wip-batrick-testing-20240318.181317 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} email: pdonnell@redhat.com first_in_suite: false job_id: '7609250' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240318.181317 ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - PG_AVAILABILITY - PG_DEGRADED - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt install: ceph: flavor: default sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs mntopts: - ms_mode=secure - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240318.181317 sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3005 sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 sleep_before_teardown: 0 subset: 433/512 suite: fs suite_branch: wip-batrick-testing-20240318.181317 suite_path: /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: e48b1f6c0608b2615e1890cc46286c3877d6e910 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAd63bO977/b/mawCBs3CO6F2T2yWIBGBSz1MOF1jQiaRieZ1ZasJBXxWv49Hi0BC0cfbdv6zD/LV0Y0MCslZww= smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPU31mKDEia961VDaU8NKccsLQDlbNPuZu9H1aQo9Dr3JfCnKtUN+xBFuAvcLRcPWb4uUi+cjrNh/KoHUMbPtMU= smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPjH2AFSIXZ2/ePqN38IWe8cxvPIf8Oz0/QA2tYVE5PDQiHKBamC85lViL2mrgcuXupFD2IlfNjE3UiheQ/HMRE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 32 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_21:05:49 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-19T00:09:29.647 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa; will attempt to use it 2024-03-19T00:09:29.648 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_e48b1f6c0608b2615e1890cc46286c3877d6e910/qa/tasks 2024-03-19T00:09:29.648 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-19T00:09:29.649 INFO:teuthology.task.internal:Checking packages... 2024-03-19T00:09:29.673 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b6ae24918b03b1b8a19c9263a50655bba80bb810' 2024-03-19T00:09:29.674 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-19T00:09:29.674 INFO:teuthology.packaging:ref: None 2024-03-19T00:09:29.674 INFO:teuthology.packaging:tag: None 2024-03-19T00:09:29.674 INFO:teuthology.packaging:branch: wip-batrick-testing-20240318.181317 2024-03-19T00:09:29.674 INFO:teuthology.packaging:sha1: b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T00:09:29.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240318.181317 2024-03-19T00:09:29.898 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2206.gb6ae2491 2024-03-19T00:09:29.900 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-19T00:09:29.906 INFO:teuthology.task.internal:no buildpackages task found 2024-03-19T00:09:29.907 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-19T00:09:29.949 INFO:teuthology.task.internal:Saving configuration 2024-03-19T00:09:29.969 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-19T00:09:29.975 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T00:09:29.999 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 00:04:07.975846', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAd63bO977/b/mawCBs3CO6F2T2yWIBGBSz1MOF1jQiaRieZ1ZasJBXxWv49Hi0BC0cfbdv6zD/LV0Y0MCslZww='} 2024-03-19T00:09:30.021 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 00:04:07.974081', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPU31mKDEia961VDaU8NKccsLQDlbNPuZu9H1aQo9Dr3JfCnKtUN+xBFuAvcLRcPWb4uUi+cjrNh/KoHUMbPtMU='} 2024-03-19T00:09:30.041 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 00:04:07.977102', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPjH2AFSIXZ2/ePqN38IWe8cxvPIf8Oz0/QA2tYVE5PDQiHKBamC85lViL2mrgcuXupFD2IlfNjE3UiheQ/HMRE='} 2024-03-19T00:09:30.041 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-19T00:09:30.047 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-19T00:09:30.048 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-19T00:09:30.048 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-19T00:09:30.048 INFO:teuthology.run_tasks:Running task console_log... 2024-03-19T00:09:30.231 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2088b57160>, signals=[15]) 2024-03-19T00:09:30.231 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-19T00:09:30.247 INFO:teuthology.task.internal:Opening connections... 2024-03-19T00:09:30.247 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T00:09:30.249 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T00:09:30.338 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2024-03-19T00:09:30.339 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T00:09:30.427 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-03-19T00:09:30.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T00:09:30.517 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-19T00:09:30.523 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-03-19T00:09:30.541 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-03-19T00:09:30.542 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-03-19T00:09:30.599 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-03-19T00:09:30.599 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-03-19T00:09:30.599 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-03-19T00:09:30.599 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-03-19T00:09:30.599 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-03-19T00:09:30.599 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-03-19T00:09:30.599 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T00:09:30.599 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-03-19T00:09:30.599 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-03-19T00:09:30.599 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T00:09:30.599 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-03-19T00:09:30.599 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T00:09:30.600 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T00:09:30.600 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T00:09:30.600 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-03-19T00:09:30.625 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2024-03-19T00:09:30.642 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2024-03-19T00:09:30.643 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2024-03-19T00:09:30.700 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2024-03-19T00:09:30.700 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="9" 2024-03-19T00:09:30.700 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2024-03-19T00:09:30.700 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2024-03-19T00:09:30.700 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="9" 2024-03-19T00:09:30.700 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el9" 2024-03-19T00:09:30.700 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T00:09:30.700 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2024-03-19T00:09:30.700 INFO:teuthology.orchestra.run.smithi156.stdout:LOGO="fedora-logo-icon" 2024-03-19T00:09:30.700 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T00:09:30.700 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2024-03-19T00:09:30.701 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T00:09:30.701 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T00:09:30.701 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T00:09:30.701 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2024-03-19T00:09:30.725 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-03-19T00:09:30.742 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-03-19T00:09:30.742 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-03-19T00:09:30.799 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-03-19T00:09:30.799 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2024-03-19T00:09:30.799 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-03-19T00:09:30.799 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-03-19T00:09:30.799 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2024-03-19T00:09:30.799 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2024-03-19T00:09:30.799 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-19T00:09:30.799 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-03-19T00:09:30.800 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2024-03-19T00:09:30.800 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-19T00:09:30.800 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-03-19T00:09:30.800 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-19T00:09:30.800 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-19T00:09:30.800 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-19T00:09:30.800 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-03-19T00:09:30.825 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-19T00:09:30.836 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-19T00:09:30.844 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-19T00:09:30.844 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-03-19T00:09:30.847 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2024-03-19T00:09:30.849 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-03-19T00:09:30.864 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-19T00:09:30.906 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-19T00:09:30.906 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-03-19T00:09:30.909 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2024-03-19T00:09:30.911 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-03-19T00:09:31.230 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-19T00:09:31.304 INFO:teuthology.run_tasks:Running task kernel... 2024-03-19T00:09:31.318 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-03-19T00:09:31.319 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-19T00:09:31.319 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-19T00:09:31.319 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T00:09:31.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T00:09:31.335 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-19T00:09:31.393 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-361.el9.x86_64 2024-03-19T00:09:31.393 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-361.el9.x86_64 2024-03-19T00:09:31.393 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2024-03-19T00:09:33.788 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-19T00:09:35.547 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2024-03-19T00:09:40.395 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-03-19T00:22:46.670 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 25 kB/s | 19 MB 13:05 2024-03-19T00:22:51.129 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 6.3 MB/s | 6.1 MB 00:00 2024-03-19T00:22:52.605 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-03-19T00:23:49.452 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 373 kB/s | 21 MB 00:56 2024-03-19T00:23:54.425 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-19T00:23:56.539 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-19T00:23:56.540 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-19T00:23:56.629 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-19T00:23:56.641 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-19T00:23:56.641 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-03-19T00:23:56.641 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-19T00:23:56.641 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-19T00:23:56.641 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-19T00:23:56.641 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-19T00:23:56.641 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-19T00:23:56.641 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-19T00:23:56.641 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-19T00:23:56.641 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:23:56.641 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-19T00:23:56.642 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-03-19T00:23:56.642 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-03-19T00:23:56.642 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:23:56.642 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 76 M 2024-03-19T00:23:56.642 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 123 M 2024-03-19T00:23:56.642 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-19T00:23:56.892 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 353 kB/s | 44 kB 00:00 2024-03-19T00:24:00.334 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 4.2 MB/s | 15 MB 00:03 2024-03-19T00:24:02.259 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 5.1 MB/s | 27 MB 00:05 2024-03-19T00:24:02.967 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 5.4 MB/s | 33 MB 00:06 2024-03-19T00:24:02.968 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-19T00:24:02.968 INFO:teuthology.orchestra.run.smithi116.stdout:Total 12 MB/s | 76 MB 00:06 2024-03-19T00:24:03.631 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-19T00:24:03.992 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-19T00:24:03.992 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-19T00:24:04.988 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-19T00:24:04.989 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-19T00:24:07.477 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-19T00:24:08.175 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-19T00:24:08.253 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T00:24:11.014 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T00:24:11.146 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T00:24:15.871 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T00:24:15.912 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T00:24:20.653 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T00:24:49.645 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T00:24:49.646 INFO:teuthology.orchestra.run.smithi116.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-19T00:24:49.646 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:24:49.689 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-19T00:24:50.075 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-19T00:24:50.075 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-19T00:24:50.075 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-19T00:24:50.075 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-19T00:24:50.455 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-19T00:24:50.455 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:24:50.455 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-19T00:24:50.455 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-19T00:24:50.455 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-19T00:24:50.455 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-19T00:24:50.455 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-19T00:24:50.455 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:24:50.455 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-19T00:24:50.788 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T00:24:50.887 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T00:24:50.888 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T00:24:50.888 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-19T00:24:50.888 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-19T00:24:50.888 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-19T00:24:50.888 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-19T00:24:50.888 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-03-19T00:24:50.889 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-03-19T00:24:50.968 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-19T00:24:50.969 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-19T00:24:50.969 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2024-03-19T00:24:51.002 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2024-03-19T00:24:51.002 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2024-03-19T00:24:51.002 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2024-03-19T00:24:51.003 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2024-03-19T00:24:51.004 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:24:51.004 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-19T00:24:51.004 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-19T00:24:51.004 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-19T00:24:51.004 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2024-03-19T00:24:51.004 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:24:51.004 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2024-03-19T00:24:51.005 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-19T00:24:51.005 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 5.14.0-430.el9.x86_64 2024-03-19T00:24:51.005 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-19T00:24:51.607 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-03-19T00:24:52.987 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-19T00:24:53.024 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-03-19T00:24:53.026 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-03-19T00:24:53.063 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.n70U0a3izY 2024-03-19T00:24:53.063 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.n70U0a3izY 2024-03-19T00:24:53.137 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.n70U0a3izY 2024-03-19T00:24:53.277 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.n70U0a3izY is 6KB 2024-03-19T00:24:53.325 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.n70U0a3izY 2024-03-19T00:24:53.339 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2024-03-19T00:24:53.404 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-19T00:24:53.404 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-19T00:24:53.404 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-19T00:24:53.404 INFO:teuthology.orchestra.run.smithi116.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-19T00:24:53.406 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-19T00:24:53.568 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-03-19T00:25:23.571 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-19T00:25:23.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T00:25:23.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T00:25:41.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-19T00:25:50.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T00:25:50.921 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T00:25:53.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-19T00:26:05.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T00:26:05.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T00:26:24.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-03-19T00:26:39.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T00:26:39.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T00:26:39.738 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-19T00:26:40.141 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T00:26:40.141 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-430.el9.x86_64"... 2024-03-19T00:26:40.141 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-03-19T00:26:40.155 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-430.el9.x86_64 2024-03-19T00:26:40.155 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-19T00:26:40.155 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-19T00:26:40.155 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-19T00:26:41.156 DEBUG:teuthology.parallel:result is None 2024-03-19T00:26:41.157 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-19T00:26:41.165 INFO:teuthology.task.internal:Creating test directory... 2024-03-19T00:26:41.165 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T00:26:41.167 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T00:26:41.170 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-19T00:26:41.189 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-19T00:26:41.196 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-19T00:26:41.205 INFO:teuthology.task.internal:Creating archive directory... 2024-03-19T00:26:41.206 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T00:26:41.221 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T00:26:41.228 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-19T00:26:41.353 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-19T00:26:41.360 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-19T00:26:41.360 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T00:26:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T00:26:41.378 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-19T00:26:41.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T00:26:41.395 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T00:26:41.397 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T00:26:41.421 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-19T00:26:41.455 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T00:26:41.470 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T00:26:41.474 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T00:26:41.474 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T00:26:41.500 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T00:26:41.528 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-19T00:26:41.529 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-19T00:26:41.536 INFO:teuthology.task.internal:Configuring sudo... 2024-03-19T00:26:41.536 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T00:26:41.538 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T00:26:41.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-19T00:26:41.570 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-19T00:26:41.580 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-19T00:26:41.580 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T00:26:41.606 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T00:26:41.612 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-19T00:26:41.627 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T00:26:41.699 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T00:26:41.768 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T00:26:41.811 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T00:26:41.853 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T00:26:41.853 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T00:26:41.920 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T00:26:41.966 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T00:26:42.085 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T00:26:42.127 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T00:26:42.173 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-19T00:26:42.173 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T00:26:42.242 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T00:26:42.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-19T00:26:42.344 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T00:26:42.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-19T00:26:42.473 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T00:26:42.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-19T00:26:42.543 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-03-19T00:26:42.545 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2024-03-19T00:26:42.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-03-19T00:26:42.583 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T00:26:42.612 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T00:26:42.621 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T00:26:43.001 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-19T00:26:43.011 INFO:teuthology.task.internal:Starting timer... 2024-03-19T00:26:43.011 INFO:teuthology.run_tasks:Running task pcp... 2024-03-19T00:26:43.055 INFO:teuthology.run_tasks:Running task selinux... 2024-03-19T00:26:43.064 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-19T00:26:43.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-03-19T00:26:43.149 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-03-19T00:26:43.152 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2024-03-19T00:26:43.250 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2024-03-19T00:26:43.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-03-19T00:26:43.319 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-03-19T00:26:43.321 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-19T00:26:43.321 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-03-19T00:26:43.344 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-03-19T00:26:43.344 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2024-03-19T00:26:43.377 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2024-03-19T00:26:43.377 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-03-19T00:26:43.403 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-03-19T00:26:43.403 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi156.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi162.front.sepia.ceph.com': 'permissive'} 2024-03-19T00:26:43.403 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T00:26:43.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T00:26:43.432 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T00:26:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T00:26:43.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T00:26:43.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T00:26:43.491 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-19T00:26:43.491 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-03-19T00:26:43.521 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2024-03-19T00:26:43.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-03-19T00:26:43.605 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-19T00:26:43.615 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-19T00:26:43.635 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-19T00:26:43.637 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi156.front.sepia.ceph.com,smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-19T00:50:52.536 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com'), Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2024-03-19T00:50:52.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T00:50:52.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T00:50:52.631 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-03-19T00:50:52.704 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-03-19T00:50:52.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-19T00:50:52.705 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T00:50:52.793 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-03-19T00:50:52.861 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-03-19T00:50:52.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-19T00:50:52.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-19T00:50:52.953 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-03-19T00:50:53.021 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-19T00:50:53.022 INFO:teuthology.run_tasks:Running task clock... 2024-03-19T00:50:53.033 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-19T00:50:53.034 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T00:50:53.034 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T00:50:53.037 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T00:50:53.037 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T00:50:53.039 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-19T00:50:53.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T00:50:53.066 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T00:50:53.068 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T00:50:53.082 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T00:50:53.086 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T00:50:53.093 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-19T00:50:53.111 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-03-19T00:50:53.111 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-19T00:50:53.116 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2024-03-19T00:50:53.124 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-03-19T00:50:53.129 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2024-03-19T00:50:53.141 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T00:50:53.142 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-03-19T00:50:53.146 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T00:50:53.154 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-03-19T00:50:53.157 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T00:50:53.162 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T00:50:53.171 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-19T00:50:53.188 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-19T00:50:53.206 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-03-19T00:50:53.210 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T00:50:53.210 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-03-19T00:50:53.210 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T00:50:53.210 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T00:50:53.210 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T00:50:53.210 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T00:50:53.211 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-19T00:50:53.213 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T00:50:53.213 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-19T00:50:53.213 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T00:50:53.213 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T00:50:53.213 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T00:50:53.213 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T00:50:53.232 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-03-19T00:50:53.235 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T00:50:53.235 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-03-19T00:50:53.235 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T00:50:53.235 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T00:50:53.235 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T00:50:53.236 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T00:50:53.236 INFO:teuthology.run_tasks:Running task install... 2024-03-19T00:50:53.246 DEBUG:teuthology.task.install:project ceph 2024-03-19T00:50:53.247 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'}} 2024-03-19T00:50:53.247 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810'} 2024-03-19T00:50:53.247 INFO:teuthology.task.install:Using flavor: default 2024-03-19T00:50:53.255 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-19T00:50:53.255 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-19T00:50:53.255 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T00:50:53.256 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T00:50:53.258 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T00:50:53.258 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T00:50:53.259 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'tag': None, 'wait_for_package': False} 2024-03-19T00:50:53.259 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T00:50:53.401 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T00:50:53.401 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T00:50:53.416 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T00:50:53.416 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T00:50:53.421 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/ 2024-03-19T00:50:53.422 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2206.gb6ae2491 2024-03-19T00:50:53.512 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T00:50:53.513 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T00:50:53.513 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T00:50:53.525 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T00:50:53.525 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-19T00:50:53.525 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T00:50:53.540 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T00:50:53.540 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T00:50:53.554 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T00:50:53.554 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T00:50:53.593 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-19T00:50:53.593 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T00:50:53.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-19T00:50:53.607 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T00:50:53.622 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-19T00:50:53.622 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b6ae24918b03b1b8a19c9263a50655bba80bb810/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-19T00:50:53.623 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T00:50:53.687 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T00:50:53.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-19T00:50:53.704 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T00:50:53.754 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-03-19T00:50:53.755 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-03-19T00:50:53.770 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-19T00:50:53.771 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2024-03-19T00:50:53.772 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2024-03-19T00:50:53.837 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-03-19T00:50:53.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-03-19T00:50:53.999 INFO:teuthology.orchestra.run.smithi116.stdout:74 files removed 2024-03-19T00:50:54.007 INFO:teuthology.orchestra.run.smithi156.stdout:74 files removed 2024-03-19T00:50:54.029 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T00:50:54.035 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T00:50:54.075 INFO:teuthology.orchestra.run.smithi162.stdout:74 files removed 2024-03-19T00:50:54.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-19T00:50:54.749 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 197 kB/s | 83 kB 00:00 2024-03-19T00:50:54.812 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 172 kB/s | 83 kB 00:00 2024-03-19T00:50:54.856 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 179 kB/s | 83 kB 00:00 2024-03-19T00:50:55.173 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 36 kB/s | 10 kB 00:00 2024-03-19T00:50:55.200 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-19T00:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-19T00:50:55.530 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 6.8 kB/s | 1.9 kB 00:00 2024-03-19T00:50:55.559 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-03-19T00:50:55.590 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-19T00:50:56.103 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-19T00:50:56.168 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-19T00:50:56.190 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-19T00:50:57.545 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-19T00:50:57.577 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-19T00:50:57.791 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-03-19T00:50:59.381 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.0 MB 00:01 2024-03-19T00:51:01.283 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.0 MB 00:03 2024-03-19T00:51:02.061 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-19T00:51:03.803 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-19T00:51:08.205 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 6.1 MB 00:00 2024-03-19T00:51:10.247 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 16 kB 00:00 2024-03-19T00:51:13.515 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 21 MB 00:03 2024-03-19T00:51:13.606 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:13 2024-03-19T00:51:17.857 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.1 MB 00:00 2024-03-19T00:51:18.442 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-03-19T00:51:18.756 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:16 2024-03-19T00:51:19.382 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-03-19T00:51:20.443 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-19T00:51:20.837 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T00:51:20.837 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T00:51:20.852 INFO:teuthology.orchestra.run.smithi116.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T00:51:20.853 INFO:teuthology.orchestra.run.smithi116.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T00:51:20.854 INFO:teuthology.orchestra.run.smithi116.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T00:51:20.855 INFO:teuthology.orchestra.run.smithi116.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T00:51:20.857 INFO:teuthology.orchestra.run.smithi116.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T00:51:20.860 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T00:51:20.861 INFO:teuthology.orchestra.run.smithi116.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T00:51:20.943 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-03-19T00:51:20.962 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-19T00:51:20.962 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-03-19T00:51:20.962 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-19T00:51:20.962 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-03-19T00:51:20.962 INFO:teuthology.orchestra.run.smithi116.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T00:51:20.962 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T00:51:20.962 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T00:51:20.963 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T00:51:20.963 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T00:51:20.963 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T00:51:20.963 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T00:51:20.963 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T00:51:20.963 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T00:51:20.963 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T00:51:20.963 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T00:51:20.963 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T00:51:20.963 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T00:51:20.963 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T00:51:20.963 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T00:51:20.963 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T00:51:20.964 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T00:51:20.964 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T00:51:20.964 INFO:teuthology.orchestra.run.smithi116.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T00:51:20.964 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T00:51:20.964 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T00:51:20.964 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T00:51:20.964 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T00:51:20.964 INFO:teuthology.orchestra.run.smithi116.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T00:51:20.964 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T00:51:20.964 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T00:51:20.964 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T00:51:20.964 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T00:51:20.965 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T00:51:20.965 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T00:51:20.965 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T00:51:20.965 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T00:51:20.965 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T00:51:20.965 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T00:51:20.965 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T00:51:20.965 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T00:51:20.965 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T00:51:20.965 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-03-19T00:51:20.965 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T00:51:20.965 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T00:51:20.965 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T00:51:20.966 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T00:51:20.966 INFO:teuthology.orchestra.run.smithi116.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T00:51:20.966 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T00:51:20.966 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T00:51:20.966 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T00:51:20.966 INFO:teuthology.orchestra.run.smithi116.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T00:51:20.966 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T00:51:20.966 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T00:51:20.966 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T00:51:20.966 INFO:teuthology.orchestra.run.smithi116.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T00:51:20.966 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T00:51:20.967 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T00:51:20.967 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T00:51:20.967 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T00:51:20.967 INFO:teuthology.orchestra.run.smithi116.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T00:51:20.967 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T00:51:20.967 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T00:51:20.967 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T00:51:20.967 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T00:51:20.967 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T00:51:20.967 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T00:51:20.967 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-03-19T00:51:20.967 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T00:51:20.967 INFO:teuthology.orchestra.run.smithi116.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T00:51:20.968 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T00:51:20.968 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T00:51:20.968 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T00:51:20.968 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T00:51:20.968 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T00:51:20.968 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T00:51:20.968 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T00:51:20.968 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T00:51:20.968 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T00:51:20.968 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T00:51:20.968 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T00:51:20.968 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T00:51:20.968 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T00:51:20.969 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T00:51:20.969 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T00:51:20.969 INFO:teuthology.orchestra.run.smithi116.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T00:51:20.969 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T00:51:20.969 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T00:51:20.969 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T00:51:20.969 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T00:51:20.969 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T00:51:20.969 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T00:51:20.969 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T00:51:20.969 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T00:51:20.969 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T00:51:20.969 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T00:51:20.970 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T00:51:20.970 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T00:51:20.970 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T00:51:20.970 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T00:51:20.970 INFO:teuthology.orchestra.run.smithi116.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T00:51:20.970 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T00:51:20.970 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T00:51:20.970 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T00:51:20.970 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T00:51:20.970 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T00:51:20.970 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T00:51:20.970 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T00:51:20.971 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T00:51:20.971 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T00:51:20.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T00:51:20.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T00:51:20.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T00:51:20.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T00:51:20.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T00:51:20.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T00:51:20.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T00:51:20.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T00:51:20.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T00:51:20.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T00:51:20.971 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T00:51:20.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T00:51:20.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T00:51:20.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T00:51:20.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T00:51:20.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T00:51:20.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T00:51:20.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T00:51:20.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T00:51:20.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T00:51:20.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T00:51:20.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T00:51:20.972 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T00:51:20.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T00:51:20.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T00:51:20.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T00:51:20.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T00:51:20.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T00:51:20.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T00:51:20.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T00:51:20.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T00:51:20.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T00:51:20.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T00:51:20.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T00:51:20.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T00:51:20.973 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T00:51:20.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T00:51:20.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T00:51:20.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T00:51:20.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T00:51:20.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T00:51:20.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T00:51:20.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T00:51:20.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T00:51:20.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T00:51:20.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T00:51:20.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T00:51:20.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T00:51:20.974 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T00:51:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================================ 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout:Install 150 Packages 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 24 Packages 2024-03-19T00:51:20.976 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:51:20.977 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 224 M 2024-03-19T00:51:20.977 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-03-19T00:51:21.500 INFO:teuthology.orchestra.run.smithi116.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-03-19T00:51:21.900 INFO:teuthology.orchestra.run.smithi116.stdout:(2/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 8.4 MB/s | 5.4 MB 00:00 2024-03-19T00:51:21.958 INFO:teuthology.orchestra.run.smithi116.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 2.4 MB/s | 145 kB 00:00 2024-03-19T00:51:22.234 INFO:teuthology.orchestra.run.smithi116.stdout:(4/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 8.6 MB/s | 2.4 MB 00:00 2024-03-19T00:51:22.392 INFO:teuthology.orchestra.run.smithi116.stdout:(5/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 9.4 MB/s | 1.5 MB 00:00 2024-03-19T00:51:22.417 INFO:teuthology.orchestra.run.smithi116.stdout:(6/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 971 kB/s | 891 kB 00:00 2024-03-19T00:51:22.818 INFO:teuthology.orchestra.run.smithi116.stdout:(7/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 11 MB/s | 4.7 MB 00:00 2024-03-19T00:51:23.009 INFO:teuthology.orchestra.run.smithi116.stdout:(8/174): ceph-common-19.0.0-2206.gb6ae2491.el9. 12 MB/s | 21 MB 00:01 2024-03-19T00:51:23.068 INFO:teuthology.orchestra.run.smithi116.stdout:(9/174): ceph-selinux-19.0.0-2206.gb6ae2491.el9 432 kB/s | 25 kB 00:00 2024-03-19T00:51:23.577 INFO:teuthology.orchestra.run.smithi116.stdout:(10/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el 13 MB/s | 9.8 MB 00:00 2024-03-19T00:51:23.627 INFO:teuthology.orchestra.run.smithi116.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 4.4 MB/s | 221 kB 00:00 2024-03-19T00:51:23.685 INFO:teuthology.orchestra.run.smithi116.stdout:(12/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 464 kB/s | 27 kB 00:00 2024-03-19T00:51:23.752 INFO:teuthology.orchestra.run.smithi116.stdout:(13/174): libcephfs-devel-19.0.0-2206.gb6ae2491 477 kB/s | 32 kB 00:00 2024-03-19T00:51:23.844 INFO:teuthology.orchestra.run.smithi116.stdout:(14/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 8.0 MB/s | 746 kB 00:00 2024-03-19T00:51:23.920 INFO:teuthology.orchestra.run.smithi116.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 2.1 MB/s | 163 kB 00:00 2024-03-19T00:51:23.986 INFO:teuthology.orchestra.run.smithi116.stdout:(16/174): librados-devel-19.0.0-2206.gb6ae2491. 1.9 MB/s | 127 kB 00:00 2024-03-19T00:51:24.078 INFO:teuthology.orchestra.run.smithi116.stdout:(17/174): libradosstriper1-19.0.0-2206.gb6ae249 5.3 MB/s | 501 kB 00:00 2024-03-19T00:51:24.578 INFO:teuthology.orchestra.run.smithi116.stdout:(18/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 10 MB/s | 5.1 MB 00:00 2024-03-19T00:51:24.646 INFO:teuthology.orchestra.run.smithi116.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 676 kB/s | 45 kB 00:00 2024-03-19T00:51:24.698 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.1 MB 00:02 2024-03-19T00:51:24.721 INFO:teuthology.orchestra.run.smithi116.stdout:(20/174): python3-ceph-common-19.0.0-2206.gb6ae 1.7 MB/s | 131 kB 00:00 2024-03-19T00:51:24.788 INFO:teuthology.orchestra.run.smithi116.stdout:(21/174): python3-cephfs-19.0.0-2206.gb6ae2491. 2.4 MB/s | 162 kB 00:00 2024-03-19T00:51:24.871 INFO:teuthology.orchestra.run.smithi116.stdout:(22/174): python3-rados-19.0.0-2206.gb6ae2491.e 3.8 MB/s | 321 kB 00:00 2024-03-19T00:51:24.947 INFO:teuthology.orchestra.run.smithi116.stdout:(23/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 3.9 MB/s | 298 kB 00:00 2024-03-19T00:51:25.005 INFO:teuthology.orchestra.run.smithi116.stdout:(24/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 1.7 MB/s | 99 kB 00:00 2024-03-19T00:51:25.072 INFO:teuthology.orchestra.run.smithi116.stdout:(25/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-19T00:51:25.309 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-19T00:51:25.672 INFO:teuthology.orchestra.run.smithi116.stdout:(26/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x8 5.2 MB/s | 17 MB 00:03 2024-03-19T00:51:25.731 INFO:teuthology.orchestra.run.smithi116.stdout:(27/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 4.7 MB/s | 3.1 MB 00:00 2024-03-19T00:51:25.756 INFO:teuthology.orchestra.run.smithi116.stdout:(28/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-19T00:51:25.782 INFO:teuthology.orchestra.run.smithi116.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2206.g 676 kB/s | 34 kB 00:00 2024-03-19T00:51:25.857 INFO:teuthology.orchestra.run.smithi116.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 1.4 MB/s | 139 kB 00:00 2024-03-19T00:51:26.049 INFO:teuthology.orchestra.run.smithi116.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 6.7 MB/s | 1.8 MB 00:00 2024-03-19T00:51:26.141 INFO:teuthology.orchestra.run.smithi116.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 2.6 MB/s | 248 kB 00:00 2024-03-19T00:51:26.225 INFO:teuthology.orchestra.run.smithi116.stdout:(33/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 592 kB/s | 49 kB 00:00 2024-03-19T00:51:26.308 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-19T00:51:26.633 INFO:teuthology.orchestra.run.smithi116.stdout:(34/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 13 MB/s | 48 MB 00:03 2024-03-19T00:51:26.658 INFO:teuthology.orchestra.run.smithi116.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2206.gb 38 kB/s | 16 kB 00:00 2024-03-19T00:51:26.683 INFO:teuthology.orchestra.run.smithi116.stdout:(36/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 5.2 MB/s | 264 kB 00:00 2024-03-19T00:51:26.784 INFO:teuthology.orchestra.run.smithi116.stdout:(37/174): ceph-mgr-diskprediction-local-19.0.0- 8.0 MB/s | 7.4 MB 00:00 2024-03-19T00:51:26.809 INFO:teuthology.orchestra.run.smithi116.stdout:(38/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 146 kB/s | 18 kB 00:00 2024-03-19T00:51:26.843 INFO:teuthology.orchestra.run.smithi116.stdout:(39/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 4.0 MB/s | 749 kB 00:00 2024-03-19T00:51:26.993 INFO:teuthology.orchestra.run.smithi116.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 1.0 MB/s | 106 kB 00:00 2024-03-19T00:51:27.035 INFO:teuthology.orchestra.run.smithi116.stdout:(41/174): bison-3.7.4-5.el9.x86_64.rpm 4.0 MB/s | 920 kB 00:00 2024-03-19T00:51:27.085 INFO:teuthology.orchestra.run.smithi116.stdout:(42/174): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 662 kB 00:00 2024-03-19T00:51:27.110 INFO:teuthology.orchestra.run.smithi116.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 2.6 MB/s | 307 kB 00:00 2024-03-19T00:51:27.127 INFO:teuthology.orchestra.run.smithi116.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 344 kB/s | 32 kB 00:00 2024-03-19T00:51:27.186 INFO:teuthology.orchestra.run.smithi116.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 229 kB/s | 17 kB 00:00 2024-03-19T00:51:27.236 INFO:teuthology.orchestra.run.smithi116.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 93 kB/s | 10 kB 00:00 2024-03-19T00:51:27.287 INFO:teuthology.orchestra.run.smithi116.stdout:(47/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 15 MB/s | 3.0 MB 00:00 2024-03-19T00:51:27.312 INFO:teuthology.orchestra.run.smithi116.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-19T00:51:27.337 INFO:teuthology.orchestra.run.smithi116.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 444 kB/s | 44 kB 00:00 2024-03-19T00:51:27.387 INFO:teuthology.orchestra.run.smithi116.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-19T00:51:27.413 INFO:teuthology.orchestra.run.smithi116.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 5.6 MB/s | 578 kB 00:00 2024-03-19T00:51:27.438 INFO:teuthology.orchestra.run.smithi116.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 464 kB/s | 46 kB 00:00 2024-03-19T00:51:27.463 INFO:teuthology.orchestra.run.smithi116.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-19T00:51:27.488 INFO:teuthology.orchestra.run.smithi116.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-19T00:51:27.513 INFO:teuthology.orchestra.run.smithi116.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-19T00:51:27.538 INFO:teuthology.orchestra.run.smithi116.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 458 kB/s | 34 kB 00:00 2024-03-19T00:51:27.664 INFO:teuthology.orchestra.run.smithi116.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 10 MB/s | 1.5 MB 00:00 2024-03-19T00:51:27.664 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T00:51:27.665 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T00:51:27.679 INFO:teuthology.orchestra.run.smithi156.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T00:51:27.680 INFO:teuthology.orchestra.run.smithi156.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T00:51:27.681 INFO:teuthology.orchestra.run.smithi156.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T00:51:27.682 INFO:teuthology.orchestra.run.smithi156.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T00:51:27.684 INFO:teuthology.orchestra.run.smithi156.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T00:51:27.687 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T00:51:27.688 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T00:51:27.764 INFO:teuthology.orchestra.run.smithi116.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 3.5 MB/s | 811 kB 00:00 2024-03-19T00:51:27.769 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-03-19T00:51:27.781 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-19T00:51:27.781 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-03-19T00:51:27.781 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-19T00:51:27.781 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-03-19T00:51:27.781 INFO:teuthology.orchestra.run.smithi156.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T00:51:27.781 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T00:51:27.781 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T00:51:27.781 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T00:51:27.781 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T00:51:27.782 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T00:51:27.782 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T00:51:27.782 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T00:51:27.782 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T00:51:27.782 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T00:51:27.782 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T00:51:27.782 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T00:51:27.782 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T00:51:27.782 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T00:51:27.782 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T00:51:27.782 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T00:51:27.782 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T00:51:27.782 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T00:51:27.783 INFO:teuthology.orchestra.run.smithi156.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T00:51:27.783 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T00:51:27.783 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T00:51:27.783 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T00:51:27.783 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T00:51:27.783 INFO:teuthology.orchestra.run.smithi156.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T00:51:27.783 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T00:51:27.783 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T00:51:27.783 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T00:51:27.783 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T00:51:27.783 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T00:51:27.783 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T00:51:27.784 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T00:51:27.785 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T00:51:27.785 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T00:51:27.785 INFO:teuthology.orchestra.run.smithi156.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T00:51:27.785 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T00:51:27.785 INFO:teuthology.orchestra.run.smithi156.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T00:51:27.785 INFO:teuthology.orchestra.run.smithi156.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T00:51:27.785 INFO:teuthology.orchestra.run.smithi156.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T00:51:27.785 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T00:51:27.785 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T00:51:27.785 INFO:teuthology.orchestra.run.smithi156.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T00:51:27.785 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T00:51:27.785 INFO:teuthology.orchestra.run.smithi156.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T00:51:27.785 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T00:51:27.786 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T00:51:27.786 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T00:51:27.786 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T00:51:27.786 INFO:teuthology.orchestra.run.smithi156.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T00:51:27.786 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T00:51:27.786 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-03-19T00:51:27.786 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T00:51:27.786 INFO:teuthology.orchestra.run.smithi156.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T00:51:27.786 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T00:51:27.786 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T00:51:27.786 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T00:51:27.786 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T00:51:27.787 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T00:51:27.788 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T00:51:27.788 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T00:51:27.788 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T00:51:27.788 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T00:51:27.788 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T00:51:27.788 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T00:51:27.788 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T00:51:27.788 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T00:51:27.788 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T00:51:27.788 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T00:51:27.788 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T00:51:27.788 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T00:51:27.788 INFO:teuthology.orchestra.run.smithi156.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T00:51:27.789 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T00:51:27.790 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T00:51:27.790 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T00:51:27.790 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T00:51:27.790 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T00:51:27.790 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T00:51:27.790 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T00:51:27.790 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T00:51:27.790 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T00:51:27.790 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T00:51:27.790 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T00:51:27.790 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T00:51:27.790 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T00:51:27.791 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T00:51:27.792 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T00:51:27.792 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T00:51:27.792 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T00:51:27.792 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T00:51:27.792 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T00:51:27.792 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T00:51:27.792 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T00:51:27.792 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T00:51:27.792 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T00:51:27.792 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T00:51:27.792 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T00:51:27.792 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T00:51:27.792 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T00:51:27.793 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T00:51:27.794 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T00:51:27.794 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T00:51:27.794 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T00:51:27.794 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T00:51:27.794 INFO:teuthology.orchestra.run.smithi156.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T00:51:27.794 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T00:51:27.794 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T00:51:27.794 INFO:teuthology.orchestra.run.smithi156.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T00:51:27.794 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T00:51:27.794 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T00:51:27.794 INFO:teuthology.orchestra.run.smithi156.stdout:Installing weak dependencies: 2024-03-19T00:51:27.794 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T00:51:27.794 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T00:51:27.795 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-03-19T00:51:27.795 INFO:teuthology.orchestra.run.smithi156.stdout:============================================================================================ 2024-03-19T00:51:27.795 INFO:teuthology.orchestra.run.smithi156.stdout:Install 150 Packages 2024-03-19T00:51:27.795 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 24 Packages 2024-03-19T00:51:27.795 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T00:51:27.795 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 224 M 2024-03-19T00:51:27.795 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-03-19T00:51:27.831 INFO:teuthology.orchestra.run.smithi116.stdout:(59/174): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-03-19T00:51:27.856 INFO:teuthology.orchestra.run.smithi116.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 704 kB/s | 134 kB 00:00 2024-03-19T00:51:27.965 INFO:teuthology.orchestra.run.smithi116.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 28 MB/s | 5.7 MB 00:00 2024-03-19T00:51:27.990 INFO:teuthology.orchestra.run.smithi116.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 593 kB/s | 78 kB 00:00 2024-03-19T00:51:28.020 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-19T00:51:28.032 INFO:teuthology.orchestra.run.smithi116.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-19T00:51:28.125 INFO:teuthology.orchestra.run.smithi116.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 36 MB/s | 5.8 MB 00:00 2024-03-19T00:51:28.150 INFO:teuthology.orchestra.run.smithi116.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 158 kB/s | 24 kB 00:00 2024-03-19T00:51:28.175 INFO:teuthology.orchestra.run.smithi116.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 130 kB/s | 18 kB 00:00 2024-03-19T00:51:28.200 INFO:teuthology.orchestra.run.smithi116.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-19T00:51:28.225 INFO:teuthology.orchestra.run.smithi116.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 420 kB/s | 32 kB 00:00 2024-03-19T00:51:28.309 INFO:teuthology.orchestra.run.smithi116.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-19T00:51:28.334 INFO:teuthology.orchestra.run.smithi116.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2024-03-19T00:51:28.455 INFO:teuthology.orchestra.run.smithi116.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 771 kB/s | 174 kB 00:00 2024-03-19T00:51:28.479 INFO:teuthology.orchestra.run.smithi116.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 412 kB/s | 69 kB 00:00 2024-03-19T00:51:28.502 INFO:teuthology.orchestra.run.smithi156.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-03-19T00:51:28.510 INFO:teuthology.orchestra.run.smithi116.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.3 MB/s | 238 kB 00:00 2024-03-19T00:51:28.535 INFO:teuthology.orchestra.run.smithi116.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-19T00:51:28.560 INFO:teuthology.orchestra.run.smithi116.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-19T00:51:28.586 INFO:teuthology.orchestra.run.smithi116.stdout:(76/174): python3-prettytable-0.7.2-27.el9.noar 316 kB/s | 41 kB 00:00 2024-03-19T00:51:28.611 INFO:teuthology.orchestra.run.smithi116.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 631 kB/s | 47 kB 00:00 2024-03-19T00:51:28.636 INFO:teuthology.orchestra.run.smithi116.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 569 kB/s | 43 kB 00:00 2024-03-19T00:51:28.661 INFO:teuthology.orchestra.run.smithi116.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-19T00:51:28.677 INFO:teuthology.orchestra.run.smithi156.stdout:(2/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 5.0 MB/s | 891 kB 00:00 2024-03-19T00:51:28.752 INFO:teuthology.orchestra.run.smithi156.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 1.9 MB/s | 145 kB 00:00 2024-03-19T00:51:28.787 INFO:teuthology.orchestra.run.smithi116.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 273 kB/s | 41 kB 00:00 2024-03-19T00:51:28.954 INFO:teuthology.orchestra.run.smithi116.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-19T00:51:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 955 kB/s | 300 kB 00:00 2024-03-19T00:51:28.996 INFO:teuthology.orchestra.run.smithi116.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 273 kB/s | 56 kB 00:00 2024-03-19T00:51:29.012 INFO:teuthology.orchestra.run.smithi116.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-03-19T00:51:29.038 INFO:teuthology.orchestra.run.smithi116.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-19T00:51:29.044 INFO:teuthology.orchestra.run.smithi156.stdout:(4/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 8.1 MB/s | 2.4 MB 00:00 2024-03-19T00:51:29.055 INFO:teuthology.orchestra.run.smithi116.stdout:(86/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 755 kB/s | 44 kB 00:00 2024-03-19T00:51:29.080 INFO:teuthology.orchestra.run.smithi116.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-19T00:51:29.113 INFO:teuthology.orchestra.run.smithi116.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.1 MB 00:00 2024-03-19T00:51:29.139 INFO:teuthology.orchestra.run.smithi116.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-19T00:51:29.164 INFO:teuthology.orchestra.run.smithi116.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-19T00:51:29.189 INFO:teuthology.orchestra.run.smithi116.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-19T00:51:29.403 INFO:teuthology.orchestra.run.smithi156.stdout:(5/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 4.2 MB/s | 1.5 MB 00:00 2024-03-19T00:51:29.703 INFO:teuthology.orchestra.run.smithi116.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 348 kB/s | 187 kB 00:00 2024-03-19T00:51:29.729 INFO:teuthology.orchestra.run.smithi116.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 363 kB/s | 195 kB 00:00 2024-03-19T00:51:29.795 INFO:teuthology.orchestra.run.smithi156.stdout:(6/174): ceph-common-19.0.0-2206.gb6ae2491.el9. 14 MB/s | 21 MB 00:01 2024-03-19T00:51:29.820 INFO:teuthology.orchestra.run.smithi116.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 235 kB/s | 21 kB 00:00 2024-03-19T00:51:29.845 INFO:teuthology.orchestra.run.smithi116.stdout:(95/174): python3-requests-2.25.1-8.el9.noarch. 875 kB/s | 125 kB 00:00 2024-03-19T00:51:29.870 INFO:teuthology.orchestra.run.smithi116.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 505 kB/s | 25 kB 00:00 2024-03-19T00:51:29.886 INFO:teuthology.orchestra.run.smithi156.stdout:(7/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 9.7 MB/s | 4.7 MB 00:00 2024-03-19T00:51:29.896 INFO:teuthology.orchestra.run.smithi116.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-19T00:51:29.954 INFO:teuthology.orchestra.run.smithi116.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-03-19T00:51:30.005 INFO:teuthology.orchestra.run.smithi116.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-19T00:51:30.030 INFO:teuthology.orchestra.run.smithi116.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.8 MB/s | 248 kB 00:00 2024-03-19T00:51:30.047 INFO:teuthology.orchestra.run.smithi116.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 833 kB/s | 35 kB 00:00 2024-03-19T00:51:30.064 INFO:teuthology.orchestra.run.smithi116.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-19T00:51:30.089 INFO:teuthology.orchestra.run.smithi116.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-03-19T00:51:30.106 INFO:teuthology.orchestra.run.smithi116.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 532 kB/s | 22 kB 00:00 2024-03-19T00:51:30.173 INFO:teuthology.orchestra.run.smithi116.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 13 MB/s | 4.1 MB 00:00 2024-03-19T00:51:30.207 INFO:teuthology.orchestra.run.smithi116.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 7.2 MB/s | 863 kB 00:00 2024-03-19T00:51:30.607 INFO:teuthology.orchestra.run.smithi116.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 49 kB/s | 19 kB 00:00 2024-03-19T00:51:30.874 INFO:teuthology.orchestra.run.smithi116.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 441 kB/s | 309 kB 00:00 2024-03-19T00:51:30.945 INFO:teuthology.orchestra.run.smithi156.stdout:(8/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x86 15 MB/s | 17 MB 00:01 2024-03-19T00:51:31.062 INFO:teuthology.orchestra.run.smithi156.stdout:(9/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el9 8.4 MB/s | 9.8 MB 00:01 2024-03-19T00:51:31.066 INFO:teuthology.orchestra.run.smithi116.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 571 kB/s | 548 kB 00:00 2024-03-19T00:51:31.087 INFO:teuthology.orchestra.run.smithi156.stdout:(10/174): ceph-selinux-19.0.0-2206.gb6ae2491.el 178 kB/s | 25 kB 00:00 2024-03-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi156.stdout:(11/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 3.7 MB/s | 221 kB 00:00 2024-03-19T00:51:31.221 INFO:teuthology.orchestra.run.smithi156.stdout:(12/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x 1.8 MB/s | 5.4 MB 00:02 2024-03-19T00:51:31.246 INFO:teuthology.orchestra.run.smithi156.stdout:(13/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 269 kB/s | 27 kB 00:00 2024-03-19T00:51:31.274 INFO:teuthology.orchestra.run.smithi116.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 65 kB/s | 26 kB 00:00 2024-03-19T00:51:31.280 INFO:teuthology.orchestra.run.smithi156.stdout:(14/174): libcephfs-devel-19.0.0-2206.gb6ae2491 543 kB/s | 32 kB 00:00 2024-03-19T00:51:31.338 INFO:teuthology.orchestra.run.smithi156.stdout:(15/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 7.9 MB/s | 746 kB 00:00 2024-03-19T00:51:31.381 INFO:teuthology.orchestra.run.smithi156.stdout:(16/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 1.6 MB/s | 163 kB 00:00 2024-03-19T00:51:31.397 INFO:teuthology.orchestra.run.smithi156.stdout:(17/174): librados-devel-19.0.0-2206.gb6ae2491. 2.1 MB/s | 127 kB 00:00 2024-03-19T00:51:31.575 INFO:teuthology.orchestra.run.smithi116.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 96 kB/s | 49 kB 00:00 2024-03-19T00:51:31.642 INFO:teuthology.orchestra.run.smithi156.stdout:(18/174): libradosstriper1-19.0.0-2206.gb6ae249 1.9 MB/s | 501 kB 00:00 2024-03-19T00:51:31.784 INFO:teuthology.orchestra.run.smithi156.stdout:(19/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 13 MB/s | 5.1 MB 00:00 2024-03-19T00:51:31.809 INFO:teuthology.orchestra.run.smithi156.stdout:(20/174): python3-ceph-argparse-19.0.0-2206.gb6 271 kB/s | 45 kB 00:00 2024-03-19T00:51:31.825 INFO:teuthology.orchestra.run.smithi116.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 123 kB/s | 67 kB 00:00 2024-03-19T00:51:31.842 INFO:teuthology.orchestra.run.smithi156.stdout:(21/174): python3-ceph-common-19.0.0-2206.gb6ae 2.2 MB/s | 131 kB 00:00 2024-03-19T00:51:31.885 INFO:teuthology.orchestra.run.smithi156.stdout:(22/174): python3-cephfs-19.0.0-2206.gb6ae2491. 2.1 MB/s | 162 kB 00:00 2024-03-19T00:51:31.910 INFO:teuthology.orchestra.run.smithi156.stdout:(23/174): python3-rados-19.0.0-2206.gb6ae2491.e 4.7 MB/s | 321 kB 00:00 2024-03-19T00:51:31.969 INFO:teuthology.orchestra.run.smithi156.stdout:(24/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 1.7 MB/s | 99 kB 00:00 2024-03-19T00:51:32.010 INFO:teuthology.orchestra.run.smithi156.stdout:(25/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 2.3 MB/s | 298 kB 00:00 2024-03-19T00:51:32.036 INFO:teuthology.orchestra.run.smithi156.stdout:(26/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-19T00:51:32.103 INFO:teuthology.orchestra.run.smithi156.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-19T00:51:32.161 INFO:teuthology.orchestra.run.smithi156.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2206.g 583 kB/s | 34 kB 00:00 2024-03-19T00:51:32.200 INFO:teuthology.orchestra.run.smithi116.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 242 kB/s | 151 kB 00:00 2024-03-19T00:51:32.228 INFO:teuthology.orchestra.run.smithi156.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 2.0 MB/s | 139 kB 00:00 2024-03-19T00:51:32.412 INFO:teuthology.orchestra.run.smithi156.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 9.8 MB/s | 1.8 MB 00:00 2024-03-19T00:51:32.858 INFO:teuthology.orchestra.run.smithi116.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 811 kB/s | 838 kB 00:01 2024-03-19T00:51:32.971 INFO:teuthology.orchestra.run.smithi156.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-03-19T00:51:33.038 INFO:teuthology.orchestra.run.smithi156.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 3.6 MB/s | 248 kB 00:00 2024-03-19T00:51:33.077 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-19T00:51:33.096 INFO:teuthology.orchestra.run.smithi156.stdout:(33/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 849 kB/s | 49 kB 00:00 2024-03-19T00:51:33.146 INFO:teuthology.orchestra.run.smithi156.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2206.gb 330 kB/s | 16 kB 00:00 2024-03-19T00:51:33.167 INFO:teuthology.orchestra.run.smithi116.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 567 kB/s | 548 kB 00:00 2024-03-19T00:51:33.214 INFO:teuthology.orchestra.run.smithi156.stdout:(35/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 3.9 MB/s | 264 kB 00:00 2024-03-19T00:51:33.272 INFO:teuthology.orchestra.run.smithi156.stdout:(36/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 2.4 MB/s | 3.1 MB 00:01 2024-03-19T00:51:33.306 INFO:teuthology.orchestra.run.smithi156.stdout:(37/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 8.0 MB/s | 749 kB 00:00 2024-03-19T00:51:33.343 INFO:teuthology.orchestra.run.smithi116.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 90 kB/s | 43 kB 00:00 2024-03-19T00:51:33.364 INFO:teuthology.orchestra.run.smithi156.stdout:(38/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 199 kB/s | 18 kB 00:00 2024-03-19T00:51:33.465 INFO:teuthology.orchestra.run.smithi156.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 662 kB 00:00 2024-03-19T00:51:33.498 INFO:teuthology.orchestra.run.smithi156.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 6.8 MB/s | 920 kB 00:00 2024-03-19T00:51:33.524 INFO:teuthology.orchestra.run.smithi156.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-19T00:51:33.549 INFO:teuthology.orchestra.run.smithi156.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-19T00:51:33.574 INFO:teuthology.orchestra.run.smithi156.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 639 kB/s | 32 kB 00:00 2024-03-19T00:51:33.641 INFO:teuthology.orchestra.run.smithi156.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-19T00:51:33.659 INFO:teuthology.orchestra.run.smithi156.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 212 kB/s | 17 kB 00:00 2024-03-19T00:51:33.668 INFO:teuthology.orchestra.run.smithi116.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 64 kB/s | 32 kB 00:00 2024-03-19T00:51:33.675 INFO:teuthology.orchestra.run.smithi156.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 301 kB/s | 10 kB 00:00 2024-03-19T00:51:33.701 INFO:teuthology.orchestra.run.smithi156.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-19T00:51:33.718 INFO:teuthology.orchestra.run.smithi156.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-19T00:51:33.760 INFO:teuthology.orchestra.run.smithi156.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-19T00:51:33.760 INFO:teuthology.orchestra.run.smithi116.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 36 kB/s | 15 kB 00:00 2024-03-19T00:51:33.785 INFO:teuthology.orchestra.run.smithi156.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 578 kB 00:00 2024-03-19T00:51:33.810 INFO:teuthology.orchestra.run.smithi156.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 46 kB 00:00 2024-03-19T00:51:33.844 INFO:teuthology.orchestra.run.smithi156.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-19T00:51:33.869 INFO:teuthology.orchestra.run.smithi156.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-19T00:51:33.894 INFO:teuthology.orchestra.run.smithi156.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-19T00:51:33.920 INFO:teuthology.orchestra.run.smithi156.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 688 kB/s | 34 kB 00:00 2024-03-19T00:51:33.977 INFO:teuthology.orchestra.run.smithi116.stdout:(119/174): libarrow-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 4.4 MB 00:03 2024-03-19T00:51:34.053 INFO:teuthology.orchestra.run.smithi156.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-19T00:51:34.120 INFO:teuthology.orchestra.run.smithi156.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 7.7 MB/s | 1.5 MB 00:00 2024-03-19T00:51:34.162 INFO:teuthology.orchestra.run.smithi156.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 7.5 MB/s | 811 kB 00:00 2024-03-19T00:51:34.187 INFO:teuthology.orchestra.run.smithi156.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-19T00:51:34.360 INFO:teuthology.orchestra.run.smithi116.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 248 kB/s | 172 kB 00:00 2024-03-19T00:51:34.569 INFO:teuthology.orchestra.run.smithi116.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 459 kB/s | 358 kB 00:00 2024-03-19T00:51:34.671 INFO:teuthology.orchestra.run.smithi156.stdout:(60/174): ceph-test-19.0.0-2206.gb6ae2491.el9.x 13 MB/s | 48 MB 00:03 2024-03-19T00:51:34.747 INFO:teuthology.orchestra.run.smithi156.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 9.7 MB/s | 5.7 MB 00:00 2024-03-19T00:51:34.780 INFO:teuthology.orchestra.run.smithi156.stdout:(62/174): protobuf-3.14.0-13.el9.x86_64.rpm 1.7 MB/s | 1.0 MB 00:00 2024-03-19T00:51:34.794 INFO:teuthology.orchestra.run.smithi116.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 372 kB/s | 303 kB 00:00 2024-03-19T00:51:34.797 INFO:teuthology.orchestra.run.smithi156.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 636 kB/s | 78 kB 00:00 2024-03-19T00:51:34.822 INFO:teuthology.orchestra.run.smithi156.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 589 kB/s | 24 kB 00:00 2024-03-19T00:51:34.873 INFO:teuthology.orchestra.run.smithi156.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 240 kB/s | 18 kB 00:00 2024-03-19T00:51:34.936 INFO:teuthology.orchestra.run.smithi116.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 61 kB/s | 35 kB 00:00 2024-03-19T00:51:34.948 INFO:teuthology.orchestra.run.smithi156.stdout:(66/174): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-19T00:51:34.973 INFO:teuthology.orchestra.run.smithi156.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 980 kB/s | 144 kB 00:00 2024-03-19T00:51:34.998 INFO:teuthology.orchestra.run.smithi156.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 258 kB/s | 32 kB 00:00 2024-03-19T00:51:35.024 INFO:teuthology.orchestra.run.smithi156.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.3 MB/s | 217 kB 00:00 2024-03-19T00:51:35.116 INFO:teuthology.orchestra.run.smithi156.stdout:(70/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-19T00:51:35.141 INFO:teuthology.orchestra.run.smithi156.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-19T00:51:35.166 INFO:teuthology.orchestra.run.smithi156.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 494 kB/s | 69 kB 00:00 2024-03-19T00:51:35.191 INFO:teuthology.orchestra.run.smithi156.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 548 kB/s | 41 kB 00:00 2024-03-19T00:51:35.203 INFO:teuthology.orchestra.run.smithi116.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 320 kB/s | 203 kB 00:00 2024-03-19T00:51:35.217 INFO:teuthology.orchestra.run.smithi156.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-19T00:51:35.242 INFO:teuthology.orchestra.run.smithi156.stdout:(75/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-19T00:51:35.267 INFO:teuthology.orchestra.run.smithi156.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-19T00:51:35.292 INFO:teuthology.orchestra.run.smithi156.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 637 kB/s | 47 kB 00:00 2024-03-19T00:51:35.318 INFO:teuthology.orchestra.run.smithi156.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 858 kB/s | 43 kB 00:00 2024-03-19T00:51:35.360 INFO:teuthology.orchestra.run.smithi156.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 578 kB/s | 39 kB 00:00 2024-03-19T00:51:35.402 INFO:teuthology.orchestra.run.smithi156.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 510 kB/s | 41 kB 00:00 2024-03-19T00:51:35.413 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T00:51:35.414 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-19T00:51:35.426 INFO:teuthology.orchestra.run.smithi162.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-19T00:51:35.427 INFO:teuthology.orchestra.run.smithi162.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-19T00:51:35.428 INFO:teuthology.orchestra.run.smithi162.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-19T00:51:35.429 INFO:teuthology.orchestra.run.smithi162.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-19T00:51:35.430 INFO:teuthology.orchestra.run.smithi162.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-19T00:51:35.433 INFO:teuthology.orchestra.run.smithi162.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-19T00:51:35.434 INFO:teuthology.orchestra.run.smithi162.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-19T00:51:35.452 INFO:teuthology.orchestra.run.smithi156.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.3 MB/s | 300 kB 00:00 2024-03-19T00:51:35.516 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-19T00:51:35.528 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-19T00:51:35.528 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-19T00:51:35.528 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-19T00:51:35.528 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-19T00:51:35.528 INFO:teuthology.orchestra.run.smithi162.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-19T00:51:35.528 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 6.6 k 2024-03-19T00:51:35.528 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.4 M 2024-03-19T00:51:35.529 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 891 k 2024-03-19T00:51:35.529 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 145 k 2024-03-19T00:51:35.529 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 1.5 M 2024-03-19T00:51:35.529 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 139 k 2024-03-19T00:51:35.529 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 1.8 M 2024-03-19T00:51:35.529 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 7.4 M 2024-03-19T00:51:35.529 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 49 k 2024-03-19T00:51:35.529 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 9.8 M 2024-03-19T00:51:35.529 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 48 M 2024-03-19T00:51:35.529 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 264 k 2024-03-19T00:51:35.529 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 749 k 2024-03-19T00:51:35.529 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 221 k 2024-03-19T00:51:35.529 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-shell x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 27 k 2024-03-19T00:51:35.530 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-top noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 18 k 2024-03-19T00:51:35.530 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-19T00:51:35.530 INFO:teuthology.orchestra.run.smithi162.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-19T00:51:35.530 INFO:teuthology.orchestra.run.smithi162.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-19T00:51:35.530 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 32 k 2024-03-19T00:51:35.530 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 746 k 2024-03-19T00:51:35.530 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 127 k 2024-03-19T00:51:35.530 INFO:teuthology.orchestra.run.smithi162.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-19T00:51:35.530 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-19T00:51:35.530 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-19T00:51:35.530 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-19T00:51:35.530 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-19T00:51:35.531 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-19T00:51:35.531 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 162 k 2024-03-19T00:51:35.531 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 321 k 2024-03-19T00:51:35.531 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 298 k 2024-03-19T00:51:35.531 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 99 k 2024-03-19T00:51:35.531 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 85 k 2024-03-19T00:51:35.531 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T00:51:35.531 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 170 k 2024-03-19T00:51:35.531 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-19T00:51:35.531 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-03-19T00:51:35.531 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-19T00:51:35.531 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-19T00:51:35.531 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-19T00:51:35.532 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-19T00:51:35.532 INFO:teuthology.orchestra.run.smithi162.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-19T00:51:35.532 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-19T00:51:35.532 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-19T00:51:35.532 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-19T00:51:35.532 INFO:teuthology.orchestra.run.smithi162.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-19T00:51:35.532 INFO:teuthology.orchestra.run.smithi162.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-19T00:51:35.532 INFO:teuthology.orchestra.run.smithi162.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-19T00:51:35.532 INFO:teuthology.orchestra.run.smithi162.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-19T00:51:35.532 INFO:teuthology.orchestra.run.smithi162.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-19T00:51:35.532 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.4 M 2024-03-19T00:51:35.532 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 3.1 M 2024-03-19T00:51:35.532 INFO:teuthology.orchestra.run.smithi162.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 21 M 2024-03-19T00:51:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 34 k 2024-03-19T00:51:35.534 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 2.4 M 2024-03-19T00:51:35.534 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 248 k 2024-03-19T00:51:35.534 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 4.7 M 2024-03-19T00:51:35.534 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 17 M 2024-03-19T00:51:35.534 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2206.gb6ae2491.el9 ceph-noarch 16 k 2024-03-19T00:51:35.534 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 25 k 2024-03-19T00:51:35.534 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-19T00:51:35.534 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-19T00:51:35.534 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-19T00:51:35.534 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-19T00:51:35.534 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-19T00:51:35.534 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-19T00:51:35.534 INFO:teuthology.orchestra.run.smithi162.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 163 k 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 501 k 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 5.1 M 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-19T00:51:35.535 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-19T00:51:35.536 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-19T00:51:35.537 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-19T00:51:35.537 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 45 k 2024-03-19T00:51:35.537 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:19.0.0-2206.gb6ae2491.el9 ceph 131 k 2024-03-19T00:51:35.537 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-19T00:51:35.537 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-19T00:51:35.537 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-19T00:51:35.537 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-19T00:51:35.537 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-19T00:51:35.537 INFO:teuthology.orchestra.run.smithi162.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-19T00:51:35.537 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-19T00:51:35.537 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-19T00:51:35.537 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-19T00:51:35.537 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-19T00:51:35.538 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-19T00:51:35.539 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-19T00:51:35.540 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-19T00:51:35.541 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:51:35.542 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-19T00:51:35.542 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-19T00:51:35.542 INFO:teuthology.orchestra.run.smithi162.stdout:Install 150 Packages 2024-03-19T00:51:35.542 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 24 Packages 2024-03-19T00:51:35.542 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:51:35.542 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 224 M 2024-03-19T00:51:35.542 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-19T00:51:35.570 INFO:teuthology.orchestra.run.smithi116.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 228 kB/s | 144 kB 00:00 2024-03-19T00:51:35.595 INFO:teuthology.orchestra.run.smithi116.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 27 kB/s | 11 kB 00:00 2024-03-19T00:51:35.602 INFO:teuthology.orchestra.run.smithi156.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-19T00:51:35.619 INFO:teuthology.orchestra.run.smithi156.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 263 kB/s | 56 kB 00:00 2024-03-19T00:51:35.636 INFO:teuthology.orchestra.run.smithi156.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 345 kB/s | 63 kB 00:00 2024-03-19T00:51:35.654 INFO:teuthology.orchestra.run.smithi156.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-19T00:51:35.678 INFO:teuthology.orchestra.run.smithi156.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-19T00:51:35.703 INFO:teuthology.orchestra.run.smithi156.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-19T00:51:35.720 INFO:teuthology.orchestra.run.smithi156.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-19T00:51:35.754 INFO:teuthology.orchestra.run.smithi156.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.1 MB 00:00 2024-03-19T00:51:35.779 INFO:teuthology.orchestra.run.smithi156.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 124 kB 00:00 2024-03-19T00:51:35.804 INFO:teuthology.orchestra.run.smithi156.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-19T00:51:35.970 INFO:teuthology.orchestra.run.smithi116.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 44 kB/s | 18 kB 00:00 2024-03-19T00:51:36.004 INFO:teuthology.orchestra.run.smithi116.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 57 kB/s | 23 kB 00:00 2024-03-19T00:51:36.070 INFO:teuthology.orchestra.run.smithi162.stdout:(1/174): ceph-19.0.0-2206.gb6ae2491.el9.x86_64. 41 kB/s | 6.6 kB 00:00 2024-03-19T00:51:36.270 INFO:teuthology.orchestra.run.smithi162.stdout:(2/174): ceph-fuse-19.0.0-2206.gb6ae2491.el9.x8 4.4 MB/s | 891 kB 00:00 2024-03-19T00:51:36.336 INFO:teuthology.orchestra.run.smithi162.stdout:(3/174): ceph-immutable-object-cache-19.0.0-220 2.1 MB/s | 145 kB 00:00 2024-03-19T00:51:36.383 INFO:teuthology.orchestra.run.smithi116.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 47 kB/s | 19 kB 00:00 2024-03-19T00:51:36.413 INFO:teuthology.orchestra.run.smithi116.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 48 kB/s | 20 kB 00:00 2024-03-19T00:51:36.621 INFO:teuthology.orchestra.run.smithi116.stdout:(131/174): python3-grpcio-1.46.7-10.el9.x86_64. 1.1 MB/s | 2.0 MB 00:01 2024-03-19T00:51:36.645 INFO:teuthology.orchestra.run.smithi162.stdout:(4/174): ceph-mds-19.0.0-2206.gb6ae2491.el9.x86 7.7 MB/s | 2.4 MB 00:00 2024-03-19T00:51:36.887 INFO:teuthology.orchestra.run.smithi162.stdout:(5/174): ceph-mgr-19.0.0-2206.gb6ae2491.el9.x86 6.2 MB/s | 1.5 MB 00:00 2024-03-19T00:51:36.913 INFO:teuthology.orchestra.run.smithi116.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 93 kB/s | 46 kB 00:00 2024-03-19T00:51:37.197 INFO:teuthology.orchestra.run.smithi116.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 137 kB/s | 79 kB 00:00 2024-03-19T00:51:37.397 INFO:teuthology.orchestra.run.smithi116.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 119 kB/s | 58 kB 00:00 2024-03-19T00:51:37.487 INFO:teuthology.orchestra.run.smithi162.stdout:(6/174): ceph-common-19.0.0-2206.gb6ae2491.el9. 13 MB/s | 21 MB 00:01 2024-03-19T00:51:37.539 INFO:teuthology.orchestra.run.smithi116.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 905 kB/s | 1.0 MB 00:01 2024-03-19T00:51:37.571 INFO:teuthology.orchestra.run.smithi162.stdout:(7/174): ceph-mon-19.0.0-2206.gb6ae2491.el9.x86 6.9 MB/s | 4.7 MB 00:00 2024-03-19T00:51:37.679 INFO:teuthology.orchestra.run.smithi162.stdout:(8/174): ceph-base-19.0.0-2206.gb6ae2491.el9.x8 3.0 MB/s | 5.4 MB 00:01 2024-03-19T00:51:37.789 INFO:teuthology.orchestra.run.smithi116.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 42 kB/s | 16 kB 00:00 2024-03-19T00:51:37.796 INFO:teuthology.orchestra.run.smithi162.stdout:(9/174): ceph-selinux-19.0.0-2206.gb6ae2491.el9 216 kB/s | 25 kB 00:00 2024-03-19T00:51:37.989 INFO:teuthology.orchestra.run.smithi116.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 344 kB/s | 272 kB 00:00 2024-03-19T00:51:38.090 INFO:teuthology.orchestra.run.smithi116.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 163 kB/s | 90 kB 00:00 2024-03-19T00:51:38.256 INFO:teuthology.orchestra.run.smithi116.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 60 kB/s | 28 kB 00:00 2024-03-19T00:51:38.465 INFO:teuthology.orchestra.run.smithi116.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 65 kB/s | 31 kB 00:00 2024-03-19T00:51:38.730 INFO:teuthology.orchestra.run.smithi162.stdout:(10/174): ceph-osd-19.0.0-2206.gb6ae2491.el9.x8 14 MB/s | 17 MB 00:01 2024-03-19T00:51:38.740 INFO:teuthology.orchestra.run.smithi116.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 122 kB/s | 59 kB 00:00 2024-03-19T00:51:38.808 INFO:teuthology.orchestra.run.smithi116.stdout:(142/174): python3-routes-2.5.1-5.el9.noarch.rp 273 kB/s | 188 kB 00:00 2024-03-19T00:51:38.847 INFO:teuthology.orchestra.run.smithi162.stdout:(11/174): ceph-radosgw-19.0.0-2206.gb6ae2491.el 7.7 MB/s | 9.8 MB 00:01 2024-03-19T00:51:38.872 INFO:teuthology.orchestra.run.smithi162.stdout:(12/174): cephfs-mirror-19.0.0-2206.gb6ae2491.e 1.5 MB/s | 221 kB 00:00 2024-03-19T00:51:38.897 INFO:teuthology.orchestra.run.smithi162.stdout:(13/174): cephfs-shell-19.0.0-2206.gb6ae2491.el 540 kB/s | 27 kB 00:00 2024-03-19T00:51:38.922 INFO:teuthology.orchestra.run.smithi162.stdout:(14/174): libcephfs-devel-19.0.0-2206.gb6ae2491 630 kB/s | 32 kB 00:00 2024-03-19T00:51:38.933 INFO:teuthology.orchestra.run.smithi116.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 77 kB/s | 36 kB 00:00 2024-03-19T00:51:38.973 INFO:teuthology.orchestra.run.smithi162.stdout:(15/174): libcephsqlite-19.0.0-2206.gb6ae2491.e 3.2 MB/s | 163 kB 00:00 2024-03-19T00:51:38.998 INFO:teuthology.orchestra.run.smithi162.stdout:(16/174): libcephfs2-19.0.0-2206.gb6ae2491.el9. 7.3 MB/s | 746 kB 00:00 2024-03-19T00:51:39.023 INFO:teuthology.orchestra.run.smithi162.stdout:(17/174): librados-devel-19.0.0-2206.gb6ae2491. 2.5 MB/s | 127 kB 00:00 2024-03-19T00:51:39.066 INFO:teuthology.orchestra.run.smithi162.stdout:(18/174): libradosstriper1-19.0.0-2206.gb6ae249 7.3 MB/s | 501 kB 00:00 2024-03-19T00:51:39.124 INFO:teuthology.orchestra.run.smithi162.stdout:(19/174): python3-ceph-argparse-19.0.0-2206.gb6 777 kB/s | 45 kB 00:00 2024-03-19T00:51:39.191 INFO:teuthology.orchestra.run.smithi162.stdout:(20/174): python3-ceph-common-19.0.0-2206.gb6ae 1.9 MB/s | 131 kB 00:00 2024-03-19T00:51:39.216 INFO:teuthology.orchestra.run.smithi116.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 107 kB/s | 51 kB 00:00 2024-03-19T00:51:39.258 INFO:teuthology.orchestra.run.smithi162.stdout:(21/174): python3-cephfs-19.0.0-2206.gb6ae2491. 2.4 MB/s | 162 kB 00:00 2024-03-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi162.stdout:(22/174): librgw2-19.0.0-2206.gb6ae2491.el9.x86 15 MB/s | 5.1 MB 00:00 2024-03-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi162.stdout:(23/174): python3-rados-19.0.0-2206.gb6ae2491.e 2.2 MB/s | 321 kB 00:00 2024-03-19T00:51:39.425 INFO:teuthology.orchestra.run.smithi162.stdout:(24/174): python3-rbd-19.0.0-2206.gb6ae2491.el9 5.8 MB/s | 298 kB 00:00 2024-03-19T00:51:39.451 INFO:teuthology.orchestra.run.smithi162.stdout:(25/174): python3-rgw-19.0.0-2206.gb6ae2491.el9 1.9 MB/s | 99 kB 00:00 2024-03-19T00:51:39.468 INFO:teuthology.orchestra.run.smithi116.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 348 kB/s | 230 kB 00:00 2024-03-19T00:51:39.468 INFO:teuthology.orchestra.run.smithi162.stdout:(26/174): rbd-fuse-19.0.0-2206.gb6ae2491.el9.x8 2.0 MB/s | 85 kB 00:00 2024-03-19T00:51:39.501 INFO:teuthology.orchestra.run.smithi116.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 158 kB/s | 90 kB 00:00 2024-03-19T00:51:39.527 INFO:teuthology.orchestra.run.smithi162.stdout:(27/174): rbd-nbd-19.0.0-2206.gb6ae2491.el9.x86 2.8 MB/s | 170 kB 00:00 2024-03-19T00:51:39.585 INFO:teuthology.orchestra.run.smithi162.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2206.g 584 kB/s | 34 kB 00:00 2024-03-19T00:51:39.643 INFO:teuthology.orchestra.run.smithi162.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2206.gb6ae249 2.3 MB/s | 139 kB 00:00 2024-03-19T00:51:39.694 INFO:teuthology.orchestra.run.smithi162.stdout:(30/174): rbd-mirror-19.0.0-2206.gb6ae2491.el9. 13 MB/s | 3.1 MB 00:00 2024-03-19T00:51:39.811 INFO:teuthology.orchestra.run.smithi162.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2206.gb6ae2 11 MB/s | 1.8 MB 00:00 2024-03-19T00:51:39.885 INFO:teuthology.orchestra.run.smithi116.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 48 kB/s | 20 kB 00:00 2024-03-19T00:51:39.894 INFO:teuthology.orchestra.run.smithi162.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2206.gb6 2.9 MB/s | 248 kB 00:00 2024-03-19T00:51:39.953 INFO:teuthology.orchestra.run.smithi162.stdout:(33/174): ceph-mgr-rook-19.0.0-2206.gb6ae2491.e 845 kB/s | 49 kB 00:00 2024-03-19T00:51:40.012 INFO:teuthology.orchestra.run.smithi162.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2206.gb 282 kB/s | 16 kB 00:00 2024-03-19T00:51:40.018 INFO:teuthology.orchestra.run.smithi116.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 533 kB/s | 427 kB 00:00 2024-03-19T00:51:40.087 INFO:teuthology.orchestra.run.smithi162.stdout:(35/174): ceph-volume-19.0.0-2206.gb6ae2491.el9 3.4 MB/s | 264 kB 00:00 2024-03-19T00:51:40.127 INFO:teuthology.orchestra.run.smithi116.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 306 kB/s | 192 kB 00:00 2024-03-19T00:51:40.187 INFO:teuthology.orchestra.run.smithi162.stdout:(36/174): cephadm-19.0.0-2206.gb6ae2491.el9.noa 7.3 MB/s | 749 kB 00:00 2024-03-19T00:51:40.287 INFO:teuthology.orchestra.run.smithi162.stdout:(37/174): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-03-19T00:51:40.313 INFO:teuthology.orchestra.run.smithi162.stdout:(38/174): cephfs-top-19.0.0-2206.gb6ae2491.el9. 146 kB/s | 18 kB 00:00 2024-03-19T00:51:40.455 INFO:teuthology.orchestra.run.smithi162.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2024-03-19T00:51:40.469 INFO:teuthology.orchestra.run.smithi116.stdout:(150/174): librados2-19.0.0-2206.gb6ae2491.el9. 7.5 MB/s | 3.4 MB 00:00 2024-03-19T00:51:40.488 INFO:teuthology.orchestra.run.smithi162.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.1 MB/s | 920 kB 00:00 2024-03-19T00:51:40.511 INFO:teuthology.orchestra.run.smithi116.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi162.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-19T00:51:40.539 INFO:teuthology.orchestra.run.smithi162.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi162.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 634 kB/s | 32 kB 00:00 2024-03-19T00:51:40.648 INFO:teuthology.orchestra.run.smithi162.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-19T00:51:40.673 INFO:teuthology.orchestra.run.smithi162.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 160 kB/s | 17 kB 00:00 2024-03-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi162.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-19T00:51:40.724 INFO:teuthology.orchestra.run.smithi162.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi162.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-19T00:51:40.774 INFO:teuthology.orchestra.run.smithi162.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-19T00:51:40.800 INFO:teuthology.orchestra.run.smithi162.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-19T00:51:40.816 INFO:teuthology.orchestra.run.smithi162.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-19T00:51:40.842 INFO:teuthology.orchestra.run.smithi162.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-19T00:51:40.867 INFO:teuthology.orchestra.run.smithi162.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-19T00:51:40.892 INFO:teuthology.orchestra.run.smithi162.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi162.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 685 kB/s | 34 kB 00:00 2024-03-19T00:51:41.061 INFO:teuthology.orchestra.run.smithi116.stdout:(152/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 3.4 MB/s | 3.1 MB 00:00 2024-03-19T00:51:41.118 INFO:teuthology.orchestra.run.smithi162.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-03-19T00:51:41.160 INFO:teuthology.orchestra.run.smithi162.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 6.3 MB/s | 1.5 MB 00:00 2024-03-19T00:51:41.194 INFO:teuthology.orchestra.run.smithi162.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-19T00:51:41.219 INFO:teuthology.orchestra.run.smithi162.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-19T00:51:41.386 INFO:teuthology.orchestra.run.smithi162.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-19T00:51:41.420 INFO:teuthology.orchestra.run.smithi162.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.1 MB/s | 1.0 MB 00:00 2024-03-19T00:51:41.444 INFO:teuthology.orchestra.run.smithi162.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-19T00:51:41.537 INFO:teuthology.orchestra.run.smithi116.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 2.2 MB/s | 2.3 MB 00:01 2024-03-19T00:51:41.578 INFO:teuthology.orchestra.run.smithi116.stdout:(154/174): thrift-0.15.0-2.el9.x86_64.rpm 957 kB/s | 1.6 MB 00:01 2024-03-19T00:51:41.595 INFO:teuthology.orchestra.run.smithi162.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 33 MB/s | 5.8 MB 00:00 2024-03-19T00:51:41.620 INFO:teuthology.orchestra.run.smithi162.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 140 kB/s | 24 kB 00:00 2024-03-19T00:51:41.645 INFO:teuthology.orchestra.run.smithi116.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 388 kB/s | 42 kB 00:00 2024-03-19T00:51:41.646 INFO:teuthology.orchestra.run.smithi162.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 360 kB/s | 18 kB 00:00 2024-03-19T00:51:41.671 INFO:teuthology.orchestra.run.smithi162.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-19T00:51:41.696 INFO:teuthology.orchestra.run.smithi162.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 635 kB/s | 32 kB 00:00 2024-03-19T00:51:41.754 INFO:teuthology.orchestra.run.smithi116.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 659 kB/s | 71 kB 00:00 2024-03-19T00:51:41.805 INFO:teuthology.orchestra.run.smithi162.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-19T00:51:41.839 INFO:teuthology.orchestra.run.smithi162.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-19T00:51:41.862 INFO:teuthology.orchestra.run.smithi116.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 336 kB/s | 36 kB 00:00 2024-03-19T00:51:41.864 INFO:teuthology.orchestra.run.smithi162.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-19T00:51:41.889 INFO:teuthology.orchestra.run.smithi162.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-19T00:51:41.915 INFO:teuthology.orchestra.run.smithi162.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-19T00:51:41.940 INFO:teuthology.orchestra.run.smithi162.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-03-19T00:51:41.965 INFO:teuthology.orchestra.run.smithi162.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-19T00:51:41.971 INFO:teuthology.orchestra.run.smithi116.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.7 MB/s | 192 kB 00:00 2024-03-19T00:51:41.990 INFO:teuthology.orchestra.run.smithi162.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-19T00:51:42.013 INFO:teuthology.orchestra.run.smithi116.stdout:(159/174): NetworkManager-libnm-1.47.2-1.el9.x8 1.9 MB/s | 1.8 MB 00:00 2024-03-19T00:51:42.015 INFO:teuthology.orchestra.run.smithi162.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-03-19T00:51:42.040 INFO:teuthology.orchestra.run.smithi162.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2024-03-19T00:51:42.080 INFO:teuthology.orchestra.run.smithi116.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-19T00:51:42.122 INFO:teuthology.orchestra.run.smithi116.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 212 kB/s | 23 kB 00:00 2024-03-19T00:51:42.175 INFO:teuthology.orchestra.run.smithi162.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 293 kB/s | 39 kB 00:00 2024-03-19T00:51:42.197 INFO:teuthology.orchestra.run.smithi116.stdout:(162/174): NetworkManager-tui-1.47.2-1.el9.x86_ 402 kB/s | 248 kB 00:00 2024-03-19T00:51:42.222 INFO:teuthology.orchestra.run.smithi116.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 763 kB/s | 108 kB 00:00 2024-03-19T00:51:42.248 INFO:teuthology.orchestra.run.smithi116.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 155 kB 00:00 2024-03-19T00:51:42.323 INFO:teuthology.orchestra.run.smithi116.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 636 kB/s | 64 kB 00:00 2024-03-19T00:51:42.348 INFO:teuthology.orchestra.run.smithi116.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 286 kB/s | 29 kB 00:00 2024-03-19T00:51:42.366 INFO:teuthology.orchestra.run.smithi162.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-19T00:51:42.382 INFO:teuthology.orchestra.run.smithi116.stdout:(167/174): libmount-2.37.4-18.el9.x86_64.rpm 736 kB/s | 136 kB 00:00 2024-03-19T00:51:42.391 INFO:teuthology.orchestra.run.smithi162.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 191 kB/s | 41 kB 00:00 2024-03-19T00:51:42.417 INFO:teuthology.orchestra.run.smithi162.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-19T00:51:42.442 INFO:teuthology.orchestra.run.smithi162.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-19T00:51:42.457 INFO:teuthology.orchestra.run.smithi116.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 9.1 MB/s | 1.2 MB 00:00 2024-03-19T00:51:42.467 INFO:teuthology.orchestra.run.smithi162.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-19T00:51:42.493 INFO:teuthology.orchestra.run.smithi162.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 877 kB/s | 44 kB 00:00 2024-03-19T00:51:42.518 INFO:teuthology.orchestra.run.smithi162.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-19T00:51:42.543 INFO:teuthology.orchestra.run.smithi162.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-19T00:51:42.566 INFO:teuthology.orchestra.run.smithi116.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-03-19T00:51:42.577 INFO:teuthology.orchestra.run.smithi162.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-19T00:51:42.602 INFO:teuthology.orchestra.run.smithi162.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-19T00:51:42.627 INFO:teuthology.orchestra.run.smithi162.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-19T00:51:42.632 INFO:teuthology.orchestra.run.smithi116.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.6 MB/s | 2.2 MB 00:00 2024-03-19T00:51:42.653 INFO:teuthology.orchestra.run.smithi162.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-19T00:51:42.666 INFO:teuthology.orchestra.run.smithi116.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 310 kB/s | 31 kB 00:00 2024-03-19T00:51:42.750 INFO:teuthology.orchestra.run.smithi116.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 893 kB/s | 74 kB 00:00 2024-03-19T00:51:42.803 INFO:teuthology.orchestra.run.smithi162.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-19T00:51:42.850 INFO:teuthology.orchestra.run.smithi116.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.9 MB/s | 2.3 MB 00:00 2024-03-19T00:51:42.887 INFO:teuthology.orchestra.run.smithi162.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi162.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-03-19T00:51:42.937 INFO:teuthology.orchestra.run.smithi162.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-19T00:51:42.988 INFO:teuthology.orchestra.run.smithi162.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 285 kB/s | 21 kB 00:00 2024-03-19T00:51:43.004 INFO:teuthology.orchestra.run.smithi162.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 380 kB/s | 25 kB 00:00 2024-03-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi162.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-19T00:51:43.155 INFO:teuthology.orchestra.run.smithi162.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-03-19T00:51:43.222 INFO:teuthology.orchestra.run.smithi162.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 18 MB/s | 4.1 MB 00:00 2024-03-19T00:51:43.250 INFO:teuthology.orchestra.run.smithi162.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.3 MB/s | 216 kB 00:00 2024-03-19T00:51:43.264 INFO:teuthology.orchestra.run.smithi162.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 837 kB/s | 35 kB 00:00 2024-03-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi162.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi162.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-19T00:51:43.332 INFO:teuthology.orchestra.run.smithi162.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 531 kB/s | 22 kB 00:00 2024-03-19T00:51:43.508 INFO:teuthology.orchestra.run.smithi162.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 3.1 MB/s | 548 kB 00:00 2024-03-19T00:51:43.541 INFO:teuthology.orchestra.run.smithi162.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.7 MB/s | 863 kB 00:00 2024-03-19T00:51:43.574 INFO:teuthology.orchestra.run.smithi162.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 4.5 MB/s | 309 kB 00:00 2024-03-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi162.stdout:(108/174): ceph-test-19.0.0-2206.gb6ae2491.el9. 7.7 MB/s | 48 MB 00:06 2024-03-19T00:51:44.016 INFO:teuthology.orchestra.run.smithi162.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 41 kB/s | 19 kB 00:00 2024-03-19T00:51:44.062 INFO:teuthology.orchestra.run.smithi116.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 769 kB/s | 1.1 MB 00:01 2024-03-19T00:51:44.067 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-19T00:51:44.067 INFO:teuthology.orchestra.run.smithi116.stdout:Total 9.7 MB/s | 224 MB 00:23 2024-03-19T00:51:44.075 INFO:teuthology.orchestra.run.smithi162.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-03-19T00:51:44.092 INFO:teuthology.orchestra.run.smithi162.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 644 kB/s | 49 kB 00:00 2024-03-19T00:51:44.134 INFO:teuthology.orchestra.run.smithi162.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-19T00:51:44.168 INFO:teuthology.orchestra.run.smithi162.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-19T00:51:44.268 INFO:teuthology.orchestra.run.smithi162.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 6.4 MB/s | 4.4 MB 00:00 2024-03-19T00:51:44.310 INFO:teuthology.orchestra.run.smithi162.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-19T00:51:44.335 INFO:teuthology.orchestra.run.smithi162.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 650 kB/s | 43 kB 00:00 2024-03-19T00:51:44.361 INFO:teuthology.orchestra.run.smithi162.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 641 kB/s | 32 kB 00:00 2024-03-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi162.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-03-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi162.stdout:(119/174): python3-asyncssh-2.13.2-4.el9.noarch 2.2 MB/s | 548 kB 00:00 2024-03-19T00:51:44.437 INFO:teuthology.orchestra.run.smithi162.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-19T00:51:44.470 INFO:teuthology.orchestra.run.smithi162.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-03-19T00:51:44.495 INFO:teuthology.orchestra.run.smithi162.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 603 kB/s | 35 kB 00:00 2024-03-19T00:51:44.520 INFO:teuthology.orchestra.run.smithi162.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-19T00:51:44.545 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-03-19T00:51:44.545 INFO:teuthology.orchestra.run.smithi162.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-19T00:51:44.587 INFO:teuthology.orchestra.run.smithi162.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-19T00:51:44.613 INFO:teuthology.orchestra.run.smithi162.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 159 kB/s | 11 kB 00:00 2024-03-19T00:51:44.647 INFO:teuthology.orchestra.run.smithi162.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 300 kB/s | 18 kB 00:00 2024-03-19T00:51:44.672 INFO:teuthology.orchestra.run.smithi162.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 395 kB/s | 23 kB 00:00 2024-03-19T00:51:44.697 INFO:teuthology.orchestra.run.smithi162.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-03-19T00:51:44.714 INFO:teuthology.orchestra.run.smithi162.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 470 kB/s | 20 kB 00:00 2024-03-19T00:51:44.756 INFO:teuthology.orchestra.run.smithi162.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-03-19T00:51:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-03-19T00:51:44.820 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-03-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi162.stdout:(132/174): python3-grpcio-1.46.7-10.el9.x86_64. 6.2 MB/s | 2.0 MB 00:00 2024-03-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi162.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 943 kB/s | 79 kB 00:00 2024-03-19T00:51:44.865 INFO:teuthology.orchestra.run.smithi162.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.3 MB/s | 58 kB 00:00 2024-03-19T00:51:44.891 INFO:teuthology.orchestra.run.smithi162.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-19T00:51:44.908 INFO:teuthology.orchestra.run.smithi162.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 393 kB/s | 16 kB 00:00 2024-03-19T00:51:44.933 INFO:teuthology.orchestra.run.smithi162.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-03-19T00:51:44.950 INFO:teuthology.orchestra.run.smithi162.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 676 kB/s | 28 kB 00:00 2024-03-19T00:51:45.000 INFO:teuthology.orchestra.run.smithi162.stdout:(139/174): python3-kubernetes-26.1.0-2.el9.noar 3.4 MB/s | 1.0 MB 00:00 2024-03-19T00:51:45.017 INFO:teuthology.orchestra.run.smithi162.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-03-19T00:51:45.042 INFO:teuthology.orchestra.run.smithi162.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-03-19T00:51:45.059 INFO:teuthology.orchestra.run.smithi162.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 856 kB/s | 36 kB 00:00 2024-03-19T00:51:45.076 INFO:teuthology.orchestra.run.smithi162.stdout:(143/174): python3-rsa-4.9-2.el9.noarch.rpm 780 kB/s | 59 kB 00:00 2024-03-19T00:51:45.093 INFO:teuthology.orchestra.run.smithi162.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-19T00:51:45.119 INFO:teuthology.orchestra.run.smithi162.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-03-19T00:51:45.144 INFO:teuthology.orchestra.run.smithi162.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-19T00:51:45.161 INFO:teuthology.orchestra.run.smithi162.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 476 kB/s | 20 kB 00:00 2024-03-19T00:51:45.186 INFO:teuthology.orchestra.run.smithi162.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-03-19T00:51:45.228 INFO:teuthology.orchestra.run.smithi162.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-19T00:51:45.386 INFO:teuthology.orchestra.run.smithi162.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-03-19T00:51:45.428 INFO:teuthology.orchestra.run.smithi162.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-19T00:51:45.545 INFO:teuthology.orchestra.run.smithi162.stdout:(152/174): librados2-19.0.0-2206.gb6ae2491.el9. 9.5 MB/s | 3.4 MB 00:00 2024-03-19T00:51:45.779 INFO:teuthology.orchestra.run.smithi162.stdout:(153/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 5.7 MB/s | 3.1 MB 00:00 2024-03-19T00:51:45.887 INFO:teuthology.orchestra.run.smithi162.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 389 kB/s | 42 kB 00:00 2024-03-19T00:51:45.938 INFO:teuthology.orchestra.run.smithi162.stdout:(155/174): NetworkManager-1.47.2-1.el9.x86_64.r 4.5 MB/s | 2.3 MB 00:00 2024-03-19T00:51:45.980 INFO:teuthology.orchestra.run.smithi162.stdout:(156/174): NetworkManager-libnm-1.47.2-1.el9.x8 4.2 MB/s | 1.8 MB 00:00 2024-03-19T00:51:46.007 INFO:teuthology.orchestra.run.smithi162.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-03-19T00:51:46.032 INFO:teuthology.orchestra.run.smithi162.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 696 kB/s | 36 kB 00:00 2024-03-19T00:51:46.058 INFO:teuthology.orchestra.run.smithi162.stdout:(159/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.4 MB/s | 248 kB 00:00 2024-03-19T00:51:46.083 INFO:teuthology.orchestra.run.smithi162.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.5 MB/s | 192 kB 00:00 2024-03-19T00:51:46.100 INFO:teuthology.orchestra.run.smithi162.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 551 kB/s | 23 kB 00:00 2024-03-19T00:51:46.142 INFO:teuthology.orchestra.run.smithi162.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-19T00:51:46.167 INFO:teuthology.orchestra.run.smithi162.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-19T00:51:46.192 INFO:teuthology.orchestra.run.smithi162.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 155 kB 00:00 2024-03-19T00:51:46.218 INFO:teuthology.orchestra.run.smithi162.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-19T00:51:46.235 INFO:teuthology.orchestra.run.smithi162.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 951 kB/s | 64 kB 00:00 2024-03-19T00:51:46.251 INFO:teuthology.orchestra.run.smithi162.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 488 kB/s | 29 kB 00:00 2024-03-19T00:51:46.435 INFO:teuthology.orchestra.run.smithi162.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 5.6 MB/s | 1.2 MB 00:00 2024-03-19T00:51:46.519 INFO:teuthology.orchestra.run.smithi162.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 5.5 MB/s | 465 kB 00:00 2024-03-19T00:51:46.577 INFO:teuthology.orchestra.run.smithi162.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.3 MB/s | 2.2 MB 00:00 2024-03-19T00:51:46.594 INFO:teuthology.orchestra.run.smithi162.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 414 kB/s | 31 kB 00:00 2024-03-19T00:51:46.695 INFO:teuthology.orchestra.run.smithi162.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 5.2 MB/s | 2.3 MB 00:00 2024-03-19T00:51:46.720 INFO:teuthology.orchestra.run.smithi162.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 594 kB/s | 74 kB 00:00 2024-03-19T00:51:46.787 INFO:teuthology.orchestra.run.smithi162.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.1 MB/s | 1.1 MB 00:00 2024-03-19T00:51:46.791 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-19T00:51:46.791 INFO:teuthology.orchestra.run.smithi162.stdout:Total 20 MB/s | 224 MB 00:11 2024-03-19T00:51:46.904 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-03-19T00:51:46.905 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-03-19T00:51:47.259 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-19T00:51:47.531 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-19T00:51:47.531 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-19T00:51:49.243 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-19T00:51:49.243 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-19T00:51:53.890 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-03-19T00:51:54.096 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T00:51:54.248 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T00:51:54.286 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T00:51:54.298 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T00:51:54.355 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T00:51:54.411 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T00:51:54.448 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T00:51:54.473 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T00:51:54.525 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T00:51:54.854 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T00:51:54.907 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T00:51:55.028 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T00:51:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T00:51:55.136 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T00:51:55.174 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T00:51:55.217 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T00:51:55.415 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T00:51:55.471 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T00:51:55.498 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T00:51:55.633 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T00:51:55.666 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T00:51:55.714 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T00:51:55.753 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T00:51:55.776 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T00:51:55.792 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T00:51:55.976 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T00:51:56.007 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T00:51:56.047 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T00:51:56.210 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T00:51:56.253 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-19T00:51:56.332 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T00:51:56.348 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T00:51:56.382 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T00:51:56.384 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T00:51:56.422 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T00:51:56.431 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T00:51:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T00:51:56.490 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T00:51:56.525 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T00:51:56.535 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T00:51:56.559 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T00:51:56.593 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T00:51:56.635 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T00:51:56.752 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T00:51:56.858 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T00:51:56.956 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T00:51:56.959 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T00:51:56.991 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T00:51:57.010 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T00:51:57.017 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T00:51:57.061 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T00:51:57.115 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T00:51:57.134 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T00:51:57.145 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T00:51:57.196 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T00:51:57.228 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T00:51:57.235 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T00:51:57.261 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T00:51:57.298 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T00:51:57.302 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T00:51:57.335 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T00:51:57.377 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T00:51:57.411 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T00:51:57.439 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T00:51:57.484 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T00:51:57.507 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T00:51:57.539 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T00:51:57.552 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T00:51:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T00:51:57.595 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T00:51:57.616 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T00:51:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T00:51:57.684 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T00:51:57.693 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T00:51:57.730 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T00:51:57.757 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T00:51:57.803 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T00:51:57.805 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T00:51:57.828 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T00:51:57.844 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T00:51:57.937 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T00:51:57.990 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T00:51:58.045 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T00:51:58.085 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T00:51:58.117 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T00:51:58.214 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T00:51:58.319 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T00:51:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T00:51:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T00:51:58.509 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T00:51:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T00:51:58.594 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T00:51:58.677 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T00:51:58.720 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T00:51:58.731 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T00:51:58.739 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T00:51:58.774 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T00:51:58.811 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T00:51:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T00:51:58.849 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T00:51:58.880 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T00:51:58.923 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T00:51:58.952 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T00:51:58.979 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T00:51:59.014 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T00:51:59.089 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T00:51:59.151 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T00:51:59.189 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T00:51:59.253 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T00:51:59.296 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T00:51:59.339 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T00:51:59.345 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T00:51:59.372 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T00:51:59.400 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T00:51:59.460 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T00:51:59.496 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T00:51:59.531 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T00:51:59.560 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T00:51:59.629 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T00:51:59.675 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T00:51:59.748 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T00:51:59.749 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T00:51:59.815 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T00:51:59.859 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T00:51:59.883 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T00:51:59.936 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T00:51:59.993 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T00:52:00.062 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T00:52:00.115 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T00:52:00.181 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T00:52:00.220 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T00:52:00.257 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T00:52:00.303 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T00:52:00.340 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T00:52:00.382 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T00:52:00.424 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T00:52:00.456 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T00:52:00.540 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T00:52:00.557 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T00:52:00.605 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T00:52:00.615 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T00:52:00.635 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T00:52:00.660 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T00:52:00.668 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T00:52:00.677 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T00:52:00.700 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T00:52:00.741 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T00:52:00.770 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T00:52:00.771 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T00:52:00.800 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T00:52:00.810 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T00:52:00.851 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T00:52:00.876 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T00:52:00.900 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T00:52:00.987 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T00:52:01.022 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T00:52:01.116 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T00:52:01.219 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T00:52:01.224 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T00:52:01.267 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T00:52:01.286 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T00:52:01.315 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T00:52:01.330 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T00:52:01.371 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T00:52:01.465 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T00:52:01.509 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T00:52:01.563 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T00:52:01.564 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T00:52:01.612 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T00:52:01.637 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T00:52:01.676 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T00:52:01.713 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T00:52:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T00:52:01.759 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T00:52:01.778 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T00:52:01.796 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T00:52:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T00:52:01.839 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T00:52:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T00:52:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T00:52:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T00:52:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T00:52:01.957 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T00:52:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T00:52:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T00:52:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T00:52:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T00:52:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T00:52:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T00:52:02.202 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T00:52:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T00:52:02.254 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T00:52:02.284 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T00:52:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T00:52:02.334 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T00:52:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T00:52:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T00:52:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T00:52:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T00:52:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T00:52:02.499 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T00:52:02.525 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T00:52:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T00:52:02.549 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T00:52:02.621 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T00:52:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T00:52:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T00:52:02.743 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T00:52:02.843 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T00:52:02.892 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T00:52:02.941 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T00:52:02.996 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T00:52:03.183 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T00:52:03.230 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T00:52:03.368 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T00:52:03.408 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T00:52:03.434 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T00:52:03.458 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T00:52:03.469 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T00:52:03.499 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T00:52:03.513 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T00:52:03.527 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T00:52:03.562 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T00:52:03.565 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T00:52:03.610 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T00:52:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T00:52:03.643 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T00:52:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T00:52:03.676 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T00:52:03.748 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T00:52:03.790 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T00:52:03.823 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T00:52:03.862 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T00:52:03.865 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T00:52:03.899 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T00:52:03.908 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T00:52:03.986 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T00:52:04.005 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T00:52:04.096 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T00:52:04.156 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T00:52:04.187 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T00:52:04.226 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T00:52:04.324 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T00:52:04.381 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T00:52:04.618 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T00:52:04.729 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T00:52:05.109 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T00:52:05.161 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T00:52:05.195 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T00:52:05.243 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T00:52:05.331 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T00:52:05.363 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T00:52:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T00:52:05.577 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T00:52:05.655 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T00:52:05.945 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T00:52:06.024 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T00:52:06.055 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T00:52:06.086 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T00:52:06.114 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T00:52:06.302 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T00:52:06.409 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T00:52:06.428 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T00:52:06.469 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T00:52:06.560 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T00:52:06.590 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T00:52:06.626 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T00:52:07.085 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T00:52:07.114 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T00:52:07.167 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T00:52:07.190 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T00:52:07.627 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T00:52:07.713 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T00:52:07.744 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T00:52:07.775 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T00:52:07.802 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T00:52:08.118 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T00:52:08.157 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T00:52:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T00:52:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T00:52:08.315 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T00:52:08.779 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T00:52:08.803 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T00:52:08.857 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T00:52:08.879 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T00:52:09.474 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T00:52:10.191 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T00:52:10.191 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T00:52:10.191 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T00:52:10.191 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:52:10.233 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T00:52:11.111 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T00:52:11.143 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T00:52:11.688 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T00:52:11.730 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T00:52:11.794 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T00:52:11.795 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T00:52:11.795 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T00:52:11.795 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:52:11.823 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T00:52:22.365 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T00:52:22.365 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-03-19T00:52:22.365 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-03-19T00:52:22.365 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-03-19T00:52:22.365 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-03-19T00:52:22.365 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-03-19T00:52:22.365 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-03-19T00:52:22.365 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-03-19T00:52:22.365 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:52:22.527 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T00:52:22.542 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T00:52:22.542 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:22.543 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T00:52:22.543 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:52:23.619 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T00:52:23.619 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-03-19T00:52:23.619 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-03-19T00:52:23.619 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-03-19T00:52:23.620 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-03-19T00:52:23.620 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-03-19T00:52:23.620 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2024-03-19T00:52:23.620 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-03-19T00:52:23.620 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:52:23.796 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T00:52:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T00:52:23.812 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:23.812 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T00:52:23.812 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:52:24.040 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T00:52:24.066 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T00:52:24.066 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:24.066 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T00:52:24.066 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T00:52:24.066 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T00:52:24.066 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:52:24.282 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T00:52:24.305 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T00:52:24.305 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:24.306 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T00:52:24.306 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T00:52:24.306 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T00:52:24.306 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:52:24.750 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T00:52:24.772 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T00:52:24.772 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:24.773 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T00:52:24.773 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T00:52:24.773 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T00:52:24.773 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:52:24.860 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T00:52:25.195 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T00:52:25.225 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T00:52:25.257 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T00:52:25.317 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T00:52:25.318 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T00:52:25.343 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T00:52:25.343 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:25.343 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T00:52:25.343 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T00:52:25.344 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T00:52:25.344 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:52:25.348 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T00:52:25.560 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T00:52:25.581 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T00:52:25.582 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T00:52:25.582 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:25.582 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T00:52:25.582 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T00:52:25.582 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T00:52:25.583 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:52:25.607 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T00:52:26.028 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T00:52:26.050 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T00:52:26.050 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:26.050 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T00:52:26.050 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T00:52:26.051 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T00:52:26.051 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:52:26.146 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T00:52:26.548 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T00:52:26.569 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T00:52:26.578 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T00:52:26.599 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T00:52:26.610 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T00:52:26.671 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T00:52:26.699 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T00:52:26.710 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T00:52:26.894 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T00:52:26.934 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T00:52:26.944 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T00:52:26.957 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T00:52:26.957 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:26.957 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T00:52:26.957 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T00:52:26.958 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T00:52:26.958 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:52:26.977 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T00:52:27.001 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T00:52:27.013 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T00:52:27.060 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T00:52:27.925 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T00:52:27.945 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T00:52:27.953 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T00:52:27.969 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T00:52:27.976 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T00:52:27.977 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:27.977 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T00:52:27.977 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T00:52:27.977 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T00:52:27.977 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:52:28.023 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T00:52:28.044 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T00:52:28.044 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:28.045 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T00:52:28.045 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:52:28.070 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T00:52:28.090 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T00:52:28.112 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T00:52:28.112 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:28.112 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T00:52:28.112 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T00:52:28.112 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T00:52:28.112 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:52:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T00:52:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T00:52:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T00:52:28.296 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:28.296 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T00:52:28.296 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T00:52:28.296 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T00:52:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:52:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T00:52:28.350 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T00:52:28.398 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T00:52:28.424 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T00:52:28.447 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T00:52:28.447 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:28.447 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T00:52:28.447 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T00:52:28.448 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T00:52:28.448 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:52:29.270 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T00:52:29.302 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T00:52:29.324 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T00:52:29.324 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:29.325 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T00:52:29.325 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T00:52:29.325 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T00:52:29.325 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:52:29.369 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T00:52:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T00:52:29.391 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:29.391 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T00:52:29.391 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:52:29.438 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T00:52:29.458 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T00:52:29.459 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:29.459 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T00:52:29.459 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T00:52:29.459 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T00:52:29.459 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:52:29.770 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T00:52:29.793 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T00:52:29.793 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T00:52:29.793 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T00:52:29.793 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T00:52:29.793 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T00:52:29.794 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:52:32.920 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T00:52:32.961 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T00:52:32.991 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T00:52:33.018 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T00:52:33.204 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T00:52:33.319 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T00:52:33.358 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T00:52:33.417 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T00:52:33.475 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T00:52:33.510 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T00:52:33.551 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T00:52:33.592 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T00:52:33.623 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T00:52:33.683 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T00:52:33.770 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T00:52:33.811 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T00:52:34.233 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T00:52:34.274 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T00:52:34.305 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T00:52:34.333 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T00:52:34.417 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T00:52:34.459 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T00:52:34.480 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T00:52:34.513 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T00:52:34.641 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T00:52:34.689 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T00:52:34.729 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T00:52:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T00:52:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T00:52:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T00:52:34.908 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T00:52:34.936 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T00:52:34.990 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T00:52:35.051 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T00:52:35.085 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T00:52:35.298 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T00:52:35.371 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T00:52:35.412 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T00:52:35.502 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T00:52:35.541 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T00:52:35.566 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T00:52:35.592 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T00:52:35.611 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T00:52:35.618 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T00:52:35.618 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T00:52:35.650 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T00:52:35.670 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T00:52:35.689 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T00:52:35.719 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T00:52:35.719 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T00:52:35.795 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T00:52:35.822 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T00:52:35.854 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T00:52:35.880 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T00:52:35.905 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T00:52:35.940 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T00:52:35.986 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T00:52:36.026 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T00:52:36.051 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T00:52:36.077 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T00:52:36.102 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T00:52:36.127 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T00:52:36.134 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T00:52:36.134 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T00:52:36.322 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T00:52:36.322 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T00:52:36.405 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T00:52:36.429 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T00:52:36.455 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T00:52:36.487 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T00:52:36.492 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T00:52:36.514 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T00:52:36.580 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T00:52:36.580 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T00:52:36.620 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T00:52:36.700 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T00:52:36.742 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T00:52:36.767 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T00:52:36.793 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T00:52:36.819 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T00:52:36.819 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T00:52:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T00:52:36.920 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T00:52:36.920 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T00:52:36.988 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T00:52:37.014 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T00:52:37.047 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T00:52:37.073 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T00:52:37.098 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T00:52:37.130 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T00:52:37.179 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T00:52:37.219 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T00:52:37.245 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T00:52:37.270 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T00:52:37.295 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T00:52:37.321 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T00:52:37.327 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T00:52:37.328 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T00:52:37.516 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T00:52:37.516 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T00:52:37.590 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T00:52:37.615 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T00:52:37.640 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T00:52:37.674 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T00:52:37.700 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T00:52:37.764 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T00:52:41.367 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T00:52:41.367 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T00:52:41.367 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T00:52:41.367 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T00:52:41.367 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T00:52:41.367 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T00:52:41.367 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T00:52:41.368 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T00:52:41.369 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T00:52:41.370 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T00:52:41.370 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T00:52:41.370 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T00:52:41.370 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T00:52:41.370 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T00:52:41.370 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T00:52:41.370 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T00:52:41.370 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T00:52:41.370 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T00:52:41.370 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T00:52:41.370 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T00:52:41.370 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T00:52:41.370 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T00:52:41.371 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T00:52:41.372 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T00:52:41.373 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T00:52:41.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T00:52:41.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T00:52:41.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T00:52:41.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T00:52:41.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T00:52:41.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T00:52:41.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T00:52:41.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T00:52:41.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T00:52:41.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T00:52:41.374 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T00:52:41.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T00:52:41.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T00:52:41.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T00:52:41.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T00:52:41.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T00:52:41.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T00:52:41.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T00:52:41.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T00:52:41.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T00:52:41.378 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T00:52:41.379 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T00:52:41.380 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T00:52:41.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T00:52:41.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T00:52:41.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T00:52:41.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T00:52:41.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T00:52:41.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T00:52:41.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T00:52:41.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T00:52:41.381 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T00:52:41.382 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T00:52:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T00:52:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T00:52:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T00:52:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T00:52:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T00:52:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T00:52:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T00:52:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T00:52:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T00:52:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T00:52:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T00:52:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T00:52:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T00:52:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T00:52:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T00:52:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T00:52:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T00:52:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T00:52:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T00:52:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T00:52:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T00:52:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T00:52:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T00:52:41.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T00:52:41.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T00:52:41.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T00:52:41.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T00:52:41.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T00:52:41.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T00:52:41.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T00:52:41.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T00:52:41.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T00:52:41.385 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T00:52:41.386 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T00:52:41.386 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T00:52:41.386 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T00:52:42.839 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T00:52:42.839 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T00:52:42.839 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T00:52:42.839 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T00:52:42.839 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T00:52:42.839 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T00:52:42.839 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T00:52:42.839 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T00:52:42.839 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T00:52:42.839 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T00:52:42.840 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T00:52:42.841 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T00:52:42.842 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T00:52:42.842 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T00:52:42.842 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T00:52:42.842 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T00:52:42.842 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T00:52:42.842 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T00:52:42.842 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T00:52:42.842 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T00:52:42.842 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T00:52:42.842 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T00:52:42.842 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T00:52:42.842 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T00:52:42.843 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T00:52:42.844 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T00:52:42.845 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T00:52:42.846 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T00:52:42.847 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T00:52:42.848 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T00:52:42.848 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T00:52:42.848 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T00:52:42.848 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T00:52:42.848 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T00:52:42.848 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T00:52:42.848 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T00:52:42.848 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T00:52:42.848 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T00:52:42.848 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T00:52:42.848 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T00:52:42.848 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T00:52:42.848 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T00:52:42.849 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T00:52:42.850 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T00:52:42.851 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T00:52:42.852 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T00:52:42.853 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T00:52:42.853 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T00:52:42.853 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T00:52:42.853 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T00:52:42.853 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T00:52:42.853 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T00:52:42.853 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T00:52:42.853 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T00:52:42.853 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T00:52:44.230 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T00:52:44.230 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:52:44.230 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-03-19T00:52:44.230 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T00:52:44.231 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T00:52:44.231 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T00:52:44.231 INFO:teuthology.orchestra.run.smithi116.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T00:52:44.231 INFO:teuthology.orchestra.run.smithi116.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T00:52:44.231 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T00:52:44.231 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T00:52:44.231 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T00:52:44.231 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T00:52:44.231 INFO:teuthology.orchestra.run.smithi116.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T00:52:44.231 INFO:teuthology.orchestra.run.smithi116.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T00:52:44.231 INFO:teuthology.orchestra.run.smithi116.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T00:52:44.231 INFO:teuthology.orchestra.run.smithi116.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T00:52:44.232 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.232 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.232 INFO:teuthology.orchestra.run.smithi116.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T00:52:44.232 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T00:52:44.232 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T00:52:44.232 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T00:52:44.232 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T00:52:44.232 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T00:52:44.232 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T00:52:44.232 INFO:teuthology.orchestra.run.smithi116.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T00:52:44.233 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T00:52:44.233 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-03-19T00:52:44.233 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T00:52:44.233 INFO:teuthology.orchestra.run.smithi116.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T00:52:44.233 INFO:teuthology.orchestra.run.smithi116.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T00:52:44.233 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T00:52:44.233 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.233 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.233 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.233 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.233 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:44.234 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.234 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.234 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.234 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:44.234 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:44.234 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:44.235 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:44.235 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:44.235 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.235 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.235 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:44.235 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.235 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.235 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.235 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:44.235 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:44.235 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.235 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.235 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:44.236 INFO:teuthology.orchestra.run.smithi116.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T00:52:44.236 INFO:teuthology.orchestra.run.smithi116.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T00:52:44.236 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T00:52:44.236 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T00:52:44.236 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T00:52:44.236 INFO:teuthology.orchestra.run.smithi116.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T00:52:44.236 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T00:52:44.236 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T00:52:44.236 INFO:teuthology.orchestra.run.smithi116.stdout: jq-1.6-16.el9.x86_64 2024-03-19T00:52:44.236 INFO:teuthology.orchestra.run.smithi116.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T00:52:44.237 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T00:52:44.237 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T00:52:44.237 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.237 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.237 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.237 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T00:52:44.237 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T00:52:44.237 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T00:52:44.237 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T00:52:44.237 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T00:52:44.237 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.237 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.238 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T00:52:44.238 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.238 INFO:teuthology.orchestra.run.smithi116.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T00:52:44.238 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T00:52:44.238 INFO:teuthology.orchestra.run.smithi116.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T00:52:44.238 INFO:teuthology.orchestra.run.smithi116.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T00:52:44.238 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T00:52:44.238 INFO:teuthology.orchestra.run.smithi116.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T00:52:44.238 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T00:52:44.238 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T00:52:44.238 INFO:teuthology.orchestra.run.smithi116.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T00:52:44.239 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T00:52:44.239 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T00:52:44.239 INFO:teuthology.orchestra.run.smithi116.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T00:52:44.239 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T00:52:44.239 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T00:52:44.239 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T00:52:44.239 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T00:52:44.239 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T00:52:44.240 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T00:52:44.241 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T00:52:44.241 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T00:52:44.241 INFO:teuthology.orchestra.run.smithi116.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T00:52:44.241 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T00:52:44.242 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T00:52:44.242 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T00:52:44.242 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.242 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.242 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.242 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T00:52:44.242 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T00:52:44.242 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T00:52:44.242 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T00:52:44.242 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T00:52:44.243 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T00:52:44.244 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.245 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T00:52:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T00:52:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T00:52:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T00:52:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T00:52:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T00:52:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T00:52:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T00:52:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T00:52:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T00:52:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T00:52:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T00:52:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T00:52:44.246 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T00:52:44.247 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T00:52:44.247 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.247 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.247 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:44.247 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T00:52:44.247 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T00:52:44.247 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T00:52:44.247 INFO:teuthology.orchestra.run.smithi116.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T00:52:44.247 INFO:teuthology.orchestra.run.smithi116.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T00:52:44.247 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T00:52:44.248 INFO:teuthology.orchestra.run.smithi116.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T00:52:44.248 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T00:52:44.248 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-03-19T00:52:44.722 DEBUG:teuthology.parallel:result is None 2024-03-19T00:52:45.881 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T00:52:45.881 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:52:45.882 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-03-19T00:52:45.882 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T00:52:45.882 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T00:52:45.882 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T00:52:45.882 INFO:teuthology.orchestra.run.smithi162.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T00:52:45.882 INFO:teuthology.orchestra.run.smithi162.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T00:52:45.882 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T00:52:45.882 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T00:52:45.882 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T00:52:45.882 INFO:teuthology.orchestra.run.smithi162.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T00:52:45.882 INFO:teuthology.orchestra.run.smithi162.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T00:52:45.882 INFO:teuthology.orchestra.run.smithi162.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T00:52:45.882 INFO:teuthology.orchestra.run.smithi162.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T00:52:45.883 INFO:teuthology.orchestra.run.smithi162.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T00:52:45.883 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.883 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.883 INFO:teuthology.orchestra.run.smithi162.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T00:52:45.883 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T00:52:45.883 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T00:52:45.883 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T00:52:45.883 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T00:52:45.883 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T00:52:45.883 INFO:teuthology.orchestra.run.smithi162.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T00:52:45.883 INFO:teuthology.orchestra.run.smithi162.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T00:52:45.883 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T00:52:45.883 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-19T00:52:45.884 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T00:52:45.884 INFO:teuthology.orchestra.run.smithi162.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T00:52:45.884 INFO:teuthology.orchestra.run.smithi162.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T00:52:45.884 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T00:52:45.884 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.884 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.884 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.884 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.884 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:45.884 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.884 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.884 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.884 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:45.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:45.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:45.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:45.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:45.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:45.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.885 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:45.885 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:45.885 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.886 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.886 INFO:teuthology.orchestra.run.smithi162.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T00:52:45.886 INFO:teuthology.orchestra.run.smithi162.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T00:52:45.886 INFO:teuthology.orchestra.run.smithi162.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T00:52:45.886 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T00:52:45.886 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T00:52:45.886 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T00:52:45.886 INFO:teuthology.orchestra.run.smithi162.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T00:52:45.886 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T00:52:45.886 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T00:52:45.886 INFO:teuthology.orchestra.run.smithi162.stdout: jq-1.6-16.el9.x86_64 2024-03-19T00:52:45.886 INFO:teuthology.orchestra.run.smithi162.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T00:52:45.886 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T00:52:45.887 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T00:52:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T00:52:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T00:52:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T00:52:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T00:52:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T00:52:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T00:52:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T00:52:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T00:52:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T00:52:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T00:52:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T00:52:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T00:52:45.888 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T00:52:45.889 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T00:52:45.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T00:52:45.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T00:52:45.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T00:52:45.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T00:52:45.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T00:52:45.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T00:52:45.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T00:52:45.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T00:52:45.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T00:52:45.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T00:52:45.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T00:52:45.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T00:52:45.890 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T00:52:45.891 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.892 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T00:52:45.893 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T00:52:45.893 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T00:52:45.893 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.893 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T00:52:45.893 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T00:52:45.893 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T00:52:45.893 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T00:52:45.893 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T00:52:45.893 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T00:52:45.893 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T00:52:45.893 INFO:teuthology.orchestra.run.smithi162.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T00:52:45.893 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T00:52:45.893 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T00:52:45.894 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T00:52:45.895 INFO:teuthology.orchestra.run.smithi162.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T00:52:45.895 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T00:52:45.895 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-19T00:52:46.346 DEBUG:teuthology.parallel:result is None 2024-03-19T00:53:46.671 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] jq-1.6-16.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/jq-1.6-16.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T00:53:46.672 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T00:53:46.672 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T00:53:47.211 INFO:teuthology.orchestra.run.smithi156.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 kB/s | 195 kB 02:11 2024-03-19T00:53:47.236 INFO:teuthology.orchestra.run.smithi156.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 1.4 kB/s | 187 kB 02:11 2024-03-19T00:53:47.403 INFO:teuthology.orchestra.run.smithi156.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 651 kB/s | 125 kB 00:00 2024-03-19T00:53:47.645 INFO:teuthology.orchestra.run.smithi156.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 6.1 kB/s | 803 kB 02:11 2024-03-19T00:55:57.742 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] elfutils-libelf-devel-0.190-2.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/elfutils-libelf-devel-0.190-2.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T00:55:57.743 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] libuuid-devel-2.37.4-18.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libuuid-devel-2.37.4-18.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T00:55:57.743 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] openssl-devel-3.0.7-27.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/openssl-devel-3.0.7-27.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T00:55:58.100 INFO:teuthology.orchestra.run.smithi156.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 199 B/s | 25 kB 02:10 2024-03-19T00:55:58.126 INFO:teuthology.orchestra.run.smithi156.stdout:(97/174): elfutils-libelf-devel-0.190-2.el9.x86 168 B/s | 21 kB 02:10 2024-03-19T00:55:58.643 INFO:teuthology.orchestra.run.smithi156.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 480 kB/s | 248 kB 00:00 2024-03-19T00:55:58.726 INFO:teuthology.orchestra.run.smithi156.stdout:(99/174): python3-devel-3.9.17-2.el9.x86_64.rpm 392 kB/s | 245 kB 00:00 2024-03-19T00:55:58.835 INFO:teuthology.orchestra.run.smithi156.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.1 MB/s | 216 kB 00:00 2024-03-19T00:55:58.860 INFO:teuthology.orchestra.run.smithi156.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 261 kB/s | 35 kB 00:00 2024-03-19T00:55:58.960 INFO:teuthology.orchestra.run.smithi156.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 355 kB/s | 44 kB 00:00 2024-03-19T00:55:58.986 INFO:teuthology.orchestra.run.smithi156.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 748 kB/s | 94 kB 00:00 2024-03-19T00:55:59.061 INFO:teuthology.orchestra.run.smithi156.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-19T00:55:59.253 INFO:teuthology.orchestra.run.smithi156.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.2 MB/s | 863 kB 00:00 2024-03-19T00:55:59.336 INFO:teuthology.orchestra.run.smithi156.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-03-19T00:55:59.362 INFO:teuthology.orchestra.run.smithi156.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 2.8 MB/s | 309 kB 00:00 2024-03-19T00:55:59.379 INFO:teuthology.orchestra.run.smithi156.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 469 kB/s | 19 kB 00:00 2024-03-19T00:55:59.521 INFO:teuthology.orchestra.run.smithi156.stdout:(109/174): libarrow-9.0.0-10.el9.x86_64.rpm 28 MB/s | 4.4 MB 00:00 2024-03-19T00:55:59.538 INFO:teuthology.orchestra.run.smithi156.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 164 kB/s | 26 kB 00:00 2024-03-19T00:55:59.571 INFO:teuthology.orchestra.run.smithi156.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 967 kB/s | 49 kB 00:00 2024-03-19T00:55:59.597 INFO:teuthology.orchestra.run.smithi156.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-19T00:55:59.622 INFO:teuthology.orchestra.run.smithi156.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-03-19T00:55:59.655 INFO:teuthology.orchestra.run.smithi156.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 14 MB/s | 838 kB 00:00 2024-03-19T00:55:59.689 INFO:teuthology.orchestra.run.smithi156.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 8.1 MB/s | 548 kB 00:00 2024-03-19T00:55:59.715 INFO:teuthology.orchestra.run.smithi156.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 752 kB/s | 43 kB 00:00 2024-03-19T00:55:59.739 INFO:teuthology.orchestra.run.smithi156.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-03-19T00:55:59.765 INFO:teuthology.orchestra.run.smithi156.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-03-19T00:55:59.790 INFO:teuthology.orchestra.run.smithi156.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-03-19T00:55:59.816 INFO:teuthology.orchestra.run.smithi156.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-03-19T00:55:59.841 INFO:teuthology.orchestra.run.smithi156.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-03-19T00:55:59.866 INFO:teuthology.orchestra.run.smithi156.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 707 kB/s | 35 kB 00:00 2024-03-19T00:55:59.892 INFO:teuthology.orchestra.run.smithi156.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 3.9 MB/s | 203 kB 00:00 2024-03-19T00:55:59.959 INFO:teuthology.orchestra.run.smithi156.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 22 MB/s | 2.0 MB 00:00 2024-03-19T00:55:59.976 INFO:teuthology.orchestra.run.smithi156.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-19T00:56:00.009 INFO:teuthology.orchestra.run.smithi156.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-03-19T00:56:00.035 INFO:teuthology.orchestra.run.smithi156.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 302 kB/s | 18 kB 00:00 2024-03-19T00:56:00.060 INFO:teuthology.orchestra.run.smithi156.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-03-19T00:56:00.085 INFO:teuthology.orchestra.run.smithi156.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-03-19T00:56:00.110 INFO:teuthology.orchestra.run.smithi156.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-03-19T00:56:00.152 INFO:teuthology.orchestra.run.smithi156.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-03-19T00:56:00.178 INFO:teuthology.orchestra.run.smithi156.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 698 kB/s | 46 kB 00:00 2024-03-19T00:56:00.203 INFO:teuthology.orchestra.run.smithi156.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-19T00:56:00.228 INFO:teuthology.orchestra.run.smithi156.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-19T00:56:00.254 INFO:teuthology.orchestra.run.smithi156.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-19T00:56:00.279 INFO:teuthology.orchestra.run.smithi156.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 329 kB/s | 16 kB 00:00 2024-03-19T00:56:00.304 INFO:teuthology.orchestra.run.smithi156.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-03-19T00:56:00.330 INFO:teuthology.orchestra.run.smithi156.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 564 kB/s | 28 kB 00:00 2024-03-19T00:56:00.355 INFO:teuthology.orchestra.run.smithi156.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 615 kB/s | 31 kB 00:00 2024-03-19T00:56:00.380 INFO:teuthology.orchestra.run.smithi156.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-03-19T00:56:00.405 INFO:teuthology.orchestra.run.smithi156.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-19T00:56:00.431 INFO:teuthology.orchestra.run.smithi156.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-03-19T00:56:00.456 INFO:teuthology.orchestra.run.smithi156.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-19T00:56:00.482 INFO:teuthology.orchestra.run.smithi156.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-03-19T00:56:00.498 INFO:teuthology.orchestra.run.smithi156.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-03-19T00:56:00.532 INFO:teuthology.orchestra.run.smithi156.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 8.4 MB/s | 427 kB 00:00 2024-03-19T00:56:00.549 INFO:teuthology.orchestra.run.smithi156.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-03-19T00:56:00.583 INFO:teuthology.orchestra.run.smithi156.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-19T00:56:00.624 INFO:teuthology.orchestra.run.smithi156.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-03-19T00:56:01.400 INFO:teuthology.orchestra.run.smithi156.stdout:(150/174): librados2-19.0.0-2206.gb6ae2491.el9. 4.2 MB/s | 3.4 MB 00:00 2024-03-19T00:56:01.542 INFO:teuthology.orchestra.run.smithi156.stdout:(151/174): openssl-devel-3.0.7-27.el9.x86_64.rp 32 kB/s | 4.1 MB 02:13 2024-03-19T00:56:01.567 INFO:teuthology.orchestra.run.smithi156.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.9 MB/s | 321 kB 00:00 2024-03-19T00:56:01.651 INFO:teuthology.orchestra.run.smithi156.stdout:(153/174): librbd1-19.0.0-2206.gb6ae2491.el9.x8 3.1 MB/s | 3.1 MB 00:00 2024-03-19T00:56:01.651 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] NetworkManager-1.47.2-1.el9.x86_64.rpm: Status code: 404 for http://mirror.cpsc.ucalgary.ca/mirror/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.47.2-1.el9.x86_64.rpm (IP: 136.159.208.97) 2024-03-19T00:56:01.651 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] NetworkManager-libnm-1.47.2-1.el9.x86_64.rpm: Status code: 404 for http://mirror.cpsc.ucalgary.ca/mirror/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.47.2-1.el9.x86_64.rpm (IP: 136.159.208.97) 2024-03-19T00:56:01.735 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] NetworkManager-team-1.47.2-1.el9.x86_64.rpm: Status code: 404 for http://mirror.cpsc.ucalgary.ca/mirror/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.47.2-1.el9.x86_64.rpm (IP: 136.159.208.97) 2024-03-19T00:58:12.910 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] NetworkManager-1.47.2-1.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.47.2-1.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-19T00:58:12.911 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] NetworkManager-libnm-1.47.2-1.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-libnm-1.47.2-1.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-19T00:58:12.911 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] NetworkManager-team-1.47.2-1.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-team-1.47.2-1.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 443: Connection timed out] 2024-03-19T00:58:13.015 INFO:teuthology.orchestra.run.smithi156.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 328 B/s | 42 kB 02:11 2024-03-19T00:58:13.098 INFO:teuthology.orchestra.run.smithi156.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-19T00:58:13.140 INFO:teuthology.orchestra.run.smithi156.stdout:(156/174): NetworkManager-libnm-1.47.2-1.el9.x8 14 kB/s | 1.8 MB 02:11 2024-03-19T00:58:13.165 INFO:teuthology.orchestra.run.smithi156.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-03-19T00:58:13.191 INFO:teuthology.orchestra.run.smithi156.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 726 kB/s | 36 kB 00:00 2024-03-19T00:58:13.216 INFO:teuthology.orchestra.run.smithi156.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 192 kB 00:00 2024-03-19T00:58:13.241 INFO:teuthology.orchestra.run.smithi156.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 256 kB 00:00 2024-03-19T00:58:13.258 INFO:teuthology.orchestra.run.smithi156.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 550 kB/s | 23 kB 00:00 2024-03-19T00:58:13.283 INFO:teuthology.orchestra.run.smithi156.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 108 kB 00:00 2024-03-19T00:58:13.308 INFO:teuthology.orchestra.run.smithi156.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-19T00:58:13.334 INFO:teuthology.orchestra.run.smithi156.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-19T00:58:13.351 INFO:teuthology.orchestra.run.smithi156.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.5 MB/s | 64 kB 00:00 2024-03-19T00:58:13.367 INFO:teuthology.orchestra.run.smithi156.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 862 kB/s | 29 kB 00:00 2024-03-19T00:58:13.434 INFO:teuthology.orchestra.run.smithi156.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 14 MB/s | 1.2 MB 00:00 2024-03-19T00:58:13.477 INFO:teuthology.orchestra.run.smithi156.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 20 MB/s | 2.2 MB 00:00 2024-03-19T00:58:13.552 INFO:teuthology.orchestra.run.smithi156.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 20 MB/s | 2.3 MB 00:00 2024-03-19T00:58:13.577 INFO:teuthology.orchestra.run.smithi156.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 4.5 MB/s | 465 kB 00:00 2024-03-19T00:58:13.602 INFO:teuthology.orchestra.run.smithi156.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 621 kB/s | 31 kB 00:00 2024-03-19T00:58:13.636 INFO:teuthology.orchestra.run.smithi156.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-19T00:58:13.962 INFO:teuthology.orchestra.run.smithi156.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 207 kB/s | 74 kB 00:00 2024-03-19T01:00:23.982 INFO:teuthology.orchestra.run.smithi156.stdout:[MIRROR] NetworkManager-1.47.2-1.el9.x86_64.rpm: Curl error (28): Timeout was reached for http://ftp-nyc.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.47.2-1.el9.x86_64.rpm [Failed to connect to ftp-nyc.osuosl.org port 80: Connection timed out] 2024-03-19T01:00:24.425 INFO:teuthology.orchestra.run.smithi156.stdout:(174/174): NetworkManager-1.47.2-1.el9.x86_64.r 9.0 kB/s | 2.3 MB 04:22 2024-03-19T01:00:24.426 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-03-19T01:00:24.426 INFO:teuthology.orchestra.run.smithi156.stdout:Total 428 kB/s | 224 MB 08:56 2024-03-19T01:00:24.882 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-03-19T01:00:25.151 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-03-19T01:00:25.151 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-03-19T01:00:26.882 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-03-19T01:00:26.882 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-03-19T01:00:33.735 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-03-19T01:00:33.862 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-19T01:00:33.888 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-19T01:00:33.919 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T01:00:33.928 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-19T01:00:33.968 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-19T01:00:34.022 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-19T01:00:34.056 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-19T01:00:34.089 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-19T01:00:34.131 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-19T01:00:34.453 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-19T01:00:34.508 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T01:00:34.612 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T01:00:34.650 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T01:00:34.725 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T01:00:34.786 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T01:00:34.974 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-19T01:00:35.153 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-19T01:00:35.202 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-19T01:00:35.231 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T01:00:35.351 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T01:00:35.378 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T01:00:35.425 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 17/198 2024-03-19T01:00:35.465 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-19T01:00:35.488 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T01:00:35.503 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-19T01:00:35.688 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T01:00:35.727 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-19T01:00:35.751 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-19T01:00:35.848 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-19T01:00:35.937 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T01:00:35.989 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-19T01:00:36.060 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T01:00:36.115 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T01:00:36.331 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-19T01:00:36.437 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-19T01:00:36.533 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-19T01:00:36.562 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-19T01:00:36.589 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-19T01:00:36.624 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-19T01:00:36.699 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-19T01:00:36.753 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-19T01:00:36.791 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-19T01:00:36.855 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-19T01:00:36.890 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-19T01:00:36.933 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-19T01:00:36.966 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-19T01:00:36.994 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-19T01:00:37.063 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-19T01:00:37.099 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 39/198 2024-03-19T01:00:37.142 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 40/198 2024-03-19T01:00:37.171 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 41/198 2024-03-19T01:00:37.242 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-19T01:00:37.286 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-19T01:00:37.360 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-19T01:00:37.493 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-19T01:00:37.546 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-19T01:00:38.064 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-19T01:00:38.142 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-19T01:00:38.215 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-19T01:00:38.254 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T01:00:38.261 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-19T01:00:38.294 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-19T01:00:38.335 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-19T01:00:38.373 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 53/198 2024-03-19T01:00:38.402 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-19T01:00:38.616 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-19T01:00:38.797 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-19T01:00:38.872 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-19T01:00:38.916 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-19T01:00:39.045 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-19T01:00:39.094 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-19T01:00:39.150 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-19T01:00:39.223 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-19T01:00:39.262 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-19T01:00:39.298 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-19T01:00:39.345 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-19T01:00:39.381 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-19T01:00:39.424 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-19T01:00:39.467 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-19T01:00:39.601 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-19T01:00:39.647 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-19T01:00:39.677 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-19T01:00:39.719 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-19T01:00:39.813 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-19T01:00:39.860 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-19T01:00:39.926 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-19T01:00:39.951 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-19T01:00:39.976 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-19T01:00:40.048 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-19T01:00:40.170 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-19T01:00:40.272 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-19T01:00:40.309 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-19T01:00:40.359 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-19T01:00:40.414 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-19T01:00:40.598 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-19T01:00:40.640 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-19T01:00:40.776 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-19T01:00:40.821 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-19T01:00:40.853 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-19T01:00:40.884 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-19T01:00:40.920 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-19T01:00:40.955 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-19T01:00:41.002 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-19T01:00:41.033 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-19T01:00:41.063 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-19T01:00:41.139 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-19T01:00:41.183 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-19T01:00:41.216 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-19T01:00:41.257 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-19T01:00:41.292 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-19T01:00:41.381 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-19T01:00:41.482 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-19T01:00:41.523 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-19T01:00:41.563 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-19T01:00:41.610 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-19T01:00:41.707 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-19T01:00:41.765 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-19T01:00:42.494 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-19T01:00:42.545 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-19T01:00:42.579 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-19T01:00:42.627 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-19T01:00:42.717 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-19T01:00:42.763 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-19T01:00:42.942 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-19T01:00:42.986 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-19T01:00:43.064 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-19T01:00:43.696 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-19T01:00:43.809 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-19T01:00:45.023 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-19T01:00:45.104 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-19T01:00:45.135 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-19T01:00:45.166 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-19T01:00:45.194 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-19T01:00:45.508 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-19T01:00:45.686 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T01:00:45.948 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 124/198 2024-03-19T01:00:45.979 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 125/198 2024-03-19T01:00:46.015 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-19T01:00:46.486 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-19T01:00:46.519 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T01:00:46.573 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 128/198 2024-03-19T01:00:46.596 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 129/198 2024-03-19T01:00:48.774 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T01:00:48.817 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T01:00:49.352 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 130/198 2024-03-19T01:00:49.387 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T01:00:49.453 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 131/198 2024-03-19T01:00:49.453 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-19T01:00:49.453 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-19T01:00:49.454 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T01:00:49.480 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T01:01:01.093 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 132/198 2024-03-19T01:01:01.093 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2024-03-19T01:01:01.093 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2024-03-19T01:01:01.093 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2024-03-19T01:01:01.094 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2024-03-19T01:01:01.094 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2024-03-19T01:01:01.094 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /root 2024-03-19T01:01:01.094 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2024-03-19T01:01:01.094 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T01:01:01.273 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T01:01:01.288 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 133/198 2024-03-19T01:01:01.288 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T01:01:01.288 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-19T01:01:01.288 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T01:01:02.768 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T01:01:02.795 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 134/198 2024-03-19T01:01:02.795 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T01:01:02.795 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-19T01:01:02.795 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T01:01:02.795 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-19T01:01:02.795 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T01:01:03.019 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T01:01:03.041 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 135/198 2024-03-19T01:01:03.042 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T01:01:03.042 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-19T01:01:03.042 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T01:01:03.042 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-19T01:01:03.042 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T01:01:03.478 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T01:01:03.500 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 136/198 2024-03-19T01:01:03.500 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T01:01:03.501 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-19T01:01:03.501 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T01:01:03.501 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-19T01:01:03.501 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T01:01:03.605 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-19T01:01:03.923 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T01:01:03.953 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 138/198 2024-03-19T01:01:03.986 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 139/198 2024-03-19T01:01:04.046 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 140/198 2024-03-19T01:01:04.076 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T01:01:04.312 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 141/198 2024-03-19T01:01:04.344 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T01:01:05.314 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 142/198 2024-03-19T01:01:05.345 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T01:01:05.445 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 143/198 2024-03-19T01:01:05.599 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 144/198 2024-03-19T01:01:05.638 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T01:01:05.661 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 145/198 2024-03-19T01:01:05.661 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T01:01:05.661 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-19T01:01:05.661 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T01:01:05.661 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-19T01:01:05.661 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T01:01:05.706 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T01:01:05.716 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 146/198 2024-03-19T01:01:05.762 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 147/198 2024-03-19T01:01:06.635 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-19T01:01:06.666 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T01:01:06.688 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 149/198 2024-03-19T01:01:06.688 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T01:01:06.689 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-19T01:01:06.689 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T01:01:06.689 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-19T01:01:06.689 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T01:01:06.734 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T01:01:06.755 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 150/198 2024-03-19T01:01:06.755 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T01:01:06.755 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-19T01:01:06.755 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T01:01:06.801 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T01:01:06.823 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T01:01:06.823 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T01:01:06.823 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-19T01:01:06.823 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T01:01:06.823 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-19T01:01:06.823 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T01:01:07.135 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T01:01:07.157 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 152/198 2024-03-19T01:01:07.157 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-19T01:01:07.157 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-19T01:01:07.157 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T01:01:07.157 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-19T01:01:07.157 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T01:01:11.606 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T01:01:11.648 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 154/198 2024-03-19T01:01:11.678 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 155/198 2024-03-19T01:01:11.705 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-19T01:01:11.880 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 157/198 2024-03-19T01:01:12.005 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-19T01:01:12.046 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 159/198 2024-03-19T01:01:12.085 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-19T01:01:12.190 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-19T01:01:12.234 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-19T01:01:12.264 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 163/198 2024-03-19T01:01:12.296 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-19T01:01:12.326 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-19T01:01:12.389 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-19T01:01:12.449 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-19T01:01:12.474 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-19T01:01:13.061 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T01:01:13.098 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T01:01:13.117 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-19T01:01:13.950 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 170/198 2024-03-19T01:01:14.035 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-19T01:01:14.076 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-19T01:01:14.156 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-19T01:01:14.206 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-19T01:01:14.231 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-19T01:01:14.256 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-19T01:01:14.283 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-19T01:01:14.283 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T01:01:14.354 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-19T01:01:14.383 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-19T01:01:14.383 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T01:01:14.452 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-19T01:01:14.478 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-19T01:01:14.511 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-19T01:01:14.537 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-19T01:01:14.561 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-19T01:01:14.594 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-19T01:01:14.634 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-19T01:01:14.667 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-19T01:01:14.692 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-19T01:01:14.717 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-19T01:01:14.742 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-19T01:01:14.767 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-19T01:01:14.773 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T01:01:14.773 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T01:01:14.954 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-19T01:01:14.954 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T01:01:15.045 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-19T01:01:15.070 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-19T01:01:15.095 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-19T01:01:15.129 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-19T01:01:15.155 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-19T01:01:15.219 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T01:01:20.097 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-19T01:01:20.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 1/198 2024-03-19T01:01:20.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2/198 2024-03-19T01:01:20.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 3/198 2024-03-19T01:01:20.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 4/198 2024-03-19T01:01:20.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2 5/198 2024-03-19T01:01:20.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 6/198 2024-03-19T01:01:20.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 7/198 2024-03-19T01:01:20.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 8/198 2024-03-19T01:01:20.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 9/198 2024-03-19T01:01:20.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 10/198 2024-03-19T01:01:20.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 11/198 2024-03-19T01:01:20.098 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 12/198 2024-03-19T01:01:20.099 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 13/198 2024-03-19T01:01:20.099 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 14/198 2024-03-19T01:01:20.099 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_ 15/198 2024-03-19T01:01:20.099 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 16/198 2024-03-19T01:01:20.099 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 17/198 2024-03-19T01:01:20.099 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_6 18/198 2024-03-19T01:01:20.099 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86 19/198 2024-03-19T01:01:20.099 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 20/198 2024-03-19T01:01:20.099 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el 21/198 2024-03-19T01:01:20.099 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9. 22/198 2024-03-19T01:01:20.099 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_6 23/198 2024-03-19T01:01:20.099 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 24/198 2024-03-19T01:01:20.099 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 25/198 2024-03-19T01:01:20.100 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 26/198 2024-03-19T01:01:20.100 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 27/198 2024-03-19T01:01:20.100 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 28/198 2024-03-19T01:01:20.100 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 29/198 2024-03-19T01:01:20.100 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491. 30/198 2024-03-19T01:01:20.100 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noa 31/198 2024-03-19T01:01:20.100 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.n 32/198 2024-03-19T01:01:20.100 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6a 33/198 2024-03-19T01:01:20.100 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el 34/198 2024-03-19T01:01:20.100 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 35/198 2024-03-19T01:01:20.101 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.e 36/198 2024-03-19T01:01:20.101 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 37/198 2024-03-19T01:01:20.101 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 38/198 2024-03-19T01:01:20.101 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 39/198 2024-03-19T01:01:20.101 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-19T01:01:20.101 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-19T01:01:20.101 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-19T01:01:20.101 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-19T01:01:20.101 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-19T01:01:20.101 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-19T01:01:20.101 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-19T01:01:20.101 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-19T01:01:20.102 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-19T01:01:20.102 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-19T01:01:20.102 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-19T01:01:20.102 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-19T01:01:20.102 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-19T01:01:20.102 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-19T01:01:20.102 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-19T01:01:20.102 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-19T01:01:20.102 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-19T01:01:20.102 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-19T01:01:20.102 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-19T01:01:20.102 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-19T01:01:20.103 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-19T01:01:20.103 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-19T01:01:20.103 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-19T01:01:20.103 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-19T01:01:20.103 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-19T01:01:20.103 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-19T01:01:20.103 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-19T01:01:20.103 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-19T01:01:20.103 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-19T01:01:20.103 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-19T01:01:20.103 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-19T01:01:20.103 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-19T01:01:20.103 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-19T01:01:20.104 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-19T01:01:20.104 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-19T01:01:20.104 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-19T01:01:20.104 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-19T01:01:20.104 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-19T01:01:20.104 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-19T01:01:20.104 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-19T01:01:20.104 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-19T01:01:20.104 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-19T01:01:20.104 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-19T01:01:20.104 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-19T01:01:20.104 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-19T01:01:20.104 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-19T01:01:20.105 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-19T01:01:20.106 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-19T01:01:20.106 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-19T01:01:20.106 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-19T01:01:20.106 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-19T01:01:20.106 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-19T01:01:20.106 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-19T01:01:20.106 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-19T01:01:20.106 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-19T01:01:20.106 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-19T01:01:20.106 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-19T01:01:20.106 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-19T01:01:20.106 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-19T01:01:20.106 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-19T01:01:20.107 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-19T01:01:20.108 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 151/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 153/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-19T01:01:20.109 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-19T01:01:20.110 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-19T01:01:20.111 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-19T01:01:20.111 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-19T01:01:20.111 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-19T01:01:20.111 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-19T01:01:20.111 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-19T01:01:20.111 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-19T01:01:20.111 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-19T01:01:20.111 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-19T01:01:20.111 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-19T01:01:20.111 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-19T01:01:20.111 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-19T01:01:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-19T01:01:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-19T01:01:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-19T01:01:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-19T01:01:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-19T01:01:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-19T01:01:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-19T01:01:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-19T01:01:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-19T01:01:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-19T01:01:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-19T01:01:20.112 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-19T01:01:20.113 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-19T01:01:20.113 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-19T01:01:20.113 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-19T01:01:20.113 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-19T01:01:20.113 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-19T01:01:22.888 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-19T01:01:22.889 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T01:01:22.889 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2024-03-19T01:01:22.889 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-19T01:01:22.889 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-19T01:01:22.889 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-19T01:01:22.889 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-19T01:01:22.889 INFO:teuthology.orchestra.run.smithi156.stdout: acl-2.3.1-4.el9.x86_64 2024-03-19T01:01:22.889 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-19T01:01:22.889 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-19T01:01:22.889 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-19T01:01:22.890 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-19T01:01:22.890 INFO:teuthology.orchestra.run.smithi156.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-19T01:01:22.890 INFO:teuthology.orchestra.run.smithi156.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-19T01:01:22.890 INFO:teuthology.orchestra.run.smithi156.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-19T01:01:22.890 INFO:teuthology.orchestra.run.smithi156.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-19T01:01:22.890 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.890 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.890 INFO:teuthology.orchestra.run.smithi156.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-19T01:01:22.890 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-19T01:01:22.890 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-19T01:01:22.891 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-19T01:01:22.891 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-19T01:01:22.891 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-19T01:01:22.891 INFO:teuthology.orchestra.run.smithi156.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-19T01:01:22.891 INFO:teuthology.orchestra.run.smithi156.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-19T01:01:22.891 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-19T01:01:22.891 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-03-19T01:01:22.891 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-19T01:01:22.891 INFO:teuthology.orchestra.run.smithi156.stdout: automake-1.16.2-8.el9.noarch 2024-03-19T01:01:22.891 INFO:teuthology.orchestra.run.smithi156.stdout: bison-3.7.4-5.el9.x86_64 2024-03-19T01:01:22.891 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-19T01:01:22.892 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.892 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.892 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.892 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.892 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T01:01:22.892 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.892 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.892 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.892 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T01:01:22.892 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T01:01:22.892 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T01:01:22.893 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T01:01:22.893 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T01:01:22.893 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.893 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.893 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T01:01:22.893 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.893 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.893 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.893 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T01:01:22.893 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T01:01:22.893 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.894 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.894 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top-2:19.0.0-2206.gb6ae2491.el9.noarch 2024-03-19T01:01:22.894 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-19T01:01:22.894 INFO:teuthology.orchestra.run.smithi156.stdout: flex-2.6.4-9.el9.x86_64 2024-03-19T01:01:22.894 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-19T01:01:22.894 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-19T01:01:22.894 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-19T01:01:22.894 INFO:teuthology.orchestra.run.smithi156.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-19T01:01:22.894 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-19T01:01:22.894 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-19T01:01:22.894 INFO:teuthology.orchestra.run.smithi156.stdout: jq-1.6-16.el9.x86_64 2024-03-19T01:01:22.895 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-19T01:01:22.895 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-19T01:01:22.895 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-19T01:01:22.895 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.895 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.895 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.895 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-19T01:01:22.895 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-19T01:01:22.895 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-19T01:01:22.895 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-19T01:01:22.895 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-19T01:01:22.895 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.896 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.896 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-19T01:01:22.896 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.896 INFO:teuthology.orchestra.run.smithi156.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-19T01:01:22.896 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-19T01:01:22.896 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-19T01:01:22.896 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-19T01:01:22.896 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-19T01:01:22.896 INFO:teuthology.orchestra.run.smithi156.stdout: lua-5.4.4-4.el9.x86_64 2024-03-19T01:01:22.896 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-19T01:01:22.896 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-19T01:01:22.896 INFO:teuthology.orchestra.run.smithi156.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-19T01:01:22.897 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-19T01:01:22.897 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-19T01:01:22.897 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-19T01:01:22.897 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-19T01:01:22.897 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-13.11-1.el9.x86_64 2024-03-19T01:01:22.897 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-19T01:01:22.897 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-19T01:01:22.897 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-19T01:01:22.897 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-19T01:01:22.897 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-19T01:01:22.897 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-19T01:01:22.897 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-19T01:01:22.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-19T01:01:22.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-19T01:01:22.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-19T01:01:22.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-19T01:01:22.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-19T01:01:22.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-19T01:01:22.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-19T01:01:22.898 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-19T01:01:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-19T01:01:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-19T01:01:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-19T01:01:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-19T01:01:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-19T01:01:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-19T01:01:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-19T01:01:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-19T01:01:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-19T01:01:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-19T01:01:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-19T01:01:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-19T01:01:22.899 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-19T01:01:22.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-19T01:01:22.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-19T01:01:22.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-19T01:01:22.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-19T01:01:22.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-19T01:01:22.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-19T01:01:22.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-19T01:01:22.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-19T01:01:22.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-19T01:01:22.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-19T01:01:22.900 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-19T01:01:22.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-19T01:01:22.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply-3.11-14.el9.noarch 2024-03-19T01:01:22.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-19T01:01:22.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-19T01:01:22.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-19T01:01:22.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-19T01:01:22.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-19T01:01:22.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-19T01:01:22.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-19T01:01:22.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-19T01:01:22.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-19T01:01:22.901 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.902 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.902 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-19T01:01:22.902 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-19T01:01:22.902 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-19T01:01:22.902 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.902 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-19T01:01:22.902 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-19T01:01:22.902 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-19T01:01:22.902 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-19T01:01:22.902 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-19T01:01:22.902 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-19T01:01:22.902 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-19T01:01:22.903 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-19T01:01:22.903 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-19T01:01:22.903 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-19T01:01:22.903 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-19T01:01:22.903 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-19T01:01:22.903 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-19T01:01:22.903 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-19T01:01:22.903 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-19T01:01:22.903 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.904 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.904 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:19.0.0-2206.gb6ae2491.el9.x86_64 2024-03-19T01:01:22.904 INFO:teuthology.orchestra.run.smithi156.stdout: re2-1:20211101-3.el9.x86_64 2024-03-19T01:01:22.904 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-19T01:01:22.904 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-19T01:01:22.904 INFO:teuthology.orchestra.run.smithi156.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-19T01:01:22.904 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-19T01:01:22.904 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-19T01:01:22.905 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-19T01:01:22.905 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-03-19T01:01:22.905 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-03-19T01:01:23.389 DEBUG:teuthology.parallel:result is None 2024-03-19T01:01:23.390 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:01:23.529 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T01:01:23.556 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T01:01:23.556 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T01:01:23.556 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T01:01:23.558 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:01:23.704 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T01:01:23.726 INFO:teuthology.orchestra.run.smithi156.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T01:01:23.726 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T01:01:23.726 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T01:01:23.728 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:01:24.050 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-19T01:01:24.077 INFO:teuthology.orchestra.run.smithi162.stdout:19.0.0-2206.gb6ae2491.el9 2024-03-19T01:01:24.077 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2206.gb6ae2491.el9 2024-03-19T01:01:24.078 INFO:teuthology.task.install:The correct ceph version 19.0.0-2206.gb6ae2491 is installed. 2024-03-19T01:01:24.079 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-19T01:01:24.080 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:01:24.080 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T01:01:24.115 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-19T01:01:24.115 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T01:01:24.148 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T01:01:24.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-19T01:01:24.184 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-19T01:01:24.184 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:01:24.184 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T01:01:24.214 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T01:01:24.283 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-19T01:01:24.283 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T01:01:24.313 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T01:01:24.382 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T01:01:24.383 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-03-19T01:01:24.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-19T01:01:24.485 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-19T01:01:24.485 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:01:24.486 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T01:01:24.517 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T01:01:24.588 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-19T01:01:24.588 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T01:01:24.618 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T01:01:24.687 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T01:01:24.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-19T01:01:24.718 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-19T01:01:24.788 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-19T01:01:24.789 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:01:24.789 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T01:01:24.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T01:01:24.889 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-19T01:01:24.889 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T01:01:24.920 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T01:01:24.990 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T01:01:24.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-03-19T01:01:25.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-19T01:01:25.092 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-19T01:01:25.194 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-19T01:01:25.194 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:01:25.195 INFO:tasks.cephadm:Cluster fsid is 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:01:25.195 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-19T01:01:25.195 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.116', 'mon.b': '172.21.15.156', 'mon.c': '172.21.15.162'} 2024-03-19T01:01:25.195 INFO:tasks.cephadm:First mon is mon.a on smithi116 2024-03-19T01:01:25.195 INFO:tasks.cephadm:First mgr is x 2024-03-19T01:01:25.196 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-19T01:01:25.196 DEBUG:teuthology.orchestra.run.smithi116:> sudo hostname $(hostname -s) 2024-03-19T01:01:25.228 DEBUG:teuthology.orchestra.run.smithi156:> sudo hostname $(hostname -s) 2024-03-19T01:01:25.260 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname $(hostname -s) 2024-03-19T01:01:25.292 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-19T01:01:25.292 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:01:25.433 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'b6ae24918b03b1b8a19c9263a50655bba80bb810', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77736/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2206-gb6ae2491', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2206.gb6ae2491'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-18 20:42:27.872142', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240318.181317', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-19T01:01:25.575 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-batrick-testing-20240318.181317, sha1 b6ae24918b03b1b8a19c9263a50655bba80bb810 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:01:25.577 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm 2024-03-19T01:01:25.577 DEBUG:teuthology.orchestra.run.smithi116:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T01:01:26.037 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 19 01:01 /home/ubuntu/cephtest/cephadm 2024-03-19T01:01:26.038 DEBUG:teuthology.orchestra.run.smithi156:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T01:01:26.960 INFO:teuthology.orchestra.run.smithi156.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 19 01:01 /home/ubuntu/cephtest/cephadm 2024-03-19T01:01:26.960 DEBUG:teuthology.orchestra.run.smithi162:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240318.181317/b6ae24918b03b1b8a19c9263a50655bba80bb810/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-19T01:01:27.409 INFO:teuthology.orchestra.run.smithi162.stdout:-rw-r--r--. 1 ubuntu ubuntu 767404 Mar 19 01:01 /home/ubuntu/cephtest/cephadm 2024-03-19T01:01:27.410 DEBUG:teuthology.orchestra.run.smithi116:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T01:01:27.430 DEBUG:teuthology.orchestra.run.smithi156:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T01:01:27.449 DEBUG:teuthology.orchestra.run.smithi162:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-19T01:01:27.484 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 on all hosts... 2024-03-19T01:01:27.485 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-19T01:01:27.487 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-19T01:01:27.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 pull 2024-03-19T01:01:28.288 INFO:teuthology.orchestra.run.smithi162.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T01:01:28.308 INFO:teuthology.orchestra.run.smithi116.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T01:01:28.353 INFO:teuthology.orchestra.run.smithi156.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T01:01:55.153 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-19T01:01:55.153 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T01:01:55.153 INFO:teuthology.orchestra.run.smithi162.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-19T01:01:55.153 INFO:teuthology.orchestra.run.smithi162.stdout: "repo_digests": [ 2024-03-19T01:01:55.153 INFO:teuthology.orchestra.run.smithi162.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-19T01:01:55.153 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-03-19T01:01:55.153 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-19T01:02:15.327 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-19T01:02:15.327 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T01:02:15.327 INFO:teuthology.orchestra.run.smithi156.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-19T01:02:15.327 INFO:teuthology.orchestra.run.smithi156.stdout: "repo_digests": [ 2024-03-19T01:02:15.327 INFO:teuthology.orchestra.run.smithi156.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-19T01:02:15.327 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2024-03-19T01:02:15.328 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-19T01:02:21.454 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:02:21.454 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev)", 2024-03-19T01:02:21.455 INFO:teuthology.orchestra.run.smithi116.stdout: "image_id": "2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c", 2024-03-19T01:02:21.455 INFO:teuthology.orchestra.run.smithi116.stdout: "repo_digests": [ 2024-03-19T01:02:21.455 INFO:teuthology.orchestra.run.smithi116.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639" 2024-03-19T01:02:21.455 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T01:02:21.455 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:02:21.480 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph 2024-03-19T01:02:21.529 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph 2024-03-19T01:02:21.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph 2024-03-19T01:02:21.600 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /etc/ceph 2024-03-19T01:02:21.627 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /etc/ceph 2024-03-19T01:02:21.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /etc/ceph 2024-03-19T01:02:21.688 INFO:tasks.cephadm:Writing seed config... 2024-03-19T01:02:21.689 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-19T01:02:21.689 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-19T01:02:21.690 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-19T01:02:21.691 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-19T01:02:21.692 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-19T01:02:21.692 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-19T01:02:21.692 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-19T01:02:21.692 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-19T01:02:21.692 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-19T01:02:21.692 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-19T01:02:21.692 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-19T01:02:21.692 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-19T01:02:21.692 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-19T01:02:21.692 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-19T01:02:21.692 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-19T01:02:21.693 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:02:21.693 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-19T01:02:21.707 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 35912c14-e58c-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-19T01:02:21.708 DEBUG:teuthology.orchestra.run.smithi116:mon.a> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.a.service 2024-03-19T01:02:21.749 DEBUG:teuthology.orchestra.run.smithi116:mgr.x> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.x.service 2024-03-19T01:02:21.792 INFO:tasks.cephadm:Bootstrapping... 2024-03-19T01:02:21.792 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 -v bootstrap --fsid 35912c14-e58c-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.116 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-19T01:02:21.986 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-03-19T01:02:21.986 INFO:teuthology.orchestra.run.smithi116.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810', '-v', 'bootstrap', '--fsid', '35912c14-e58c-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.116', '--skip-admin-label'] 2024-03-19T01:02:22.011 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T01:02:22.012 INFO:teuthology.orchestra.run.smithi116.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-19T01:02:22.012 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying podman|docker is present... 2024-03-19T01:02:22.035 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T01:02:22.035 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying lvm2 is present... 2024-03-19T01:02:22.035 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying time synchronization is in place... 2024-03-19T01:02:22.043 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-19T01:02:22.043 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-19T01:02:22.049 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-19T01:02:22.049 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-19T01:02:22.055 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout enabled 2024-03-19T01:02:22.061 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout active 2024-03-19T01:02:22.061 INFO:teuthology.orchestra.run.smithi116.stdout:Unit chronyd.service is enabled and running 2024-03-19T01:02:22.061 INFO:teuthology.orchestra.run.smithi116.stdout:Repeating the final host check... 2024-03-19T01:02:22.084 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-19T01:02:22.084 INFO:teuthology.orchestra.run.smithi116.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-19T01:02:22.084 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl is present 2024-03-19T01:02:22.084 INFO:teuthology.orchestra.run.smithi116.stdout:lvcreate is present 2024-03-19T01:02:22.090 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-19T01:02:22.091 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-19T01:02:22.096 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-19T01:02:22.096 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-19T01:02:22.102 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout enabled 2024-03-19T01:02:22.108 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout active 2024-03-19T01:02:22.108 INFO:teuthology.orchestra.run.smithi116.stdout:Unit chronyd.service is enabled and running 2024-03-19T01:02:22.108 INFO:teuthology.orchestra.run.smithi116.stdout:Host looks OK 2024-03-19T01:02:22.109 INFO:teuthology.orchestra.run.smithi116.stdout:Cluster fsid: 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:02:22.109 INFO:teuthology.orchestra.run.smithi116.stdout:Acquiring lock 139914302599072 on /run/cephadm/35912c14-e58c-11ee-95c9-87774f69a715.lock 2024-03-19T01:02:22.109 INFO:teuthology.orchestra.run.smithi116.stdout:Lock 139914302599072 acquired on /run/cephadm/35912c14-e58c-11ee-95c9-87774f69a715.lock 2024-03-19T01:02:22.109 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying IP 172.21.15.116 port 3300 ... 2024-03-19T01:02:22.109 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying IP 172.21.15.116 port 6789 ... 2024-03-19T01:02:22.109 INFO:teuthology.orchestra.run.smithi116.stdout:Base mon IP(s) is [172.21.15.116:3300, 172.21.15.116:6789], mon addrv is [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2024-03-19T01:02:22.112 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.116 metric 100 2024-03-19T01:02:22.113 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.116 metric 100 2024-03-19T01:02:22.115 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-19T01:02:22.115 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-19T01:02:22.115 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1716sec hoplimit 64 pref medium 2024-03-19T01:02:22.117 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-19T01:02:22.118 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-19T01:02:22.118 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-19T01:02:22.118 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-19T01:02:22.118 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-03-19T01:02:22.118 INFO:teuthology.orchestra.run.smithi116.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-19T01:02:22.119 INFO:teuthology.orchestra.run.smithi116.stdout:Mon IP `172.21.15.116` is in CIDR network `172.21.0.0/20` 2024-03-19T01:02:22.119 INFO:teuthology.orchestra.run.smithi116.stdout:Mon IP `172.21.15.116` is in CIDR network `172.21.0.0/20` 2024-03-19T01:02:22.119 INFO:teuthology.orchestra.run.smithi116.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-19T01:02:22.119 INFO:teuthology.orchestra.run.smithi116.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-19T01:02:22.119 INFO:teuthology.orchestra.run.smithi116.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T01:02:22.665 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stdout 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c 2024-03-19T01:02:22.665 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810... 2024-03-19T01:02:22.665 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Getting image source signatures 2024-03-19T01:02:22.666 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying blob sha256:93b5a8d362aeee7e8b1f0bbf18238e1bb256c50573ad5b72aff9e382b1bb3ccb 2024-03-19T01:02:22.666 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-19T01:02:22.666 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Copying config sha256:2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c 2024-03-19T01:02:22.666 INFO:teuthology.orchestra.run.smithi116.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-19T01:02:23.553 INFO:teuthology.orchestra.run.smithi116.stdout:ceph: stdout ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev) 2024-03-19T01:02:23.553 INFO:teuthology.orchestra.run.smithi116.stdout:Ceph version: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev) 2024-03-19T01:02:23.553 INFO:teuthology.orchestra.run.smithi116.stdout:Extracting ceph user uid/gid from container image... 2024-03-19T01:02:24.517 INFO:teuthology.orchestra.run.smithi116.stdout:stat: stdout 167 167 2024-03-19T01:02:24.518 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial keys... 2024-03-19T01:02:25.397 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQAg5PhlLErAMRAANvJhpTzaj0d3n8KmPO7YkA== 2024-03-19T01:02:26.361 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQAh5PhlP7UnKxAAGZLeFSJINLpKaMEe5M64Yg== 2024-03-19T01:02:27.349 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph-authtool: stdout AQAi5PhlPnrwJxAAUiKAb2Iyp/fLzQPtajrSWw== 2024-03-19T01:02:27.350 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial monmap... 2024-03-19T01:02:28.163 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-19T01:02:28.163 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-19T01:02:28.163 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:02:28.164 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-19T01:02:28.164 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool for a [v2:172.21.15.116:3300,v1:172.21.15.116:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-19T01:02:28.164 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = quincy 2024-03-19T01:02:28.164 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: set fsid to 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:02:28.164 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-19T01:02:28.164 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:02:28.164 INFO:teuthology.orchestra.run.smithi116.stdout:Creating mon... 2024-03-19T01:02:29.035 INFO:teuthology.orchestra.run.smithi116.stdout:create mon.a on 2024-03-19T01:02:29.231 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-19T01:02:29.367 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-19T01:02:29.535 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-35912c14-e58c-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-35912c14-e58c-11ee-95c9-87774f69a715.target. 2024-03-19T01:02:29.536 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-35912c14-e58c-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-35912c14-e58c-11ee-95c9-87774f69a715.target. 2024-03-19T01:02:29.778 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.a 2024-03-19T01:02:29.778 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to reset failed state of unit ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-19T01:02:29.948 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-35912c14-e58c-11ee-95c9-87774f69a715.target.wants/ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-35912c14-e58c-11ee-95c9-87774f69a715@.service. 2024-03-19T01:02:30.493 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T01:02:30.493 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-19T01:02:30.499 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T01:02:30.500 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-19T01:02:30.500 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-19T01:02:30.500 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to enable service . firewalld.service is not available 2024-03-19T01:02:30.500 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mon to start... 2024-03-19T01:02:30.500 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mon... 2024-03-19T01:02:31.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout cluster: 2024-03-19T01:02:31.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout id: 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:02:31.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-19T01:02:31.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T01:02:31.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout services: 2024-03-19T01:02:31.477 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.510464s) 2024-03-19T01:02:31.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-19T01:02:31.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-19T01:02:31.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T01:02:31.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout data: 2024-03-19T01:02:31.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-19T01:02:31.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-19T01:02:31.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-19T01:02:31.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout pgs: 2024-03-19T01:02:31.478 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T01:02:31.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.904+0000 7f02e9179700 1 Processor -- start 2024-03-19T01:02:31.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.904+0000 7f02e9179700 1 -- start start 2024-03-19T01:02:31.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.904+0000 7f02e9179700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 0x7f02e4109680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:31.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.904+0000 7f02e9179700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02e4109c50 con 0x7f02e4109260 2024-03-19T01:02:31.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.905+0000 7f02e2d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 0x7f02e4109680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:31.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.905+0000 7f02e2d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 0x7f02e4109680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42002/0 (socket says 172.21.15.116:42002) 2024-03-19T01:02:31.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.905+0000 7f02e2d9d700 1 -- 172.21.15.116:0/1806021537 learned_addr learned my addr 172.21.15.116:0/1806021537 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:31.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.905+0000 7f02e2d9d700 1 -- 172.21.15.116:0/1806021537 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e4109dd0 con 0x7f02e4109260 2024-03-19T01:02:31.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.905+0000 7f02e2d9d700 1 --2- 172.21.15.116:0/1806021537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 0x7f02e4109680 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f02d40098a0 tx=0x7f02d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c66a5d6fc89b667 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:31.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.905+0000 7f02e1d9b700 1 -- 172.21.15.116:0/1806021537 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02d4004160 con 0x7f02e4109260 2024-03-19T01:02:31.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.905+0000 7f02e1d9b700 1 -- 172.21.15.116:0/1806021537 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02d4004300 con 0x7f02e4109260 2024-03-19T01:02:31.479 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.905+0000 7f02e1d9b700 1 -- 172.21.15.116:0/1806021537 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02d40036a0 con 0x7f02e4109260 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.906+0000 7f02e9179700 1 -- 172.21.15.116:0/1806021537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 msgr2=0x7f02e4109680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.906+0000 7f02e9179700 1 --2- 172.21.15.116:0/1806021537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 0x7f02e4109680 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f02d40098a0 tx=0x7f02d4009bb0 comp rx=0 tx=0).stop 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.906+0000 7f02e9179700 1 -- 172.21.15.116:0/1806021537 shutdown_connections 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.906+0000 7f02e9179700 1 --2- 172.21.15.116:0/1806021537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 0x7f02e4109680 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.906+0000 7f02e9179700 1 -- 172.21.15.116:0/1806021537 >> 172.21.15.116:0/1806021537 conn(0x7f02e4101880 msgr2=0x7f02e4103ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.906+0000 7f02e9179700 1 -- 172.21.15.116:0/1806021537 shutdown_connections 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.906+0000 7f02e9179700 1 -- 172.21.15.116:0/1806021537 wait complete. 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.907+0000 7f02e9179700 1 Processor -- start 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.907+0000 7f02e9179700 1 -- start start 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.907+0000 7f02e9179700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 0x7f02e419b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.907+0000 7f02e9179700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02e4105fe0 con 0x7f02e4109260 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.907+0000 7f02e2d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 0x7f02e419b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.907+0000 7f02e2d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 0x7f02e419b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42008/0 (socket says 172.21.15.116:42008) 2024-03-19T01:02:31.480 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.907+0000 7f02e2d9d700 1 -- 172.21.15.116:0/2955083637 learned_addr learned my addr 172.21.15.116:0/2955083637 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:31.481 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.907+0000 7f02e2d9d700 1 -- 172.21.15.116:0/2955083637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e419b810 con 0x7f02e4109260 2024-03-19T01:02:31.481 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.908+0000 7f02e2d9d700 1 --2- 172.21.15.116:0/2955083637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 0x7f02e419b2d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f02d4000c00 tx=0x7f02d4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:31.481 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.908+0000 7f02cbfff700 1 -- 172.21.15.116:0/2955083637 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02d4003db0 con 0x7f02e4109260 2024-03-19T01:02:31.481 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.908+0000 7f02cbfff700 1 -- 172.21.15.116:0/2955083637 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02d4011630 con 0x7f02e4109260 2024-03-19T01:02:31.481 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.908+0000 7f02e9179700 1 -- 172.21.15.116:0/2955083637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e419bb00 con 0x7f02e4109260 2024-03-19T01:02:31.481 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.908+0000 7f02cbfff700 1 -- 172.21.15.116:0/2955083637 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02d4011950 con 0x7f02e4109260 2024-03-19T01:02:31.481 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.908+0000 7f02e9179700 1 -- 172.21.15.116:0/2955083637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e419e790 con 0x7f02e4109260 2024-03-19T01:02:31.481 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.908+0000 7f02cbfff700 1 -- 172.21.15.116:0/2955083637 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f02d4011af0 con 0x7f02e4109260 2024-03-19T01:02:31.481 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.908+0000 7f02cbfff700 1 -- 172.21.15.116:0/2955083637 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f02d401c450 con 0x7f02e4109260 2024-03-19T01:02:31.481 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.909+0000 7f02e9179700 1 -- 172.21.15.116:0/2955083637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02e4063940 con 0x7f02e4109260 2024-03-19T01:02:31.481 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.911+0000 7f02cbfff700 1 -- 172.21.15.116:0/2955083637 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f02d4032020 con 0x7f02e4109260 2024-03-19T01:02:31.481 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.979+0000 7f02e9179700 1 -- 172.21.15.116:0/2955083637 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f02e402dda0 con 0x7f02e4109260 2024-03-19T01:02:31.481 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.980+0000 7f02cbfff700 1 -- 172.21.15.116:0/2955083637 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f02d4004260 con 0x7f02e4109260 2024-03-19T01:02:31.482 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.980+0000 7f02e9179700 1 -- 172.21.15.116:0/2955083637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 msgr2=0x7f02e419b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:31.482 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.980+0000 7f02e9179700 1 --2- 172.21.15.116:0/2955083637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 0x7f02e419b2d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f02d4000c00 tx=0x7f02d4004750 comp rx=0 tx=0).stop 2024-03-19T01:02:31.482 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.980+0000 7f02e9179700 1 -- 172.21.15.116:0/2955083637 shutdown_connections 2024-03-19T01:02:31.482 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.980+0000 7f02e9179700 1 --2- 172.21.15.116:0/2955083637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e4109260 0x7f02e419b2d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:31.482 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.980+0000 7f02e9179700 1 -- 172.21.15.116:0/2955083637 >> 172.21.15.116:0/2955083637 conn(0x7f02e4101880 msgr2=0x7f02e4191fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:31.482 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.980+0000 7f02e9179700 1 -- 172.21.15.116:0/2955083637 shutdown_connections 2024-03-19T01:02:31.482 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:30.980+0000 7f02e9179700 1 -- 172.21.15.116:0/2955083637 wait complete. 2024-03-19T01:02:31.482 INFO:teuthology.orchestra.run.smithi116.stdout:mon is available 2024-03-19T01:02:31.482 INFO:teuthology.orchestra.run.smithi116.stdout:Assimilating anything we can from ceph.conf... 2024-03-19T01:02:32.444 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T01:02:32.444 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [global] 2024-03-19T01:02:32.444 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout fsid = 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:02:32.444 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.116:3300,v1:172.21.15.116:6789] 2024-03-19T01:02:32.444 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-19T01:02:32.444 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-19T01:02:32.444 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-19T01:02:32.444 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-19T01:02:32.445 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T01:02:32.445 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-19T01:02:32.445 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-19T01:02:32.445 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T01:02:32.445 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [osd] 2024-03-19T01:02:32.445 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-19T01:02:32.445 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-19T01:02:32.445 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-19T01:02:32.445 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-19T01:02:32.445 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-19T01:02:32.445 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.826+0000 7f4ad9143700 1 Processor -- start 2024-03-19T01:02:32.445 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.826+0000 7f4ad9143700 1 -- start start 2024-03-19T01:02:32.445 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.826+0000 7f4ad9143700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 0x7f4ad4106370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:32.446 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.826+0000 7f4ad9143700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad4106940 con 0x7f4ad4105f50 2024-03-19T01:02:32.446 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.826+0000 7f4ad2d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 0x7f4ad4106370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:32.446 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.826+0000 7f4ad2d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 0x7f4ad4106370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42022/0 (socket says 172.21.15.116:42022) 2024-03-19T01:02:32.446 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.826+0000 7f4ad2d9d700 1 -- 172.21.15.116:0/2658096005 learned_addr learned my addr 172.21.15.116:0/2658096005 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:32.446 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.826+0000 7f4ad2d9d700 1 -- 172.21.15.116:0/2658096005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad4108ee0 con 0x7f4ad4105f50 2024-03-19T01:02:32.446 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.826+0000 7f4ad2d9d700 1 --2- 172.21.15.116:0/2658096005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 0x7f4ad4106370 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4ac40098a0 tx=0x7f4ac4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a888f3d6cee6f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:32.446 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.827+0000 7f4ad1d9b700 1 -- 172.21.15.116:0/2658096005 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ac4004160 con 0x7f4ad4105f50 2024-03-19T01:02:32.446 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.827+0000 7f4ad1d9b700 1 -- 172.21.15.116:0/2658096005 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ac4004300 con 0x7f4ad4105f50 2024-03-19T01:02:32.446 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.827+0000 7f4ad1d9b700 1 -- 172.21.15.116:0/2658096005 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ac40036a0 con 0x7f4ad4105f50 2024-03-19T01:02:32.446 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.827+0000 7f4ad9143700 1 -- 172.21.15.116:0/2658096005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 msgr2=0x7f4ad4106370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:32.446 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.827+0000 7f4ad9143700 1 --2- 172.21.15.116:0/2658096005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 0x7f4ad4106370 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4ac40098a0 tx=0x7f4ac4009bb0 comp rx=0 tx=0).stop 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.827+0000 7f4ad9143700 1 -- 172.21.15.116:0/2658096005 shutdown_connections 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.827+0000 7f4ad9143700 1 --2- 172.21.15.116:0/2658096005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 0x7f4ad4106370 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.827+0000 7f4ad9143700 1 -- 172.21.15.116:0/2658096005 >> 172.21.15.116:0/2658096005 conn(0x7f4ad41018e0 msgr2=0x7f4ad4103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.827+0000 7f4ad9143700 1 -- 172.21.15.116:0/2658096005 shutdown_connections 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.827+0000 7f4ad9143700 1 -- 172.21.15.116:0/2658096005 wait complete. 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.828+0000 7f4ad9143700 1 Processor -- start 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.828+0000 7f4ad9143700 1 -- start start 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.828+0000 7f4ad9143700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 0x7f4ad419b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.828+0000 7f4ad9143700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4ad4109410 con 0x7f4ad4105f50 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.829+0000 7f4ad2d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 0x7f4ad419b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.829+0000 7f4ad2d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 0x7f4ad419b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42036/0 (socket says 172.21.15.116:42036) 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.829+0000 7f4ad2d9d700 1 -- 172.21.15.116:0/1513720726 learned_addr learned my addr 172.21.15.116:0/1513720726 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.829+0000 7f4ad2d9d700 1 -- 172.21.15.116:0/1513720726 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ad419b8b0 con 0x7f4ad4105f50 2024-03-19T01:02:32.447 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.829+0000 7f4ad2d9d700 1 --2- 172.21.15.116:0/1513720726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 0x7f4ad419b370 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4000c00 tx=0x7f4ac4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.829+0000 7f4abbfff700 1 -- 172.21.15.116:0/1513720726 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ac4003db0 con 0x7f4ad4105f50 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.829+0000 7f4abbfff700 1 -- 172.21.15.116:0/1513720726 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ac4011630 con 0x7f4ad4105f50 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.829+0000 7f4ad9143700 1 -- 172.21.15.116:0/1513720726 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ad419bb40 con 0x7f4ad4105f50 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.829+0000 7f4abbfff700 1 -- 172.21.15.116:0/1513720726 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4ac4011950 con 0x7f4ad4105f50 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.829+0000 7f4ad9143700 1 -- 172.21.15.116:0/1513720726 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ad419e7e0 con 0x7f4ad4105f50 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.829+0000 7f4abbfff700 1 -- 172.21.15.116:0/1513720726 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4ac4011af0 con 0x7f4ad4105f50 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.829+0000 7f4abbfff700 1 -- 172.21.15.116:0/1513720726 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4ac401c450 con 0x7f4ad4105f50 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.830+0000 7f4ad9143700 1 -- 172.21.15.116:0/1513720726 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ad4063940 con 0x7f4ad4105f50 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.832+0000 7f4abbfff700 1 -- 172.21.15.116:0/1513720726 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4ac4032020 con 0x7f4ad4105f50 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.901+0000 7f4ad9143700 1 -- 172.21.15.116:0/1513720726 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f4ad402dd30 con 0x7f4ad4105f50 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.904+0000 7f4abbfff700 1 -- 172.21.15.116:0/1513720726 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+431 (secure 0 0 0) 0x7f4ac4004260 con 0x7f4ad4105f50 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.904+0000 7f4abbfff700 1 -- 172.21.15.116:0/1513720726 <== mon.0 v2:172.21.15.116:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4ac401cb60 con 0x7f4ad4105f50 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.905+0000 7f4ad9143700 1 -- 172.21.15.116:0/1513720726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 msgr2=0x7f4ad419b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:32.448 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.905+0000 7f4ad9143700 1 --2- 172.21.15.116:0/1513720726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 0x7f4ad419b370 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4000c00 tx=0x7f4ac4004750 comp rx=0 tx=0).stop 2024-03-19T01:02:32.449 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.905+0000 7f4ad9143700 1 -- 172.21.15.116:0/1513720726 shutdown_connections 2024-03-19T01:02:32.449 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.905+0000 7f4ad9143700 1 --2- 172.21.15.116:0/1513720726 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ad4105f50 0x7f4ad419b370 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:32.449 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.905+0000 7f4ad9143700 1 -- 172.21.15.116:0/1513720726 >> 172.21.15.116:0/1513720726 conn(0x7f4ad41018e0 msgr2=0x7f4ad4191fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:32.449 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.905+0000 7f4ad9143700 1 -- 172.21.15.116:0/1513720726 shutdown_connections 2024-03-19T01:02:32.449 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:31.905+0000 7f4ad9143700 1 -- 172.21.15.116:0/1513720726 wait complete. 2024-03-19T01:02:32.449 INFO:teuthology.orchestra.run.smithi116.stdout:Generating new minimal ceph.conf... 2024-03-19T01:02:33.405 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.840+0000 7f533df34700 1 Processor -- start 2024-03-19T01:02:33.405 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.840+0000 7f533df34700 1 -- start start 2024-03-19T01:02:33.405 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.840+0000 7f533df34700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 0x7f533807c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:33.405 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.840+0000 7f533df34700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f533807c850 con 0x7f533807de20 2024-03-19T01:02:33.405 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.840+0000 7f53377fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 0x7f533807c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:33.405 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.840+0000 7f53377fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 0x7f533807c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42046/0 (socket says 172.21.15.116:42046) 2024-03-19T01:02:33.405 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.840+0000 7f53377fe700 1 -- 172.21.15.116:0/1970201401 learned_addr learned my addr 172.21.15.116:0/1970201401 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:33.405 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.840+0000 7f53377fe700 1 -- 172.21.15.116:0/1970201401 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533807c9d0 con 0x7f533807de20 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.840+0000 7f53377fe700 1 --2- 172.21.15.116:0/1970201401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 0x7f533807c280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f532c0098a0 tx=0x7f532c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc5bf66a21b785a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.840+0000 7f53367fc700 1 -- 172.21.15.116:0/1970201401 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f532c004160 con 0x7f533807de20 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.841+0000 7f53367fc700 1 -- 172.21.15.116:0/1970201401 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f532c004300 con 0x7f533807de20 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.841+0000 7f53367fc700 1 -- 172.21.15.116:0/1970201401 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f532c011b50 con 0x7f533807de20 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.841+0000 7f533df34700 1 -- 172.21.15.116:0/1970201401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 msgr2=0x7f533807c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.841+0000 7f533df34700 1 --2- 172.21.15.116:0/1970201401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 0x7f533807c280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f532c0098a0 tx=0x7f532c009bb0 comp rx=0 tx=0).stop 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.841+0000 7f533df34700 1 -- 172.21.15.116:0/1970201401 shutdown_connections 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.841+0000 7f533df34700 1 --2- 172.21.15.116:0/1970201401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 0x7f533807c280 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.841+0000 7f533df34700 1 -- 172.21.15.116:0/1970201401 >> 172.21.15.116:0/1970201401 conn(0x7f5338103200 msgr2=0x7f5338105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.841+0000 7f533df34700 1 -- 172.21.15.116:0/1970201401 shutdown_connections 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.841+0000 7f533df34700 1 -- 172.21.15.116:0/1970201401 wait complete. 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.842+0000 7f533df34700 1 Processor -- start 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.842+0000 7f533df34700 1 -- start start 2024-03-19T01:02:33.406 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.842+0000 7f533df34700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 0x7f533819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.842+0000 7f533df34700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f533810e210 con 0x7f533807de20 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.842+0000 7f53377fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 0x7f533819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.842+0000 7f53377fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 0x7f533819de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42048/0 (socket says 172.21.15.116:42048) 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.842+0000 7f53377fe700 1 -- 172.21.15.116:0/123761689 learned_addr learned my addr 172.21.15.116:0/123761689 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.842+0000 7f53377fe700 1 -- 172.21.15.116:0/123761689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533819e3b0 con 0x7f533807de20 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.842+0000 7f53377fe700 1 --2- 172.21.15.116:0/123761689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 0x7f533819de70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f532c011850 tx=0x7f532c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.843+0000 7f5334ff9700 1 -- 172.21.15.116:0/123761689 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f532c003af0 con 0x7f533807de20 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.843+0000 7f5334ff9700 1 -- 172.21.15.116:0/123761689 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f532c020070 con 0x7f533807de20 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.843+0000 7f533df34700 1 -- 172.21.15.116:0/123761689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f533819e640 con 0x7f533807de20 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.843+0000 7f5334ff9700 1 -- 172.21.15.116:0/123761689 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f532c0195a0 con 0x7f533807de20 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.843+0000 7f533df34700 1 -- 172.21.15.116:0/123761689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53381a1330 con 0x7f533807de20 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.843+0000 7f5334ff9700 1 -- 172.21.15.116:0/123761689 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f532c019c30 con 0x7f533807de20 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.843+0000 7f5334ff9700 1 -- 172.21.15.116:0/123761689 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f532c011e00 con 0x7f533807de20 2024-03-19T01:02:33.407 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.844+0000 7f533df34700 1 -- 172.21.15.116:0/123761689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5338063940 con 0x7f533807de20 2024-03-19T01:02:33.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.846+0000 7f5334ff9700 1 -- 172.21.15.116:0/123761689 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f532c003c90 con 0x7f533807de20 2024-03-19T01:02:33.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.913+0000 7f533df34700 1 -- 172.21.15.116:0/123761689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f53381a1500 con 0x7f533807de20 2024-03-19T01:02:33.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.913+0000 7f5334ff9700 1 -- 172.21.15.116:0/123761689 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f532c01a430 con 0x7f533807de20 2024-03-19T01:02:33.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.914+0000 7f533df34700 1 -- 172.21.15.116:0/123761689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 msgr2=0x7f533819de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:33.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.914+0000 7f533df34700 1 --2- 172.21.15.116:0/123761689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 0x7f533819de70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f532c011850 tx=0x7f532c011c10 comp rx=0 tx=0).stop 2024-03-19T01:02:33.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.914+0000 7f533df34700 1 -- 172.21.15.116:0/123761689 shutdown_connections 2024-03-19T01:02:33.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.914+0000 7f533df34700 1 --2- 172.21.15.116:0/123761689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f533807de20 0x7f533819de70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:33.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.914+0000 7f533df34700 1 -- 172.21.15.116:0/123761689 >> 172.21.15.116:0/123761689 conn(0x7f5338103200 msgr2=0x7f5338103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:33.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.914+0000 7f533df34700 1 -- 172.21.15.116:0/123761689 shutdown_connections 2024-03-19T01:02:33.408 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:32.914+0000 7f533df34700 1 -- 172.21.15.116:0/123761689 wait complete. 2024-03-19T01:02:33.408 INFO:teuthology.orchestra.run.smithi116.stdout:Restarting the monitor... 2024-03-19T01:02:34.371 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 podman[62585]: 2024-03-19 01:02:34.176973897 +0000 UTC m=+0.485598966 container remove e3e67515c4a2098525dcfef44b41d9172cbb53ddb6fac712c2a5b313b15fa9d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:02:34.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 systemd[1]: ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-19T01:02:34.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 systemd[1]: Stopped Ceph mon.a for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:02:34.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 systemd[1]: Starting Ceph mon.a for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:02:34.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 podman[62682]: 2024-03-19 01:02:34.511010835 +0000 UTC m=+0.018772337 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:02:34.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 podman[62682]: 2024-03-19 01:02:34.611371232 +0000 UTC m=+0.119132766 container create a41987fc403771ac6802512e0eec1abdf17d68d4c71ee454a69989dd7de95e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-a, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2) 2024-03-19T01:02:34.893 INFO:teuthology.orchestra.run.smithi116.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-19T01:02:35.018 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 podman[62682]: 2024-03-19 01:02:34.813576687 +0000 UTC m=+0.321338189 container init a41987fc403771ac6802512e0eec1abdf17d68d4c71ee454a69989dd7de95e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-19T01:02:35.018 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 podman[62682]: 2024-03-19 01:02:34.816375117 +0000 UTC m=+0.324136606 container start a41987fc403771ac6802512e0eec1abdf17d68d4c71ee454a69989dd7de95e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-19T01:02:35.018 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: set uid:gid to 167:167 (ceph:ceph) 2024-03-19T01:02:35.018 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev), process ceph-mon, pid 2 2024-03-19T01:02:35.018 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: pidfile_write: ignore empty --pid-file 2024-03-19T01:02:35.018 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: load: jerasure load: lrc 2024-03-19T01:02:35.018 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: RocksDB version: 7.9.2 2024-03-19T01:02:35.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Git sha 0 2024-03-19T01:02:35.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Compile date 2024-03-18 18:57:12 2024-03-19T01:02:35.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: DB SUMMARY 2024-03-19T01:02:35.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: DB Session ID: FXIBM2NSTDC7LDL7XU7Q 2024-03-19T01:02:35.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: CURRENT file: CURRENT 2024-03-19T01:02:35.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: IDENTITY file: IDENTITY 2024-03-19T01:02:35.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-19T01:02:35.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-19T01:02:35.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104330 ; 2024-03-19T01:02:35.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.error_if_exists: 0 2024-03-19T01:02:35.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.create_if_missing: 0 2024-03-19T01:02:35.019 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.paranoid_checks: 1 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.env: 0x55fabb88cc60 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.fs: PosixFileSystem 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.info_log: 0x55fabdb97ae0 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_file_opening_threads: 16 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.statistics: (nil) 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.use_fsync: 0 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_log_file_size: 0 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.keep_log_file_num: 1000 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.recycle_log_file_num: 0 2024-03-19T01:02:35.020 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.allow_fallocate: 1 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.allow_mmap_reads: 0 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.allow_mmap_writes: 0 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.use_direct_reads: 0 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.create_missing_column_families: 0 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.db_log_dir: 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.wal_dir: 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.advise_random_on_open: 1 2024-03-19T01:02:35.021 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.db_write_buffer_size: 0 2024-03-19T01:02:35.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.write_buffer_manager: 0x55fabdb6a1e0 2024-03-19T01:02:35.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-19T01:02:35.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-19T01:02:35.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-19T01:02:35.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.rate_limiter: (nil) 2024-03-19T01:02:35.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-19T01:02:35.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.wal_recovery_mode: 2 2024-03-19T01:02:35.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.enable_thread_tracking: 0 2024-03-19T01:02:35.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.enable_pipelined_write: 0 2024-03-19T01:02:35.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.unordered_write: 0 2024-03-19T01:02:35.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-19T01:02:35.022 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-19T01:02:35.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-19T01:02:35.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-19T01:02:35.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.row_cache: None 2024-03-19T01:02:35.023 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.wal_filter: None 2024-03-19T01:02:35.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-19T01:02:35.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.allow_ingest_behind: 0 2024-03-19T01:02:35.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.two_write_queues: 0 2024-03-19T01:02:35.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.manual_wal_flush: 0 2024-03-19T01:02:35.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.wal_compression: 0 2024-03-19T01:02:35.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.atomic_flush: 0 2024-03-19T01:02:35.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-19T01:02:35.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-19T01:02:35.024 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.log_readahead_size: 0 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.best_efforts_recovery: 0 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.allow_data_in_errors: 0 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.db_host_id: __hostname__ 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_background_jobs: 2 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_background_compactions: -1 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_subcompactions: 1 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-19T01:02:35.025 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_total_wal_size: 0 2024-03-19T01:02:35.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-19T01:02:35.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-19T01:02:35.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-19T01:02:35.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-19T01:02:35.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_open_files: -1 2024-03-19T01:02:35.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.bytes_per_sync: 0 2024-03-19T01:02:35.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-19T01:02:35.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-19T01:02:35.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compaction_readahead_size: 0 2024-03-19T01:02:35.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_background_flushes: -1 2024-03-19T01:02:35.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Compression algorithms supported: 2024-03-19T01:02:35.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-19T01:02:35.026 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: kZSTD supported: 0 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: kXpressCompression supported: 0 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: kLZ4HCCompression supported: 1 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: kZlibCompression supported: 1 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: kSnappyCompression supported: 1 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: kLZ4Compression supported: 1 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: kBZip2Compression supported: 0 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.merge_operator: 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compaction_filter: None 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compaction_filter_factory: None 2024-03-19T01:02:35.027 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.sst_partitioner_factory: None 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fabdb97be0) 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: cache_index_and_filter_blocks: 1 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: pin_top_level_index_and_filter: 1 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_type: 0 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: data_block_index_type: 0 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_shortening: 1 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: checksum: 4 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: no_block_cache: 0 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache: 0x55fabdb9d090 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_name: BinnedLRUCache 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_options: 2024-03-19T01:02:35.028 INFO:journalctl@ceph.mon.a.smithi116.stdout: capacity : 536870912 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: num_shard_bits : 4 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: strict_capacity_limit : 0 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: high_pri_pool_ratio: 0.000 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_cache_compressed: (nil) 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: persistent_cache: (nil) 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_size: 4096 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_size_deviation: 10 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_restart_interval: 16 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: index_block_restart_interval: 1 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: metadata_block_size: 4096 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: partition_filters: 0 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: use_delta_encoding: 1 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: filter_policy: bloomfilter 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: whole_key_filtering: 1 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: verify_compression: 0 2024-03-19T01:02:35.029 INFO:journalctl@ceph.mon.a.smithi116.stdout: read_amp_bytes_per_bit: 0 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout: format_version: 5 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout: enable_index_compression: 1 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout: block_align: 0 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout: max_auto_readahead_size: 262144 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout: prepopulate_block_cache: 0 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout: initial_auto_readahead_size: 8192 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout: num_file_reads_for_auto_readahead: 2 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.write_buffer_size: 33554432 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_write_buffer_number: 2 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compression: NoCompression 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.bottommost_compression: Disabled 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.prefix_extractor: nullptr 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-19T01:02:35.030 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.num_levels: 7 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compression_opts.level: 32767 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compression_opts.strategy: 0 2024-03-19T01:02:35.031 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compression_opts.enabled: false 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.target_file_size_base: 67108864 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-19T01:02:35.032 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.arena_block_size: 1048576 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.disable_auto_compactions: 0 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-19T01:02:35.033 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.table_properties_collectors: 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.inplace_update_support: 0 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.bloom_locality: 0 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.max_successive_merges: 0 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-19T01:02:35.034 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.paranoid_file_checks: 0 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.force_consistency_checks: 1 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.report_bg_io_stats: 0 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.ttl: 2592000 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.enable_blob_files: false 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.min_blob_size: 0 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.blob_file_size: 268435456 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-19T01:02:35.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-19T01:02:35.036 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.blob_file_starting_level: 0 2024-03-19T01:02:35.036 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-19T01:02:35.036 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-19T01:02:35.036 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-19T01:02:35.036 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d3455e5e-7839-4bf2-be99-30e2d9742d8e 2024-03-19T01:02:35.036 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710810154843993, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-19T01:02:35.036 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-19T01:02:35.036 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710810154845040, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99489, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 322, "table_properties": {"data_size": 97461, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13967, "raw_average_key_size": 49, "raw_value_size": 89848, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 281, "num_filter_entries": 281, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710810154, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d3455e5e-7839-4bf2-be99-30e2d9742d8e", "db_session_id": "FXIBM2NSTDC7LDL7XU7Q", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-19T01:02:35.036 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710810154845121, "job": 1, "event": "recovery_finished"} 2024-03-19T01:02:35.036 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-19T01:02:35.036 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-19T01:02:35.036 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fabdc90000 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: DB pointer 0x55fabdc7a000 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** DB Stats ** 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T01:02:35.037 INFO:journalctl@ceph.mon.a.smithi116.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: L0 2/0 98.99 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: Sum 2/0 98.99 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** Compaction Stats [default] ** 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 136.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-19T01:02:35.038 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout: Cumulative compaction: 0.00 GB write, 31.95 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout: Interval compaction: 0.00 GB write, 31.95 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout: Block cache BinnedLRUCache@0x55fabdb9d090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: starting mon.a rank 0 at public addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] at bind addrs [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: mon.a@-1(???) e1 preinit fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: mon.a@-1(???).mds e1 new map 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: mon.a@-1(???).mds e1 print_map 2024-03-19T01:02:35.039 INFO:journalctl@ceph.mon.a.smithi116.stdout: e1 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout: legacy client fscid: -1 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout: 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout: No filesystems configured 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 bash[62682]: a41987fc403771ac6802512e0eec1abdf17d68d4c71ee454a69989dd7de95e31 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 systemd[1]: Started Ceph mon.a for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: monmap epoch 1 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:02:35.040 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: last_changed 2024-03-19T01:02:27.623031+0000 2024-03-19T01:02:35.041 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: created 2024-03-19T01:02:27.623031+0000 2024-03-19T01:02:35.041 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: min_mon_release 19 (squid) 2024-03-19T01:02:35.041 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: election_strategy: 1 2024-03-19T01:02:35.041 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T01:02:35.041 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: fsmap 2024-03-19T01:02:35.041 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: osdmap e1: 0 total, 0 up, 0 in 2024-03-19T01:02:35.041 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:34 smithi116 ceph-mon[62701]: mgrmap e1: no daemons active 2024-03-19T01:02:36.103 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.352+0000 7fb72c6ec700 1 Processor -- start 2024-03-19T01:02:36.103 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.352+0000 7fb72c6ec700 1 -- start start 2024-03-19T01:02:36.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.352+0000 7fb72c6ec700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 0x7fb724106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:36.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.352+0000 7fb72c6ec700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb72407bdf0 con 0x7fb724106510 2024-03-19T01:02:36.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.352+0000 7fb72a488700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 0x7fb724106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:36.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.352+0000 7fb72a488700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 0x7fb724106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42054/0 (socket says 172.21.15.116:42054) 2024-03-19T01:02:36.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.352+0000 7fb72a488700 1 -- 172.21.15.116:0/962049784 learned_addr learned my addr 172.21.15.116:0/962049784 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:36.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.353+0000 7fb72a488700 1 -- 172.21.15.116:0/962049784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb724106e70 con 0x7fb724106510 2024-03-19T01:02:36.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.353+0000 7fb72a488700 1 --2- 172.21.15.116:0/962049784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 0x7fb724106930 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb7200098a0 tx=0x7fb720009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685f87e80e2ab45e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:36.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.353+0000 7fb729486700 1 -- 172.21.15.116:0/962049784 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb720004160 con 0x7fb724106510 2024-03-19T01:02:36.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.353+0000 7fb729486700 1 -- 172.21.15.116:0/962049784 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb720004300 con 0x7fb724106510 2024-03-19T01:02:36.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.353+0000 7fb729486700 1 -- 172.21.15.116:0/962049784 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb720011b50 con 0x7fb724106510 2024-03-19T01:02:36.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.354+0000 7fb72c6ec700 1 -- 172.21.15.116:0/962049784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 msgr2=0x7fb724106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:36.104 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.354+0000 7fb72c6ec700 1 --2- 172.21.15.116:0/962049784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 0x7fb724106930 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb7200098a0 tx=0x7fb720009bb0 comp rx=0 tx=0).stop 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.354+0000 7fb72c6ec700 1 -- 172.21.15.116:0/962049784 shutdown_connections 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.354+0000 7fb72c6ec700 1 --2- 172.21.15.116:0/962049784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 0x7fb724106930 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.354+0000 7fb72c6ec700 1 -- 172.21.15.116:0/962049784 >> 172.21.15.116:0/962049784 conn(0x7fb724102130 msgr2=0x7fb724104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.354+0000 7fb72c6ec700 1 -- 172.21.15.116:0/962049784 shutdown_connections 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.354+0000 7fb72c6ec700 1 -- 172.21.15.116:0/962049784 wait complete. 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.354+0000 7fb72c6ec700 1 Processor -- start 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.354+0000 7fb72c6ec700 1 -- start start 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.354+0000 7fb72c6ec700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 0x7fb72419de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.354+0000 7fb72c6ec700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb724107e90 con 0x7fb724106510 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.355+0000 7fb72a488700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 0x7fb72419de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.355+0000 7fb72a488700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 0x7fb72419de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42068/0 (socket says 172.21.15.116:42068) 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.355+0000 7fb72a488700 1 -- 172.21.15.116:0/3115846092 learned_addr learned my addr 172.21.15.116:0/3115846092 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.355+0000 7fb72a488700 1 -- 172.21.15.116:0/3115846092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb72419e360 con 0x7fb724106510 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.355+0000 7fb72a488700 1 --2- 172.21.15.116:0/3115846092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 0x7fb72419de20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb720009fb0 tx=0x7fb720011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:36.105 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.355+0000 7fb71b7fe700 1 -- 172.21.15.116:0/3115846092 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb7200038b0 con 0x7fb724106510 2024-03-19T01:02:36.106 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.355+0000 7fb72c6ec700 1 -- 172.21.15.116:0/3115846092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb72419e5f0 con 0x7fb724106510 2024-03-19T01:02:36.106 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.355+0000 7fb71b7fe700 1 -- 172.21.15.116:0/3115846092 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb720011e50 con 0x7fb724106510 2024-03-19T01:02:36.106 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.355+0000 7fb71b7fe700 1 -- 172.21.15.116:0/3115846092 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb720019420 con 0x7fb724106510 2024-03-19T01:02:36.106 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.355+0000 7fb72c6ec700 1 -- 172.21.15.116:0/3115846092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7241a12e0 con 0x7fb724106510 2024-03-19T01:02:36.106 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.355+0000 7fb71b7fe700 1 -- 172.21.15.116:0/3115846092 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb720019ad0 con 0x7fb724106510 2024-03-19T01:02:36.106 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.355+0000 7fb71b7fe700 1 -- 172.21.15.116:0/3115846092 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb720024730 con 0x7fb724106510 2024-03-19T01:02:36.106 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.356+0000 7fb72c6ec700 1 -- 172.21.15.116:0/3115846092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb724063940 con 0x7fb724106510 2024-03-19T01:02:36.106 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.358+0000 7fb71b7fe700 1 -- 172.21.15.116:0/3115846092 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb720043dd0 con 0x7fb724106510 2024-03-19T01:02:36.106 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.429+0000 7fb72c6ec700 1 -- 172.21.15.116:0/3115846092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fb72402de20 con 0x7fb724106510 2024-03-19T01:02:36.106 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.431+0000 7fb71b7fe700 1 -- 172.21.15.116:0/3115846092 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fb720032030 con 0x7fb724106510 2024-03-19T01:02:36.106 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.432+0000 7fb72c6ec700 1 -- 172.21.15.116:0/3115846092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 msgr2=0x7fb72419de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:36.106 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.432+0000 7fb72c6ec700 1 --2- 172.21.15.116:0/3115846092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 0x7fb72419de20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb720009fb0 tx=0x7fb720011bf0 comp rx=0 tx=0).stop 2024-03-19T01:02:36.106 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.432+0000 7fb72c6ec700 1 -- 172.21.15.116:0/3115846092 shutdown_connections 2024-03-19T01:02:36.107 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.432+0000 7fb72c6ec700 1 --2- 172.21.15.116:0/3115846092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb724106510 0x7fb72419de20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:36.107 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.432+0000 7fb72c6ec700 1 -- 172.21.15.116:0/3115846092 >> 172.21.15.116:0/3115846092 conn(0x7fb724102130 msgr2=0x7fb724194ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:36.107 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.432+0000 7fb72c6ec700 1 -- 172.21.15.116:0/3115846092 shutdown_connections 2024-03-19T01:02:36.107 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:35.432+0000 7fb72c6ec700 1 -- 172.21.15.116:0/3115846092 wait complete. 2024-03-19T01:02:36.107 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-19T01:02:36.120 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-19T01:02:36.120 INFO:teuthology.orchestra.run.smithi116.stdout:Creating mgr... 2024-03-19T01:02:36.120 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-19T01:02:36.121 INFO:teuthology.orchestra.run.smithi116.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-19T01:02:36.370 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.x 2024-03-19T01:02:36.370 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Failed to reset failed state of unit ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-19T01:02:36.520 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-35912c14-e58c-11ee-95c9-87774f69a715.target.wants/ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-35912c14-e58c-11ee-95c9-87774f69a715@.service. 2024-03-19T01:02:36.547 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:36 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3115846092' entity='client.admin' 2024-03-19T01:02:36.548 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:36 smithi116 systemd[1]: Starting Ceph mgr.x for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:02:36.853 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:36 smithi116 podman[62921]: 2024-03-19 01:02:36.763965069 +0000 UTC m=+0.113604233 container create 1e8e8ed90d700f79cd94f118089968db39a86a6092d556ff6d425d3c49cce8c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-19T01:02:36.853 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:36 smithi116 podman[62921]: 2024-03-19 01:02:36.668095847 +0000 UTC m=+0.017735012 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 2024-03-19T01:02:37.029 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T01:02:37.030 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-19T01:02:37.036 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T01:02:37.037 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-19T01:02:37.037 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-19T01:02:37.037 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to enable service . firewalld.service is not available 2024-03-19T01:02:37.045 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T01:02:37.045 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-19T01:02:37.052 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T01:02:37.052 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-19T01:02:37.052 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-19T01:02:37.052 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-19T01:02:37.052 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr to start... 2024-03-19T01:02:37.052 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr... 2024-03-19T01:02:37.155 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:36 smithi116 podman[62921]: 2024-03-19 01:02:36.941379532 +0000 UTC m=+0.291018698 container init 1e8e8ed90d700f79cd94f118089968db39a86a6092d556ff6d425d3c49cce8c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux ) 2024-03-19T01:02:37.155 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:36 smithi116 podman[62921]: 2024-03-19 01:02:36.943748823 +0000 UTC m=+0.293387988 container start 1e8e8ed90d700f79cd94f118089968db39a86a6092d556ff6d425d3c49cce8c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-19T01:02:37.155 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:37 smithi116 bash[62921]: 1e8e8ed90d700f79cd94f118089968db39a86a6092d556ff6d425d3c49cce8c3 2024-03-19T01:02:37.155 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:37 smithi116 systemd[1]: Started Ceph mgr.x for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:02:37.889 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:37 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/538856567' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T01:02:37.889 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:37.501+0000 7f6324144200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T01:02:37.889 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:37.599+0000 7f6324144200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T01:02:38.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T01:02:38.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T01:02:38.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "35912c14-e58c-11ee-95c9-87774f69a715", 2024-03-19T01:02:38.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T01:02:38.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T01:02:38.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T01:02:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T01:02:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T01:02:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T01:02:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-19T01:02:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T01:02:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T01:02:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T01:02:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T01:02:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-19T01:02:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T01:02:38.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T01:02:38.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T01:02:38.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T01:02:38.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:38.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T01:02:38.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T01:02:38.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T01:02:38.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T01:02:38.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T01:02:38.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T01:02:38.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T01:02:38.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T01:02:38.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:38.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T01:02:38.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T01:02:38.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T01:02:38.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T01:02:38.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T01:02:38.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T01:02:38.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T01:02:38.188 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T01:02:38.189 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T01:02:30.471552+0000", 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.477+0000 7f2c6f1b6700 1 Processor -- start 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.477+0000 7f2c6f1b6700 1 -- start start 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.478+0000 7f2c6f1b6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 0x7f2c6810ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.478+0000 7f2c6f1b6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6807bdf0 con 0x7f2c68108730 2024-03-19T01:02:38.190 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.478+0000 7f2c6cf52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 0x7f2c6810ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.478+0000 7f2c6cf52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 0x7f2c6810ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58120/0 (socket says 172.21.15.116:58120) 2024-03-19T01:02:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.478+0000 7f2c6cf52700 1 -- 172.21.15.116:0/1393447650 learned_addr learned my addr 172.21.15.116:0/1393447650 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.478+0000 7f2c6cf52700 1 -- 172.21.15.116:0/1393447650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6810b060 con 0x7f2c68108730 2024-03-19T01:02:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.478+0000 7f2c6cf52700 1 --2- 172.21.15.116:0/1393447650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 0x7f2c6810ab20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0098a0 tx=0x7f2c5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ffee63e6fb82f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.478+0000 7f2c677fe700 1 -- 172.21.15.116:0/1393447650 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c5c004160 con 0x7f2c68108730 2024-03-19T01:02:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.478+0000 7f2c677fe700 1 -- 172.21.15.116:0/1393447650 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2c5c004300 con 0x7f2c68108730 2024-03-19T01:02:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.479+0000 7f2c677fe700 1 -- 172.21.15.116:0/1393447650 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c5c011b50 con 0x7f2c68108730 2024-03-19T01:02:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.479+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/1393447650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 msgr2=0x7f2c6810ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.479+0000 7f2c6f1b6700 1 --2- 172.21.15.116:0/1393447650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 0x7f2c6810ab20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0098a0 tx=0x7f2c5c009bb0 comp rx=0 tx=0).stop 2024-03-19T01:02:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.479+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/1393447650 shutdown_connections 2024-03-19T01:02:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.479+0000 7f2c6f1b6700 1 --2- 172.21.15.116:0/1393447650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 0x7f2c6810ab20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.479+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/1393447650 >> 172.21.15.116:0/1393447650 conn(0x7f2c68102140 msgr2=0x7f2c68104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:38.191 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.479+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/1393447650 shutdown_connections 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.479+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/1393447650 wait complete. 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.480+0000 7f2c6f1b6700 1 Processor -- start 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.480+0000 7f2c6f1b6700 1 -- start start 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.480+0000 7f2c6f1b6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 0x7f2c6819bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.480+0000 7f2c6f1b6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c6810c010 con 0x7f2c68108730 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.480+0000 7f2c6cf52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 0x7f2c6819bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.480+0000 7f2c6cf52700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 0x7f2c6819bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58132/0 (socket says 172.21.15.116:58132) 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.480+0000 7f2c6cf52700 1 -- 172.21.15.116:0/538856567 learned_addr learned my addr 172.21.15.116:0/538856567 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.480+0000 7f2c6cf52700 1 -- 172.21.15.116:0/538856567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6819c120 con 0x7f2c68108730 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.481+0000 7f2c6cf52700 1 --2- 172.21.15.116:0/538856567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 0x7f2c6819bbe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c011b60 tx=0x7f2c5c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.481+0000 7f2c65ffb700 1 -- 172.21.15.116:0/538856567 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c5c003970 con 0x7f2c68108730 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.481+0000 7f2c65ffb700 1 -- 172.21.15.116:0/538856567 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2c5c003b10 con 0x7f2c68108730 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.481+0000 7f2c65ffb700 1 -- 172.21.15.116:0/538856567 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2c5c024610 con 0x7f2c68108730 2024-03-19T01:02:38.192 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.481+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/538856567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c6819c3b0 con 0x7f2c68108730 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.481+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/538856567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c6819f0a0 con 0x7f2c68108730 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.481+0000 7f2c65ffb700 1 -- 172.21.15.116:0/538856567 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2c5c024910 con 0x7f2c68108730 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.481+0000 7f2c65ffb700 1 -- 172.21.15.116:0/538856567 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2c5c011e00 con 0x7f2c68108730 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.482+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/538856567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c68063940 con 0x7f2c68108730 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.484+0000 7f2c65ffb700 1 -- 172.21.15.116:0/538856567 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2c5c01a7c0 con 0x7f2c68108730 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.556+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/538856567 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f2c6819f270 con 0x7f2c68108730 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.557+0000 7f2c65ffb700 1 -- 172.21.15.116:0/538856567 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f2c5c02de10 con 0x7f2c68108730 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.558+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/538856567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 msgr2=0x7f2c6819bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.558+0000 7f2c6f1b6700 1 --2- 172.21.15.116:0/538856567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 0x7f2c6819bbe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c011b60 tx=0x7f2c5c011c40 comp rx=0 tx=0).stop 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.558+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/538856567 shutdown_connections 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.558+0000 7f2c6f1b6700 1 --2- 172.21.15.116:0/538856567 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c68108730 0x7f2c6819bbe0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.558+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/538856567 >> 172.21.15.116:0/538856567 conn(0x7f2c68102140 msgr2=0x7f2c68102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.558+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/538856567 shutdown_connections 2024-03-19T01:02:38.193 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:37.558+0000 7f2c6f1b6700 1 -- 172.21.15.116:0/538856567 wait complete. 2024-03-19T01:02:38.194 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (1/15)... 2024-03-19T01:02:38.481 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:38 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:38.251+0000 7f6324144200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T01:02:38.481 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:38 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:38.342+0000 7f6324144200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T01:02:38.481 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:38 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:38.480+0000 7f6324144200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T01:02:38.905 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:38 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:38.574+0000 7f6324144200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T01:02:38.905 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:38 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:38.655+0000 7f6324144200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T01:02:39.197 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:38 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:38.913+0000 7f6324144200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T01:02:39.197 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:39 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:39.019+0000 7f6324144200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T01:02:39.197 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:39 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:39.104+0000 7f6324144200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T01:02:39.197 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:39 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:39.196+0000 7f6324144200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T01:02:39.655 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:39 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:39.378+0000 7f6324144200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T01:02:40.572 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:40 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:40.344+0000 7f6324144200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T01:02:40.573 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:40 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:40.477+0000 7f6324144200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T01:02:40.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:40 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4202713858' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T01:02:40.905 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:40 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:40.821+0000 7f6324144200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T01:02:41.289 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:40 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:40.909+0000 7f6324144200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T01:02:41.289 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:41.104+0000 7f6324144200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:02:41.289 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:41.288+0000 7f6324144200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T01:02:41.830 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:41.570+0000 7f6324144200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:02:41.830 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:41.829+0000 7f6324144200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T01:02:42.155 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:42 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:42.067+0000 7f6324144200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T01:02:43.155 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:42 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:42.869+0000 7f6324144200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T01:02:43.762 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:43.444+0000 7f6324144200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T01:02:43.762 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:43.544+0000 7f6324144200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T01:02:43.762 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:43.761+0000 7f6324144200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T01:02:44.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-mon[62701]: Activating manager daemon x 2024-03-19T01:02:44.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-mon[62701]: mgrmap e2: x(active, starting, since 0.00294816s) 2024-03-19T01:02:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-mon[62701]: from='mgr.14100 172.21.15.116:0/1255153082' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T01:02:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-mon[62701]: from='mgr.14100 172.21.15.116:0/1255153082' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T01:02:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-mon[62701]: from='mgr.14100 172.21.15.116:0/1255153082' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T01:02:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-mon[62701]: from='mgr.14100 172.21.15.116:0/1255153082' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:02:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-mon[62701]: from='mgr.14100 172.21.15.116:0/1255153082' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T01:02:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-mon[62701]: Manager daemon x is now available 2024-03-19T01:02:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-mon[62701]: from='mgr.14100 172.21.15.116:0/1255153082' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T01:02:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-mon[62701]: from='mgr.14100 172.21.15.116:0/1255153082' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T01:02:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-mon[62701]: from='mgr.14100 172.21.15.116:0/1255153082' entity='mgr.x' 2024-03-19T01:02:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-mon[62701]: from='mgr.14100 172.21.15.116:0/1255153082' entity='mgr.x' 2024-03-19T01:02:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:43 smithi116 ceph-mon[62701]: from='mgr.14100 172.21.15.116:0/1255153082' entity='mgr.x' 2024-03-19T01:02:46.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:45 smithi116 ceph-mon[62701]: mgrmap e3: x(active, since 1.00762s) 2024-03-19T01:02:46.205 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T01:02:46.207 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T01:02:46.207 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "35912c14-e58c-11ee-95c9-87774f69a715", 2024-03-19T01:02:46.208 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T01:02:46.208 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T01:02:46.208 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T01:02:46.208 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T01:02:46.208 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:46.208 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T01:02:46.208 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T01:02:46.208 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-19T01:02:46.208 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T01:02:46.208 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T01:02:46.209 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T01:02:46.209 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T01:02:46.209 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-19T01:02:46.209 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T01:02:46.209 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T01:02:46.209 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T01:02:46.209 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T01:02:46.209 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:46.209 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T01:02:46.209 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T01:02:46.209 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T01:02:46.209 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T01:02:46.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T01:02:46.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T01:02:30.471552+0000", 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.571+0000 7fea67959700 1 Processor -- start 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.572+0000 7fea67959700 1 -- start start 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.572+0000 7fea67959700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 0x7fea6010ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.572+0000 7fea67959700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea6007bdf0 con 0x7fea60108750 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.572+0000 7fea656f5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 0x7fea6010ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.572+0000 7fea656f5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 0x7fea6010ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58142/0 (socket says 172.21.15.116:58142) 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.572+0000 7fea656f5700 1 -- 172.21.15.116:0/2791191112 learned_addr learned my addr 172.21.15.116:0/2791191112 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:46.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.572+0000 7fea656f5700 1 -- 172.21.15.116:0/2791191112 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea6010b080 con 0x7fea60108750 2024-03-19T01:02:46.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.573+0000 7fea656f5700 1 --2- 172.21.15.116:0/2791191112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 0x7fea6010ab40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fea5c0098a0 tx=0x7fea5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=510f703cd4e70261 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:46.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.573+0000 7fea57fff700 1 -- 172.21.15.116:0/2791191112 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea5c004160 con 0x7fea60108750 2024-03-19T01:02:46.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.573+0000 7fea57fff700 1 -- 172.21.15.116:0/2791191112 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fea5c004300 con 0x7fea60108750 2024-03-19T01:02:46.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.573+0000 7fea57fff700 1 -- 172.21.15.116:0/2791191112 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea5c011b50 con 0x7fea60108750 2024-03-19T01:02:46.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.573+0000 7fea67959700 1 -- 172.21.15.116:0/2791191112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 msgr2=0x7fea6010ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:46.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.573+0000 7fea67959700 1 --2- 172.21.15.116:0/2791191112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 0x7fea6010ab40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fea5c0098a0 tx=0x7fea5c009bb0 comp rx=0 tx=0).stop 2024-03-19T01:02:46.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.573+0000 7fea67959700 1 -- 172.21.15.116:0/2791191112 shutdown_connections 2024-03-19T01:02:46.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.573+0000 7fea67959700 1 --2- 172.21.15.116:0/2791191112 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 0x7fea6010ab40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:46.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.573+0000 7fea67959700 1 -- 172.21.15.116:0/2791191112 >> 172.21.15.116:0/2791191112 conn(0x7fea60102140 msgr2=0x7fea60104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:46.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.574+0000 7fea67959700 1 -- 172.21.15.116:0/2791191112 shutdown_connections 2024-03-19T01:02:46.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.574+0000 7fea67959700 1 -- 172.21.15.116:0/2791191112 wait complete. 2024-03-19T01:02:46.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.574+0000 7fea67959700 1 Processor -- start 2024-03-19T01:02:46.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.574+0000 7fea67959700 1 -- start start 2024-03-19T01:02:46.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.574+0000 7fea67959700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 0x7fea6019bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:46.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.574+0000 7fea67959700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea6010c030 con 0x7fea60108750 2024-03-19T01:02:46.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.574+0000 7fea656f5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 0x7fea6019bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:46.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.574+0000 7fea656f5700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 0x7fea6019bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58150/0 (socket says 172.21.15.116:58150) 2024-03-19T01:02:46.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.574+0000 7fea656f5700 1 -- 172.21.15.116:0/4202713858 learned_addr learned my addr 172.21.15.116:0/4202713858 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:46.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.575+0000 7fea656f5700 1 -- 172.21.15.116:0/4202713858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea6019c140 con 0x7fea60108750 2024-03-19T01:02:46.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.575+0000 7fea656f5700 1 --2- 172.21.15.116:0/4202713858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 0x7fea6019bc00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fea5c009fb0 tx=0x7fea5c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:46.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.575+0000 7fea567fc700 1 -- 172.21.15.116:0/4202713858 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea5c0038b0 con 0x7fea60108750 2024-03-19T01:02:46.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.575+0000 7fea567fc700 1 -- 172.21.15.116:0/4202713858 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fea5c011e50 con 0x7fea60108750 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.575+0000 7fea67959700 1 -- 172.21.15.116:0/4202713858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea6019c3d0 con 0x7fea60108750 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.575+0000 7fea567fc700 1 -- 172.21.15.116:0/4202713858 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea5c019420 con 0x7fea60108750 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.575+0000 7fea67959700 1 -- 172.21.15.116:0/4202713858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea6019f0c0 con 0x7fea60108750 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.575+0000 7fea567fc700 1 -- 172.21.15.116:0/4202713858 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fea5c019ad0 con 0x7fea60108750 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.575+0000 7fea567fc700 1 -- 172.21.15.116:0/4202713858 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fea5c024730 con 0x7fea60108750 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.576+0000 7fea67959700 1 -- 172.21.15.116:0/4202713858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea60063940 con 0x7fea60108750 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.578+0000 7fea567fc700 1 -- 172.21.15.116:0/4202713858 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fea5c043dd0 con 0x7fea60108750 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.651+0000 7fea67959700 1 -- 172.21.15.116:0/4202713858 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fea6019f600 con 0x7fea60108750 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.651+0000 7fea567fc700 1 -- 172.21.15.116:0/4202713858 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fea5c032030 con 0x7fea60108750 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.653+0000 7fea67959700 1 -- 172.21.15.116:0/4202713858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 msgr2=0x7fea6019bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.653+0000 7fea67959700 1 --2- 172.21.15.116:0/4202713858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 0x7fea6019bc00 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fea5c009fb0 tx=0x7fea5c011bf0 comp rx=0 tx=0).stop 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.653+0000 7fea67959700 1 -- 172.21.15.116:0/4202713858 shutdown_connections 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.653+0000 7fea67959700 1 --2- 172.21.15.116:0/4202713858 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea60108750 0x7fea6019bc00 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.653+0000 7fea67959700 1 -- 172.21.15.116:0/4202713858 >> 172.21.15.116:0/4202713858 conn(0x7fea60102140 msgr2=0x7fea60102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:46.223 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.653+0000 7fea67959700 1 -- 172.21.15.116:0/4202713858 shutdown_connections 2024-03-19T01:02:46.224 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:40.653+0000 7fea67959700 1 -- 172.21.15.116:0/4202713858 wait complete. 2024-03-19T01:02:46.224 INFO:teuthology.orchestra.run.smithi116.stdout:mgr not available, waiting (2/15)... 2024-03-19T01:02:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:47 smithi116 ceph-mon[62701]: mgrmap e4: x(active, since 3s) 2024-03-19T01:02:51.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:51 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/890334643' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-19T01:02:51.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T01:02:51.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T01:02:51.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsid": "35912c14-e58c-11ee-95c9-87774f69a715", 2024-03-19T01:02:51.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "health": { 2024-03-19T01:02:51.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-19T01:02:51.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-19T01:02:51.924 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 0 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "a" 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:51.925 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-19T01:02:51.926 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-19T01:02:51.934 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-19T01:02:51.934 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-19T01:02:51.934 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-19T01:02:51.934 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:51.934 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-19T01:02:51.934 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T01:02:51.934 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-19T01:02:51.934 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-19T01:02:51.934 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:51.934 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "restful" 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ], 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "modified": "2024-03-19T01:02:30.471552+0000", 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout }, 2024-03-19T01:02:51.935 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-19T01:02:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T01:02:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.103+0000 7fd2ede5a700 1 Processor -- start 2024-03-19T01:02:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.103+0000 7fd2ede5a700 1 -- start start 2024-03-19T01:02:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.103+0000 7fd2ede5a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 0x7fd2e8106fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.103+0000 7fd2ede5a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e8107590 con 0x7fd2e8106ba0 2024-03-19T01:02:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.103+0000 7fd2e77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 0x7fd2e8106fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.103+0000 7fd2e77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 0x7fd2e8106fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55712/0 (socket says 172.21.15.116:55712) 2024-03-19T01:02:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.103+0000 7fd2e77fe700 1 -- 172.21.15.116:0/41898187 learned_addr learned my addr 172.21.15.116:0/41898187 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.104+0000 7fd2e77fe700 1 -- 172.21.15.116:0/41898187 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e8107da0 con 0x7fd2e8106ba0 2024-03-19T01:02:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.104+0000 7fd2e77fe700 1 --2- 172.21.15.116:0/41898187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 0x7fd2e8106fc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80098a0 tx=0x7fd2d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d6589c9d4eb1fc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.104+0000 7fd2e67fc700 1 -- 172.21.15.116:0/41898187 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2d8004160 con 0x7fd2e8106ba0 2024-03-19T01:02:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.104+0000 7fd2e67fc700 1 -- 172.21.15.116:0/41898187 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd2d8004300 con 0x7fd2e8106ba0 2024-03-19T01:02:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.104+0000 7fd2e67fc700 1 -- 172.21.15.116:0/41898187 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2d8011b50 con 0x7fd2e8106ba0 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.105+0000 7fd2ede5a700 1 -- 172.21.15.116:0/41898187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 msgr2=0x7fd2e8106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.105+0000 7fd2ede5a700 1 --2- 172.21.15.116:0/41898187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 0x7fd2e8106fc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80098a0 tx=0x7fd2d8009bb0 comp rx=0 tx=0).stop 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.105+0000 7fd2ede5a700 1 -- 172.21.15.116:0/41898187 shutdown_connections 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.105+0000 7fd2ede5a700 1 --2- 172.21.15.116:0/41898187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 0x7fd2e8106fc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.105+0000 7fd2ede5a700 1 -- 172.21.15.116:0/41898187 >> 172.21.15.116:0/41898187 conn(0x7fd2e8102140 msgr2=0x7fd2e8104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.105+0000 7fd2ede5a700 1 -- 172.21.15.116:0/41898187 shutdown_connections 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.105+0000 7fd2ede5a700 1 -- 172.21.15.116:0/41898187 wait complete. 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.106+0000 7fd2ede5a700 1 Processor -- start 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.106+0000 7fd2ede5a700 1 -- start start 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.106+0000 7fd2ede5a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 0x7fd2e819ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.106+0000 7fd2ede5a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e810b150 con 0x7fd2e8106ba0 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.106+0000 7fd2e77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 0x7fd2e819ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.106+0000 7fd2e77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 0x7fd2e819ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55728/0 (socket says 172.21.15.116:55728) 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.106+0000 7fd2e77fe700 1 -- 172.21.15.116:0/890334643 learned_addr learned my addr 172.21.15.116:0/890334643 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:51.937 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.107+0000 7fd2e77fe700 1 -- 172.21.15.116:0/890334643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e819b0a0 con 0x7fd2e8106ba0 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.107+0000 7fd2e77fe700 1 --2- 172.21.15.116:0/890334643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 0x7fd2e819ab60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8011850 tx=0x7fd2d8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.107+0000 7fd2e4ff9700 1 -- 172.21.15.116:0/890334643 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2d8003af0 con 0x7fd2e8106ba0 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.107+0000 7fd2e4ff9700 1 -- 172.21.15.116:0/890334643 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd2d8020070 con 0x7fd2e8106ba0 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.107+0000 7fd2ede5a700 1 -- 172.21.15.116:0/890334643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2e819b330 con 0x7fd2e8106ba0 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.107+0000 7fd2e4ff9700 1 -- 172.21.15.116:0/890334643 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd2d80195a0 con 0x7fd2e8106ba0 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.107+0000 7fd2ede5a700 1 -- 172.21.15.116:0/890334643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2e819b750 con 0x7fd2e8106ba0 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.108+0000 7fd2ede5a700 1 -- 172.21.15.116:0/890334643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2e8063940 con 0x7fd2e8106ba0 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.108+0000 7fd2e4ff9700 1 -- 172.21.15.116:0/890334643 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fd2d80198a0 con 0x7fd2e8106ba0 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.108+0000 7fd2e4ff9700 1 --2- 172.21.15.116:0/890334643 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fd2d003d2e0 0x7fd2d003f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.108+0000 7fd2e4ff9700 1 -- 172.21.15.116:0/890334643 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd2d8011e00 con 0x7fd2e8106ba0 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.109+0000 7fd2e6ffd700 1 --2- 172.21.15.116:0/890334643 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fd2d003d2e0 0x7fd2d003f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.109+0000 7fd2e6ffd700 1 --2- 172.21.15.116:0/890334643 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fd2d003d2e0 0x7fd2d003f7a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc006fd0 tx=0x7fd2dc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.114+0000 7fd2e4ff9700 1 -- 172.21.15.116:0/890334643 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd2d8003c90 con 0x7fd2e8106ba0 2024-03-19T01:02:51.938 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.398+0000 7fd2ede5a700 1 -- 172.21.15.116:0/890334643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd2e807b630 con 0x7fd2e8106ba0 2024-03-19T01:02:51.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.398+0000 7fd2e4ff9700 1 -- 172.21.15.116:0/890334643 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fd2d8022d90 con 0x7fd2e8106ba0 2024-03-19T01:02:51.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.400+0000 7fd2ede5a700 1 -- 172.21.15.116:0/890334643 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fd2d003d2e0 msgr2=0x7fd2d003f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:51.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.400+0000 7fd2ede5a700 1 --2- 172.21.15.116:0/890334643 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fd2d003d2e0 0x7fd2d003f7a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc006fd0 tx=0x7fd2dc006e40 comp rx=0 tx=0).stop 2024-03-19T01:02:51.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.400+0000 7fd2ede5a700 1 -- 172.21.15.116:0/890334643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 msgr2=0x7fd2e819ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:51.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.400+0000 7fd2ede5a700 1 --2- 172.21.15.116:0/890334643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 0x7fd2e819ab60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8011850 tx=0x7fd2d8011c10 comp rx=0 tx=0).stop 2024-03-19T01:02:51.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.400+0000 7fd2ede5a700 1 -- 172.21.15.116:0/890334643 shutdown_connections 2024-03-19T01:02:51.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.400+0000 7fd2ede5a700 1 --2- 172.21.15.116:0/890334643 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fd2d003d2e0 0x7fd2d003f7a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:51.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.400+0000 7fd2ede5a700 1 --2- 172.21.15.116:0/890334643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2e8106ba0 0x7fd2e819ab60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:51.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.400+0000 7fd2ede5a700 1 -- 172.21.15.116:0/890334643 >> 172.21.15.116:0/890334643 conn(0x7fd2e8102140 msgr2=0x7fd2e8102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:51.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.401+0000 7fd2ede5a700 1 -- 172.21.15.116:0/890334643 shutdown_connections 2024-03-19T01:02:51.939 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:51.401+0000 7fd2ede5a700 1 -- 172.21.15.116:0/890334643 wait complete. 2024-03-19T01:02:51.939 INFO:teuthology.orchestra.run.smithi116.stdout:mgr is available 2024-03-19T01:02:52.832 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:52 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/418595783' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-19T01:02:53.083 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T01:02:53.083 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [global] 2024-03-19T01:02:53.084 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout fsid = 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:02:53.084 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-19T01:02:53.084 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-19T01:02:53.084 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-19T01:02:53.084 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-19T01:02:53.084 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T01:02:53.084 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-19T01:02:53.084 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-19T01:02:53.084 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 2024-03-19T01:02:53.084 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout [osd] 2024-03-19T01:02:53.084 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-19T01:02:53.084 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-19T01:02:53.084 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-19T01:02:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-19T01:02:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-19T01:02:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.326+0000 7fd29f09e700 1 Processor -- start 2024-03-19T01:02:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.326+0000 7fd29f09e700 1 -- start start 2024-03-19T01:02:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.326+0000 7fd29f09e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 0x7fd298106fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.326+0000 7fd29f09e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd298107590 con 0x7fd298106ba0 2024-03-19T01:02:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.327+0000 7fd29ce3a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 0x7fd298106fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.327+0000 7fd29ce3a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 0x7fd298106fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55732/0 (socket says 172.21.15.116:55732) 2024-03-19T01:02:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.327+0000 7fd29ce3a700 1 -- 172.21.15.116:0/236498454 learned_addr learned my addr 172.21.15.116:0/236498454 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.327+0000 7fd29ce3a700 1 -- 172.21.15.116:0/236498454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd298107da0 con 0x7fd298106ba0 2024-03-19T01:02:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.327+0000 7fd29ce3a700 1 --2- 172.21.15.116:0/236498454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 0x7fd298106fc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd28c0098a0 tx=0x7fd28c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f37d6d91950829 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.327+0000 7fd2977fe700 1 -- 172.21.15.116:0/236498454 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd28c004160 con 0x7fd298106ba0 2024-03-19T01:02:53.085 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.327+0000 7fd2977fe700 1 -- 172.21.15.116:0/236498454 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd28c004300 con 0x7fd298106ba0 2024-03-19T01:02:53.086 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.327+0000 7fd2977fe700 1 -- 172.21.15.116:0/236498454 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd28c011b50 con 0x7fd298106ba0 2024-03-19T01:02:53.086 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.328+0000 7fd29f09e700 1 -- 172.21.15.116:0/236498454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 msgr2=0x7fd298106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:53.086 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.328+0000 7fd29f09e700 1 --2- 172.21.15.116:0/236498454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 0x7fd298106fc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd28c0098a0 tx=0x7fd28c009bb0 comp rx=0 tx=0).stop 2024-03-19T01:02:53.086 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.328+0000 7fd29f09e700 1 -- 172.21.15.116:0/236498454 shutdown_connections 2024-03-19T01:02:53.086 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.328+0000 7fd29f09e700 1 --2- 172.21.15.116:0/236498454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 0x7fd298106fc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:53.086 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.328+0000 7fd29f09e700 1 -- 172.21.15.116:0/236498454 >> 172.21.15.116:0/236498454 conn(0x7fd298102140 msgr2=0x7fd298104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:53.086 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.328+0000 7fd29f09e700 1 -- 172.21.15.116:0/236498454 shutdown_connections 2024-03-19T01:02:53.086 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.328+0000 7fd29f09e700 1 -- 172.21.15.116:0/236498454 wait complete. 2024-03-19T01:02:53.086 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.328+0000 7fd29f09e700 1 Processor -- start 2024-03-19T01:02:53.086 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.329+0000 7fd29f09e700 1 -- start start 2024-03-19T01:02:53.086 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.329+0000 7fd29f09e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 0x7fd29819bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:53.086 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.329+0000 7fd29f09e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd29810b150 con 0x7fd298106ba0 2024-03-19T01:02:53.086 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.329+0000 7fd29ce3a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 0x7fd29819bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.329+0000 7fd29ce3a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 0x7fd29819bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55746/0 (socket says 172.21.15.116:55746) 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.329+0000 7fd29ce3a700 1 -- 172.21.15.116:0/418595783 learned_addr learned my addr 172.21.15.116:0/418595783 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.329+0000 7fd29ce3a700 1 -- 172.21.15.116:0/418595783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd29819c1e0 con 0x7fd298106ba0 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.329+0000 7fd29ce3a700 1 --2- 172.21.15.116:0/418595783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 0x7fd29819bca0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd28c009fb0 tx=0x7fd28c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.329+0000 7fd295ffb700 1 -- 172.21.15.116:0/418595783 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd28c0038b0 con 0x7fd298106ba0 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.329+0000 7fd295ffb700 1 -- 172.21.15.116:0/418595783 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd28c011e50 con 0x7fd298106ba0 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.329+0000 7fd29f09e700 1 -- 172.21.15.116:0/418595783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd29819c470 con 0x7fd298106ba0 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.329+0000 7fd295ffb700 1 -- 172.21.15.116:0/418595783 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd28c019550 con 0x7fd298106ba0 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.330+0000 7fd29f09e700 1 -- 172.21.15.116:0/418595783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd29819f160 con 0x7fd298106ba0 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.330+0000 7fd295ffb700 1 -- 172.21.15.116:0/418595783 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7fd28c019850 con 0x7fd298106ba0 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.331+0000 7fd29f09e700 1 -- 172.21.15.116:0/418595783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd298063940 con 0x7fd298106ba0 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.331+0000 7fd295ffb700 1 --2- 172.21.15.116:0/418595783 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fd288038e80 0x7fd28803b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.331+0000 7fd295ffb700 1 -- 172.21.15.116:0/418595783 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd28c04d120 con 0x7fd298106ba0 2024-03-19T01:02:53.087 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.331+0000 7fd297fff700 1 --2- 172.21.15.116:0/418595783 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fd288038e80 0x7fd28803b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.331+0000 7fd297fff700 1 --2- 172.21.15.116:0/418595783 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fd288038e80 0x7fd28803b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd284006fd0 tx=0x7fd284006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.336+0000 7fd295ffb700 1 -- 172.21.15.116:0/418595783 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd28c02d960 con 0x7fd298106ba0 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.528+0000 7fd29f09e700 1 -- 172.21.15.116:0/418595783 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fd2980008d0 con 0x7fd298106ba0 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.529+0000 7fd295ffb700 1 -- 172.21.15.116:0/418595783 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7fd28c02d440 con 0x7fd298106ba0 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.531+0000 7fd29f09e700 1 -- 172.21.15.116:0/418595783 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fd288038e80 msgr2=0x7fd28803b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.531+0000 7fd29f09e700 1 --2- 172.21.15.116:0/418595783 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fd288038e80 0x7fd28803b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd284006fd0 tx=0x7fd284006e40 comp rx=0 tx=0).stop 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.531+0000 7fd29f09e700 1 -- 172.21.15.116:0/418595783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 msgr2=0x7fd29819bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.531+0000 7fd29f09e700 1 --2- 172.21.15.116:0/418595783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 0x7fd29819bca0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd28c009fb0 tx=0x7fd28c011bf0 comp rx=0 tx=0).stop 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.531+0000 7fd29f09e700 1 -- 172.21.15.116:0/418595783 shutdown_connections 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.531+0000 7fd29f09e700 1 --2- 172.21.15.116:0/418595783 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fd288038e80 0x7fd28803b340 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.531+0000 7fd29f09e700 1 --2- 172.21.15.116:0/418595783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298106ba0 0x7fd29819bca0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.531+0000 7fd29f09e700 1 -- 172.21.15.116:0/418595783 >> 172.21.15.116:0/418595783 conn(0x7fd298102140 msgr2=0x7fd2981030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.532+0000 7fd29f09e700 1 -- 172.21.15.116:0/418595783 shutdown_connections 2024-03-19T01:02:53.088 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:52.532+0000 7fd29f09e700 1 -- 172.21.15.116:0/418595783 wait complete. 2024-03-19T01:02:53.089 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling cephadm module... 2024-03-19T01:02:54.063 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:53 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3358197177' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-19T01:02:54.064 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:53 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: ignoring --setuser ceph since I am not root 2024-03-19T01:02:54.064 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:53 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: ignoring --setgroup ceph since I am not root 2024-03-19T01:02:54.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.477+0000 7f14bc63a700 1 Processor -- start 2024-03-19T01:02:54.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.477+0000 7f14bc63a700 1 -- start start 2024-03-19T01:02:54.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.477+0000 7f14bc63a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 0x7f14b410aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:54.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.477+0000 7f14bc63a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f14b407be10 con 0x7f14b4108700 2024-03-19T01:02:54.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.477+0000 7f14ba3d6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 0x7f14b410aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:54.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.477+0000 7f14ba3d6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 0x7f14b410aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55750/0 (socket says 172.21.15.116:55750) 2024-03-19T01:02:54.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.477+0000 7f14ba3d6700 1 -- 172.21.15.116:0/981259577 learned_addr learned my addr 172.21.15.116:0/981259577 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:54.304 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.477+0000 7f14ba3d6700 1 -- 172.21.15.116:0/981259577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b410b030 con 0x7f14b4108700 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.478+0000 7f14ba3d6700 1 --2- 172.21.15.116:0/981259577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 0x7f14b410aaf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f14b00098a0 tx=0x7f14b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8a58454b2be32d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.478+0000 7f14b93d4700 1 -- 172.21.15.116:0/981259577 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b0004160 con 0x7f14b4108700 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.478+0000 7f14b93d4700 1 -- 172.21.15.116:0/981259577 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f14b0004300 con 0x7f14b4108700 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.478+0000 7f14b93d4700 1 -- 172.21.15.116:0/981259577 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b0011b50 con 0x7f14b4108700 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.478+0000 7f14bc63a700 1 -- 172.21.15.116:0/981259577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 msgr2=0x7f14b410aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.478+0000 7f14bc63a700 1 --2- 172.21.15.116:0/981259577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 0x7f14b410aaf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f14b00098a0 tx=0x7f14b0009bb0 comp rx=0 tx=0).stop 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.478+0000 7f14bc63a700 1 -- 172.21.15.116:0/981259577 shutdown_connections 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.478+0000 7f14bc63a700 1 --2- 172.21.15.116:0/981259577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 0x7f14b410aaf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.478+0000 7f14bc63a700 1 -- 172.21.15.116:0/981259577 >> 172.21.15.116:0/981259577 conn(0x7f14b4102130 msgr2=0x7f14b4104550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.478+0000 7f14bc63a700 1 -- 172.21.15.116:0/981259577 shutdown_connections 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.478+0000 7f14bc63a700 1 -- 172.21.15.116:0/981259577 wait complete. 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.479+0000 7f14bc63a700 1 Processor -- start 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.479+0000 7f14bc63a700 1 -- start start 2024-03-19T01:02:54.305 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.479+0000 7f14bc63a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 0x7f14b419bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.479+0000 7f14bc63a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f14b410bf70 con 0x7f14b4108700 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.479+0000 7f14ba3d6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 0x7f14b419bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.479+0000 7f14ba3d6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 0x7f14b419bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55764/0 (socket says 172.21.15.116:55764) 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.479+0000 7f14ba3d6700 1 -- 172.21.15.116:0/3358197177 learned_addr learned my addr 172.21.15.116:0/3358197177 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.480+0000 7f14ba3d6700 1 -- 172.21.15.116:0/3358197177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b419c100 con 0x7f14b4108700 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.480+0000 7f14ba3d6700 1 --2- 172.21.15.116:0/3358197177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 0x7f14b419bbc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009fb0 tx=0x7f14b0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.480+0000 7f14ab7fe700 1 -- 172.21.15.116:0/3358197177 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b00038b0 con 0x7f14b4108700 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.480+0000 7f14ab7fe700 1 -- 172.21.15.116:0/3358197177 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f14b0011e50 con 0x7f14b4108700 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.480+0000 7f14ab7fe700 1 -- 172.21.15.116:0/3358197177 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14b00193c0 con 0x7f14b4108700 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.480+0000 7f14bc63a700 1 -- 172.21.15.116:0/3358197177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b419c3f0 con 0x7f14b4108700 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.480+0000 7f14bc63a700 1 -- 172.21.15.116:0/3358197177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b419f080 con 0x7f14b4108700 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.481+0000 7f14ab7fe700 1 -- 172.21.15.116:0/3358197177 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f14b001a430 con 0x7f14b4108700 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.481+0000 7f14ab7fe700 1 --2- 172.21.15.116:0/3358197177 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7f14a0038ed0 0x7f14a003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:54.306 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.481+0000 7f14bc63a700 1 -- 172.21.15.116:0/3358197177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14b4063940 con 0x7f14b4108700 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.481+0000 7f14ab7fe700 1 -- 172.21.15.116:0/3358197177 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f14b004cff0 con 0x7f14b4108700 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.481+0000 7f14b9bd5700 1 --2- 172.21.15.116:0/3358197177 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7f14a0038ed0 0x7f14a003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.481+0000 7f14b9bd5700 1 --2- 172.21.15.116:0/3358197177 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7f14a0038ed0 0x7f14a003b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f14a4006fd0 tx=0x7f14a4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.486+0000 7f14ab7fe700 1 -- 172.21.15.116:0/3358197177 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f14b002e020 con 0x7f14b4108700 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.732+0000 7f14bc63a700 1 -- 172.21.15.116:0/3358197177 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f14b419f800 con 0x7f14b4108700 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.784+0000 7f14ab7fe700 1 -- 172.21.15.116:0/3358197177 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f14b0003a50 con 0x7f14b4108700 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.785+0000 7f14ab7fe700 1 -- 172.21.15.116:0/3358197177 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f14b002ece0 con 0x7f14b4108700 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.787+0000 7f14bc63a700 1 -- 172.21.15.116:0/3358197177 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7f14a0038ed0 msgr2=0x7f14a003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.787+0000 7f14bc63a700 1 --2- 172.21.15.116:0/3358197177 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7f14a0038ed0 0x7f14a003b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f14a4006fd0 tx=0x7f14a4006e40 comp rx=0 tx=0).stop 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.787+0000 7f14bc63a700 1 -- 172.21.15.116:0/3358197177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 msgr2=0x7f14b419bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.787+0000 7f14bc63a700 1 --2- 172.21.15.116:0/3358197177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 0x7f14b419bbc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009fb0 tx=0x7f14b0011bf0 comp rx=0 tx=0).stop 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.787+0000 7f14bc63a700 1 -- 172.21.15.116:0/3358197177 shutdown_connections 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.787+0000 7f14bc63a700 1 --2- 172.21.15.116:0/3358197177 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7f14a0038ed0 0x7f14a003b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:54.307 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.787+0000 7f14bc63a700 1 --2- 172.21.15.116:0/3358197177 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14b4108700 0x7f14b419bbc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:54.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.787+0000 7f14bc63a700 1 -- 172.21.15.116:0/3358197177 >> 172.21.15.116:0/3358197177 conn(0x7f14b4102130 msgr2=0x7f14b4102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:54.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.787+0000 7f14bc63a700 1 -- 172.21.15.116:0/3358197177 shutdown_connections 2024-03-19T01:02:54.308 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:53.787+0000 7f14bc63a700 1 -- 172.21.15.116:0/3358197177 wait complete. 2024-03-19T01:02:54.633 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:54 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:54.347+0000 7f4d104c3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T01:02:54.634 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:54 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:54.442+0000 7f4d104c3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T01:02:54.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:54 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3358197177' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-19T01:02:54.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:54 smithi116 ceph-mon[62701]: mgrmap e5: x(active, since 10s) 2024-03-19T01:02:55.326 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:55 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:55.093+0000 7f4d104c3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T01:02:55.326 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:55 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:55.184+0000 7f4d104c3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T01:02:55.326 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:55 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:55.325+0000 7f4d104c3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T01:02:55.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T01:02:55.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-19T01:02:55.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T01:02:55.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-19T01:02:55.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-19T01:02:55.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T01:02:55.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.731+0000 7fc8443c1700 1 Processor -- start 2024-03-19T01:02:55.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.731+0000 7fc8443c1700 1 -- start start 2024-03-19T01:02:55.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.731+0000 7fc8443c1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 0x7fc83c107490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:55.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.731+0000 7fc8443c1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c107a60 con 0x7fc83c107070 2024-03-19T01:02:55.537 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.731+0000 7fc84215d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 0x7fc83c107490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.731+0000 7fc84215d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 0x7fc83c107490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55796/0 (socket says 172.21.15.116:55796) 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.731+0000 7fc84215d700 1 -- 172.21.15.116:0/1656188824 learned_addr learned my addr 172.21.15.116:0/1656188824 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.732+0000 7fc84215d700 1 -- 172.21.15.116:0/1656188824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83c107d90 con 0x7fc83c107070 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.732+0000 7fc84215d700 1 --2- 172.21.15.116:0/1656188824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 0x7fc83c107490 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc838009ab0 tx=0x7fc838009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48cd015df2f6d072 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.732+0000 7fc84115b700 1 -- 172.21.15.116:0/1656188824 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc838004160 con 0x7fc83c107070 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.732+0000 7fc84115b700 1 -- 172.21.15.116:0/1656188824 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc83800bca0 con 0x7fc83c107070 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.732+0000 7fc84115b700 1 -- 172.21.15.116:0/1656188824 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc838003d60 con 0x7fc83c107070 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.733+0000 7fc8443c1700 1 -- 172.21.15.116:0/1656188824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 msgr2=0x7fc83c107490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.733+0000 7fc8443c1700 1 --2- 172.21.15.116:0/1656188824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 0x7fc83c107490 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc838009ab0 tx=0x7fc838009dc0 comp rx=0 tx=0).stop 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.733+0000 7fc8443c1700 1 -- 172.21.15.116:0/1656188824 shutdown_connections 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.733+0000 7fc8443c1700 1 --2- 172.21.15.116:0/1656188824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 0x7fc83c107490 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.733+0000 7fc8443c1700 1 -- 172.21.15.116:0/1656188824 >> 172.21.15.116:0/1656188824 conn(0x7fc83c1018d0 msgr2=0x7fc83c103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.733+0000 7fc8443c1700 1 -- 172.21.15.116:0/1656188824 shutdown_connections 2024-03-19T01:02:55.538 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.733+0000 7fc8443c1700 1 -- 172.21.15.116:0/1656188824 wait complete. 2024-03-19T01:02:55.539 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.733+0000 7fc8443c1700 1 Processor -- start 2024-03-19T01:02:55.539 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.733+0000 7fc8443c1700 1 -- start start 2024-03-19T01:02:55.539 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.734+0000 7fc8443c1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 0x7fc83c19bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:55.539 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.734+0000 7fc8443c1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c10b140 con 0x7fc83c107070 2024-03-19T01:02:55.539 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.734+0000 7fc84215d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 0x7fc83c19bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:02:55.539 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.734+0000 7fc84215d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 0x7fc83c19bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55800/0 (socket says 172.21.15.116:55800) 2024-03-19T01:02:55.539 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.734+0000 7fc84215d700 1 -- 172.21.15.116:0/946398722 learned_addr learned my addr 172.21.15.116:0/946398722 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:02:55.539 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.734+0000 7fc84215d700 1 -- 172.21.15.116:0/946398722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83c19c180 con 0x7fc83c107070 2024-03-19T01:02:55.539 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.734+0000 7fc84215d700 1 --2- 172.21.15.116:0/946398722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 0x7fc83c19bc40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc838003aa0 tx=0x7fc838003e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:02:55.539 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.734+0000 7fc8337fe700 1 -- 172.21.15.116:0/946398722 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc83800be50 con 0x7fc83c107070 2024-03-19T01:02:55.539 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.734+0000 7fc8337fe700 1 -- 172.21.15.116:0/946398722 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc838004340 con 0x7fc83c107070 2024-03-19T01:02:55.539 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.734+0000 7fc8443c1700 1 -- 172.21.15.116:0/946398722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83c19c410 con 0x7fc83c107070 2024-03-19T01:02:55.539 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.734+0000 7fc8337fe700 1 -- 172.21.15.116:0/946398722 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc83801a780 con 0x7fc83c107070 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.734+0000 7fc8443c1700 1 -- 172.21.15.116:0/946398722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc83c19f0b0 con 0x7fc83c107070 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.735+0000 7fc8337fe700 1 -- 172.21.15.116:0/946398722 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fc838022020 con 0x7fc83c107070 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.735+0000 7fc8443c1700 1 -- 172.21.15.116:0/946398722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc83c063940 con 0x7fc83c107070 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.735+0000 7fc8337fe700 1 --2- 172.21.15.116:0/946398722 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fc828038f20 0x7fc82803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.736+0000 7fc8337fe700 1 -- 172.21.15.116:0/946398722 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc83804cc20 con 0x7fc83c107070 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.736+0000 7fc84195c700 1 -- 172.21.15.116:0/946398722 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fc828038f20 msgr2=0x7fc82803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3676795487 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.736+0000 7fc84195c700 1 --2- 172.21.15.116:0/946398722 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fc828038f20 0x7fc82803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.740+0000 7fc8337fe700 1 -- 172.21.15.116:0/946398722 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc8380172b0 con 0x7fc83c107070 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.937+0000 7fc84195c700 1 -- 172.21.15.116:0/946398722 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fc828038f20 msgr2=0x7fc82803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3676795487 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.937+0000 7fc84195c700 1 --2- 172.21.15.116:0/946398722 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fc828038f20 0x7fc82803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.994+0000 7fc8443c1700 1 -- 172.21.15.116:0/946398722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fc83c0008d0 con 0x7fc83c107070 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.995+0000 7fc8337fe700 1 -- 172.21.15.116:0/946398722 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fc8380044f0 con 0x7fc83c107070 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.997+0000 7fc8443c1700 1 -- 172.21.15.116:0/946398722 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fc828038f20 msgr2=0x7fc82803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:02:55.540 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.997+0000 7fc8443c1700 1 --2- 172.21.15.116:0/946398722 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fc828038f20 0x7fc82803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:55.541 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.997+0000 7fc8443c1700 1 -- 172.21.15.116:0/946398722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 msgr2=0x7fc83c19bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:02:55.541 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.997+0000 7fc8443c1700 1 --2- 172.21.15.116:0/946398722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 0x7fc83c19bc40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc838003aa0 tx=0x7fc838003e60 comp rx=0 tx=0).stop 2024-03-19T01:02:55.541 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.997+0000 7fc8443c1700 1 -- 172.21.15.116:0/946398722 shutdown_connections 2024-03-19T01:02:55.541 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.997+0000 7fc8443c1700 1 --2- 172.21.15.116:0/946398722 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fc828038f20 0x7fc82803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:55.541 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.997+0000 7fc8443c1700 1 --2- 172.21.15.116:0/946398722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc83c107070 0x7fc83c19bc40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:02:55.541 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.997+0000 7fc8443c1700 1 -- 172.21.15.116:0/946398722 >> 172.21.15.116:0/946398722 conn(0x7fc83c1018d0 msgr2=0x7fc83c1027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:02:55.541 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.997+0000 7fc8443c1700 1 -- 172.21.15.116:0/946398722 shutdown_connections 2024-03-19T01:02:55.541 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:54.997+0000 7fc8443c1700 1 -- 172.21.15.116:0/946398722 wait complete. 2024-03-19T01:02:55.541 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for the mgr to restart... 2024-03-19T01:02:55.541 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr epoch 5... 2024-03-19T01:02:55.646 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:55 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:55.420+0000 7f4d104c3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T01:02:55.646 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:55 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:55.504+0000 7f4d104c3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T01:02:55.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:02:55 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/946398722' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-19T01:02:55.905 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:55 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:55.774+0000 7f4d104c3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T01:02:55.905 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:55 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:55.886+0000 7f4d104c3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T01:02:56.250 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:55 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:55.977+0000 7f4d104c3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T01:02:56.251 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:56 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:56.073+0000 7f4d104c3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T01:02:56.537 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:56 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:56.249+0000 7f4d104c3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T01:02:57.653 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:57 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:57.197+0000 7f4d104c3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T01:02:57.654 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:57 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:57.323+0000 7f4d104c3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T01:02:57.654 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:57 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:57.653+0000 7f4d104c3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T01:02:57.905 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:57 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:57.742+0000 7f4d104c3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T01:02:58.404 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:57 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:57.937+0000 7f4d104c3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:02:58.404 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:58 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:58.121+0000 7f4d104c3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T01:02:58.655 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:58 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:58.403+0000 7f4d104c3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:02:58.940 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:58 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:58.660+0000 7f4d104c3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T01:02:58.940 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:58 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:58.894+0000 7f4d104c3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T01:03:00.155 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:02:59 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:02:59.691+0000 7f4d104c3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T01:03:00.572 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:00.257+0000 7f4d104c3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T01:03:00.572 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:00.355+0000 7f4d104c3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T01:03:00.572 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:00.571+0000 7f4d104c3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T01:03:00.854 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: Active manager daemon x restarted 2024-03-19T01:03:00.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: Activating manager daemon x 2024-03-19T01:03:00.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: osdmap e2: 0 total, 0 up, 0 in 2024-03-19T01:03:00.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: mgrmap e6: x(active, starting, since 0.00310718s) 2024-03-19T01:03:00.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:03:00.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T01:03:00.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T01:03:00.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T01:03:00.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T01:03:00.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: Manager daemon x is now available 2024-03-19T01:03:00.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:00.855 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:00.856 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:03:00.856 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:03:00.856 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:00 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:03:01.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:01 smithi116 ceph-mon[62701]: Found migration_current of "None". Setting to last migration. 2024-03-19T01:03:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:01 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T01:03:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:01 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T01:03:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:01 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:01 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:01 smithi116 ceph-mon[62701]: mgrmap e7: x(active, since 1.00538s) 2024-03-19T01:03:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:01 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:03:02.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T01:03:02.173 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-19T01:03:02.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-19T01:03:02.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T01:03:02.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.930+0000 7fa44cc81700 1 Processor -- start 2024-03-19T01:03:02.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.930+0000 7fa44cc81700 1 -- start start 2024-03-19T01:03:02.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.930+0000 7fa44cc81700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 0x7fa448106fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:02.174 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.930+0000 7fa44cc81700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa448107570 con 0x7fa448106b80 2024-03-19T01:03:02.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.931+0000 7fa44659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 0x7fa448106fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:02.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.931+0000 7fa44659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 0x7fa448106fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55810/0 (socket says 172.21.15.116:55810) 2024-03-19T01:03:02.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.931+0000 7fa44659c700 1 -- 172.21.15.116:0/676728907 learned_addr learned my addr 172.21.15.116:0/676728907 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:02.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.931+0000 7fa44659c700 1 -- 172.21.15.116:0/676728907 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa448107d80 con 0x7fa448106b80 2024-03-19T01:03:02.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.931+0000 7fa44659c700 1 --2- 172.21.15.116:0/676728907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 0x7fa448106fa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa4380098a0 tx=0x7fa438009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9be3494a5e68fda server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:02.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.931+0000 7fa44559a700 1 -- 172.21.15.116:0/676728907 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa438004160 con 0x7fa448106b80 2024-03-19T01:03:02.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.931+0000 7fa44559a700 1 -- 172.21.15.116:0/676728907 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa438004300 con 0x7fa448106b80 2024-03-19T01:03:02.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.931+0000 7fa44559a700 1 -- 172.21.15.116:0/676728907 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa438011b50 con 0x7fa448106b80 2024-03-19T01:03:02.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.932+0000 7fa44cc81700 1 -- 172.21.15.116:0/676728907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 msgr2=0x7fa448106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:02.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.932+0000 7fa44cc81700 1 --2- 172.21.15.116:0/676728907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 0x7fa448106fa0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa4380098a0 tx=0x7fa438009bb0 comp rx=0 tx=0).stop 2024-03-19T01:03:02.178 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.932+0000 7fa44cc81700 1 -- 172.21.15.116:0/676728907 shutdown_connections 2024-03-19T01:03:02.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.932+0000 7fa44cc81700 1 --2- 172.21.15.116:0/676728907 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 0x7fa448106fa0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:02.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.932+0000 7fa44cc81700 1 -- 172.21.15.116:0/676728907 >> 172.21.15.116:0/676728907 conn(0x7fa448102140 msgr2=0x7fa448104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:02.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.932+0000 7fa44cc81700 1 -- 172.21.15.116:0/676728907 shutdown_connections 2024-03-19T01:03:02.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.932+0000 7fa44cc81700 1 -- 172.21.15.116:0/676728907 wait complete. 2024-03-19T01:03:02.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.933+0000 7fa44cc81700 1 Processor -- start 2024-03-19T01:03:02.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.933+0000 7fa44cc81700 1 -- start start 2024-03-19T01:03:02.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.933+0000 7fa44cc81700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 0x7fa44819bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:02.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.933+0000 7fa44cc81700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa44810b130 con 0x7fa448106b80 2024-03-19T01:03:02.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.933+0000 7fa44659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 0x7fa44819bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:02.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.933+0000 7fa44659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 0x7fa44819bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55818/0 (socket says 172.21.15.116:55818) 2024-03-19T01:03:02.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.933+0000 7fa44659c700 1 -- 172.21.15.116:0/2794342309 learned_addr learned my addr 172.21.15.116:0/2794342309 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:02.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.933+0000 7fa44659c700 1 -- 172.21.15.116:0/2794342309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa44819c200 con 0x7fa448106b80 2024-03-19T01:03:02.179 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.933+0000 7fa44659c700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 0x7fa44819bcc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa438011850 tx=0x7fa438011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.934+0000 7fa4377fe700 1 -- 172.21.15.116:0/2794342309 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa438003af0 con 0x7fa448106b80 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.934+0000 7fa4377fe700 1 -- 172.21.15.116:0/2794342309 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa438020070 con 0x7fa448106b80 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.934+0000 7fa4377fe700 1 -- 172.21.15.116:0/2794342309 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4380195a0 con 0x7fa448106b80 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.934+0000 7fa44cc81700 1 -- 172.21.15.116:0/2794342309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa44819c490 con 0x7fa448106b80 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.934+0000 7fa44cc81700 1 -- 172.21.15.116:0/2794342309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa44819f130 con 0x7fa448106b80 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.935+0000 7fa4377fe700 1 -- 172.21.15.116:0/2794342309 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7fa4380198a0 con 0x7fa448106b80 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.935+0000 7fa4377fe700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 0x7fa43003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.935+0000 7fa4377fe700 1 -- 172.21.15.116:0/2794342309 --> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa43003baf0 con 0x7fa430038f20 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.935+0000 7fa445d9b700 1 -- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 msgr2=0x7fa43003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3676795487 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.935+0000 7fa4377fe700 1 -- 172.21.15.116:0/2794342309 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa438011e00 con 0x7fa448106b80 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:55.935+0000 7fa445d9b700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 0x7fa43003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:56.136+0000 7fa445d9b700 1 -- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 msgr2=0x7fa43003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3676795487 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:56.136+0000 7fa445d9b700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 0x7fa43003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:03:02.180 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:56.537+0000 7fa445d9b700 1 -- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 msgr2=0x7fa43003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3676795487 2024-03-19T01:03:02.181 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:56.537+0000 7fa445d9b700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 0x7fa43003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:03:02.181 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:57.338+0000 7fa445d9b700 1 -- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 msgr2=0x7fa43003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3676795487 2024-03-19T01:03:02.181 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:57.338+0000 7fa445d9b700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 0x7fa43003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:03:02.181 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:58.939+0000 7fa445d9b700 1 -- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 msgr2=0x7fa43003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/3676795487 2024-03-19T01:03:02.181 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:02:58.939+0000 7fa445d9b700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 0x7fa43003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-19T01:03:02.181 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:00.586+0000 7fa4377fe700 1 -- 172.21.15.116:0/2794342309 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7fa43802bd00 con 0x7fa448106b80 2024-03-19T01:03:02.181 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:00.586+0000 7fa4377fe700 1 -- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 msgr2=0x7fa43003b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:03:02.181 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:00.586+0000 7fa4377fe700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 0x7fa43003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:02.181 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.587+0000 7fa4377fe700 1 -- 172.21.15.116:0/2794342309 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7fa43804f070 con 0x7fa448106b80 2024-03-19T01:03:02.181 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.587+0000 7fa4377fe700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fa43003c810 0x7fa43003ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:02.181 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.587+0000 7fa4377fe700 1 -- 172.21.15.116:0/2794342309 --> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa43800ae50 con 0x7fa43003c810 2024-03-19T01:03:02.181 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.588+0000 7fa445d9b700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fa43003c810 0x7fa43003ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:02.181 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.588+0000 7fa445d9b700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fa43003c810 0x7fa43003ec00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa43c003a10 tx=0x7fa43c009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:02.182 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.589+0000 7fa4377fe700 1 -- 172.21.15.116:0/2794342309 <== mgr.14118 v2:172.21.15.116:6800/1107242393 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fa43800ae50 con 0x7fa43003c810 2024-03-19T01:03:02.182 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.595+0000 7fa44cc81700 1 -- 172.21.15.116:0/2794342309 --> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fa448063940 con 0x7fa43003c810 2024-03-19T01:03:02.182 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.595+0000 7fa4377fe700 1 -- 172.21.15.116:0/2794342309 <== mgr.14118 v2:172.21.15.116:6800/1107242393 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fa448063940 con 0x7fa43003c810 2024-03-19T01:03:02.182 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.596+0000 7fa44cc81700 1 -- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fa43003c810 msgr2=0x7fa43003ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:02.182 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.596+0000 7fa44cc81700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fa43003c810 0x7fa43003ec00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa43c003a10 tx=0x7fa43c009290 comp rx=0 tx=0).stop 2024-03-19T01:03:02.182 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.596+0000 7fa44cc81700 1 -- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 msgr2=0x7fa44819bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:02.182 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.596+0000 7fa44cc81700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 0x7fa44819bcc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa438011850 tx=0x7fa438011c10 comp rx=0 tx=0).stop 2024-03-19T01:03:02.182 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.596+0000 7fa44cc81700 1 -- 172.21.15.116:0/2794342309 shutdown_connections 2024-03-19T01:03:02.182 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.596+0000 7fa44cc81700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fa43003c810 0x7fa43003ec00 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:02.182 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.596+0000 7fa44cc81700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa448106b80 0x7fa44819bcc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:02.182 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.596+0000 7fa44cc81700 1 --2- 172.21.15.116:0/2794342309 >> [v2:172.21.15.116:6800/3676795487,v1:172.21.15.116:6801/3676795487] conn(0x7fa430038f20 0x7fa43003b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:02.182 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.596+0000 7fa44cc81700 1 -- 172.21.15.116:0/2794342309 >> 172.21.15.116:0/2794342309 conn(0x7fa448102140 msgr2=0x7fa4481030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:02.182 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.596+0000 7fa44cc81700 1 -- 172.21.15.116:0/2794342309 shutdown_connections 2024-03-19T01:03:02.183 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:01.596+0000 7fa44cc81700 1 -- 172.21.15.116:0/2794342309 wait complete. 2024-03-19T01:03:02.183 INFO:teuthology.orchestra.run.smithi116.stdout:mgr epoch 5 is available 2024-03-19T01:03:02.183 INFO:teuthology.orchestra.run.smithi116.stdout:Setting orchestrator backend to cephadm... 2024-03-19T01:03:02.782 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:02 smithi116 ceph-mon[62701]: [19/Mar/2024:01:03:01] ENGINE Bus STARTING 2024-03-19T01:03:02.782 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:02 smithi116 ceph-mon[62701]: [19/Mar/2024:01:03:01] ENGINE Serving on http://172.21.15.116:8765 2024-03-19T01:03:02.783 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:02 smithi116 ceph-mon[62701]: [19/Mar/2024:01:03:01] ENGINE Serving on https://172.21.15.116:7150 2024-03-19T01:03:02.783 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:02 smithi116 ceph-mon[62701]: [19/Mar/2024:01:03:01] ENGINE Bus STARTED 2024-03-19T01:03:02.783 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:02 smithi116 ceph-mon[62701]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-19T01:03:02.783 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:02 smithi116 ceph-mon[62701]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-19T01:03:03.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.539+0000 7fd4892c0700 1 Processor -- start 2024-03-19T01:03:03.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.539+0000 7fd4892c0700 1 -- start start 2024-03-19T01:03:03.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.539+0000 7fd4892c0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 0x7fd484109660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:03.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.539+0000 7fd4892c0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd484109c30 con 0x7fd484109240 2024-03-19T01:03:03.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.539+0000 7fd482ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 0x7fd484109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:03.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.539+0000 7fd482ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 0x7fd484109660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35422/0 (socket says 172.21.15.116:35422) 2024-03-19T01:03:03.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.539+0000 7fd482ffd700 1 -- 172.21.15.116:0/245802822 learned_addr learned my addr 172.21.15.116:0/245802822 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:03.332 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.539+0000 7fd482ffd700 1 -- 172.21.15.116:0/245802822 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd484109db0 con 0x7fd484109240 2024-03-19T01:03:03.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.540+0000 7fd482ffd700 1 --2- 172.21.15.116:0/245802822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 0x7fd484109660 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd4740098a0 tx=0x7fd474009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d55f06d54f7b0d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:03.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.540+0000 7fd481ffb700 1 -- 172.21.15.116:0/245802822 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd474004160 con 0x7fd484109240 2024-03-19T01:03:03.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.540+0000 7fd481ffb700 1 -- 172.21.15.116:0/245802822 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd474004300 con 0x7fd484109240 2024-03-19T01:03:03.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.540+0000 7fd481ffb700 1 -- 172.21.15.116:0/245802822 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd474011b50 con 0x7fd484109240 2024-03-19T01:03:03.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.540+0000 7fd4892c0700 1 -- 172.21.15.116:0/245802822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 msgr2=0x7fd484109660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:03.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.540+0000 7fd4892c0700 1 --2- 172.21.15.116:0/245802822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 0x7fd484109660 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd4740098a0 tx=0x7fd474009bb0 comp rx=0 tx=0).stop 2024-03-19T01:03:03.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.540+0000 7fd4892c0700 1 -- 172.21.15.116:0/245802822 shutdown_connections 2024-03-19T01:03:03.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.540+0000 7fd4892c0700 1 --2- 172.21.15.116:0/245802822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 0x7fd484109660 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:03.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.540+0000 7fd4892c0700 1 -- 172.21.15.116:0/245802822 >> 172.21.15.116:0/245802822 conn(0x7fd4841018e0 msgr2=0x7fd484103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:03.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.540+0000 7fd4892c0700 1 -- 172.21.15.116:0/245802822 shutdown_connections 2024-03-19T01:03:03.333 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.540+0000 7fd4892c0700 1 -- 172.21.15.116:0/245802822 wait complete. 2024-03-19T01:03:03.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.541+0000 7fd4892c0700 1 Processor -- start 2024-03-19T01:03:03.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.541+0000 7fd4892c0700 1 -- start start 2024-03-19T01:03:03.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.541+0000 7fd4892c0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 0x7fd484108860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:03.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.541+0000 7fd4892c0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd484106a40 con 0x7fd484109240 2024-03-19T01:03:03.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.541+0000 7fd482ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 0x7fd484108860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:03.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.541+0000 7fd482ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 0x7fd484108860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35424/0 (socket says 172.21.15.116:35424) 2024-03-19T01:03:03.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.541+0000 7fd482ffd700 1 -- 172.21.15.116:0/3727455648 learned_addr learned my addr 172.21.15.116:0/3727455648 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:03.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.542+0000 7fd482ffd700 1 -- 172.21.15.116:0/3727455648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd484108da0 con 0x7fd484109240 2024-03-19T01:03:03.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.542+0000 7fd482ffd700 1 --2- 172.21.15.116:0/3727455648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 0x7fd484108860 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd474009fb0 tx=0x7fd474011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:03.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.542+0000 7fd46bfff700 1 -- 172.21.15.116:0/3727455648 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd4740038b0 con 0x7fd484109240 2024-03-19T01:03:03.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.542+0000 7fd46bfff700 1 -- 172.21.15.116:0/3727455648 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd474011e50 con 0x7fd484109240 2024-03-19T01:03:03.334 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.542+0000 7fd4892c0700 1 -- 172.21.15.116:0/3727455648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd484106f90 con 0x7fd484109240 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.542+0000 7fd46bfff700 1 -- 172.21.15.116:0/3727455648 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd474019550 con 0x7fd484109240 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.542+0000 7fd4892c0700 1 -- 172.21.15.116:0/3727455648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4841074d0 con 0x7fd484109240 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.543+0000 7fd46bfff700 1 -- 172.21.15.116:0/3727455648 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7fd474019850 con 0x7fd484109240 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.543+0000 7fd4892c0700 1 -- 172.21.15.116:0/3727455648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd48410c210 con 0x7fd484109240 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.543+0000 7fd46bfff700 1 --2- 172.21.15.116:0/3727455648 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fd46c038dc0 0x7fd46c03b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.543+0000 7fd46bfff700 1 -- 172.21.15.116:0/3727455648 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd47404ceb0 con 0x7fd484109240 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.543+0000 7fd4827fc700 1 --2- 172.21.15.116:0/3727455648 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fd46c038dc0 0x7fd46c03b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.543+0000 7fd4827fc700 1 --2- 172.21.15.116:0/3727455648 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fd46c038dc0 0x7fd46c03b280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd478006fd0 tx=0x7fd478006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.548+0000 7fd46bfff700 1 -- 172.21.15.116:0/3727455648 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd47402d960 con 0x7fd484109240 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.638+0000 7fd46bfff700 1 -- 172.21.15.116:0/3727455648 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fd47402d440 con 0x7fd484109240 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.761+0000 7fd4892c0700 1 -- 172.21.15.116:0/3727455648 --> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fd484107e40 con 0x7fd46c038dc0 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.767+0000 7fd46bfff700 1 -- 172.21.15.116:0/3727455648 <== mgr.14118 v2:172.21.15.116:6800/1107242393 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd47800a040 con 0x7fd46c038dc0 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.769+0000 7fd4892c0700 1 -- 172.21.15.116:0/3727455648 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fd46c038dc0 msgr2=0x7fd46c03b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.769+0000 7fd4892c0700 1 --2- 172.21.15.116:0/3727455648 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fd46c038dc0 0x7fd46c03b280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd478006fd0 tx=0x7fd478006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:03.335 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.769+0000 7fd4892c0700 1 -- 172.21.15.116:0/3727455648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 msgr2=0x7fd484108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:03.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.769+0000 7fd4892c0700 1 --2- 172.21.15.116:0/3727455648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 0x7fd484108860 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd474009fb0 tx=0x7fd474011bf0 comp rx=0 tx=0).stop 2024-03-19T01:03:03.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.769+0000 7fd4892c0700 1 -- 172.21.15.116:0/3727455648 shutdown_connections 2024-03-19T01:03:03.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.769+0000 7fd4892c0700 1 --2- 172.21.15.116:0/3727455648 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fd46c038dc0 0x7fd46c03b280 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:03.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.769+0000 7fd4892c0700 1 --2- 172.21.15.116:0/3727455648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd484109240 0x7fd484108860 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:03.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.769+0000 7fd4892c0700 1 -- 172.21.15.116:0/3727455648 >> 172.21.15.116:0/3727455648 conn(0x7fd4841018e0 msgr2=0x7fd4841024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:03.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.769+0000 7fd4892c0700 1 -- 172.21.15.116:0/3727455648 shutdown_connections 2024-03-19T01:03:03.336 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:02.769+0000 7fd4892c0700 1 -- 172.21.15.116:0/3727455648 wait complete. 2024-03-19T01:03:03.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:03 smithi116 ceph-mon[62701]: mgrmap e8: x(active, since 2s) 2024-03-19T01:03:03.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:03 smithi116 ceph-mon[62701]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:03:03.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:03 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:03.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:03 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:03:04.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-19T01:03:04.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.704+0000 7fee49b76700 1 Processor -- start 2024-03-19T01:03:04.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.704+0000 7fee49b76700 1 -- start start 2024-03-19T01:03:04.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.704+0000 7fee49b76700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 0x7fee441080a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:04.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.704+0000 7fee49b76700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fee44108670 con 0x7fee44107c80 2024-03-19T01:03:04.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.704+0000 7fee437fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 0x7fee441080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:04.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.704+0000 7fee437fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 0x7fee441080a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35438/0 (socket says 172.21.15.116:35438) 2024-03-19T01:03:04.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.704+0000 7fee437fe700 1 -- 172.21.15.116:0/3849389436 learned_addr learned my addr 172.21.15.116:0/3849389436 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:04.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.704+0000 7fee437fe700 1 -- 172.21.15.116:0/3849389436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee44108e80 con 0x7fee44107c80 2024-03-19T01:03:04.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.705+0000 7fee437fe700 1 --2- 172.21.15.116:0/3849389436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 0x7fee441080a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fee340098a0 tx=0x7fee34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d916834545d596f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:04.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.705+0000 7fee427fc700 1 -- 172.21.15.116:0/3849389436 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee34004160 con 0x7fee44107c80 2024-03-19T01:03:04.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.705+0000 7fee427fc700 1 -- 172.21.15.116:0/3849389436 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fee34004300 con 0x7fee44107c80 2024-03-19T01:03:04.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.705+0000 7fee427fc700 1 -- 172.21.15.116:0/3849389436 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee34011b50 con 0x7fee44107c80 2024-03-19T01:03:04.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.705+0000 7fee49b76700 1 -- 172.21.15.116:0/3849389436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 msgr2=0x7fee441080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:04.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.705+0000 7fee49b76700 1 --2- 172.21.15.116:0/3849389436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 0x7fee441080a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fee340098a0 tx=0x7fee34009bb0 comp rx=0 tx=0).stop 2024-03-19T01:03:04.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.705+0000 7fee49b76700 1 -- 172.21.15.116:0/3849389436 shutdown_connections 2024-03-19T01:03:04.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.705+0000 7fee49b76700 1 --2- 172.21.15.116:0/3849389436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 0x7fee441080a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:04.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.705+0000 7fee49b76700 1 -- 172.21.15.116:0/3849389436 >> 172.21.15.116:0/3849389436 conn(0x7fee44103200 msgr2=0x7fee44105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:04.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.705+0000 7fee49b76700 1 -- 172.21.15.116:0/3849389436 shutdown_connections 2024-03-19T01:03:04.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.705+0000 7fee49b76700 1 -- 172.21.15.116:0/3849389436 wait complete. 2024-03-19T01:03:04.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.706+0000 7fee49b76700 1 Processor -- start 2024-03-19T01:03:04.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.706+0000 7fee49b76700 1 -- start start 2024-03-19T01:03:04.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.706+0000 7fee49b76700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 0x7fee4419bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:04.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.706+0000 7fee49b76700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fee4410c230 con 0x7fee44107c80 2024-03-19T01:03:04.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.706+0000 7fee437fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 0x7fee4419bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:04.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.707+0000 7fee437fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 0x7fee4419bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35442/0 (socket says 172.21.15.116:35442) 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.707+0000 7fee437fe700 1 -- 172.21.15.116:0/2191541315 learned_addr learned my addr 172.21.15.116:0/2191541315 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.707+0000 7fee437fe700 1 -- 172.21.15.116:0/2191541315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4419c180 con 0x7fee44107c80 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.707+0000 7fee437fe700 1 --2- 172.21.15.116:0/2191541315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 0x7fee4419bc40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee34011850 tx=0x7fee34011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.707+0000 7fee40ff9700 1 -- 172.21.15.116:0/2191541315 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee34003af0 con 0x7fee44107c80 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.707+0000 7fee40ff9700 1 -- 172.21.15.116:0/2191541315 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fee34020070 con 0x7fee44107c80 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.707+0000 7fee40ff9700 1 -- 172.21.15.116:0/2191541315 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee340195a0 con 0x7fee44107c80 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.707+0000 7fee49b76700 1 -- 172.21.15.116:0/2191541315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee4419c410 con 0x7fee44107c80 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.707+0000 7fee49b76700 1 -- 172.21.15.116:0/2191541315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee4419f100 con 0x7fee44107c80 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.708+0000 7fee40ff9700 1 -- 172.21.15.116:0/2191541315 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fee340198a0 con 0x7fee44107c80 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.708+0000 7fee49b76700 1 -- 172.21.15.116:0/2191541315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee44063940 con 0x7fee44107c80 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.708+0000 7fee40ff9700 1 --2- 172.21.15.116:0/2191541315 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fee2c038f20 0x7fee2c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.708+0000 7fee40ff9700 1 -- 172.21.15.116:0/2191541315 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fee34011e00 con 0x7fee44107c80 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.708+0000 7fee42ffd700 1 --2- 172.21.15.116:0/2191541315 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fee2c038f20 0x7fee2c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:04.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.708+0000 7fee42ffd700 1 --2- 172.21.15.116:0/2191541315 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fee2c038f20 0x7fee2c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fee38006fd0 tx=0x7fee38006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.713+0000 7fee40ff9700 1 -- 172.21.15.116:0/2191541315 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fee34019b90 con 0x7fee44107c80 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.910+0000 7fee49b76700 1 -- 172.21.15.116:0/2191541315 --> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fee44104580 con 0x7fee2c038f20 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.911+0000 7fee40ff9700 1 -- 172.21.15.116:0/2191541315 <== mgr.14118 v2:172.21.15.116:6800/1107242393 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fee44104580 con 0x7fee2c038f20 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.913+0000 7fee49b76700 1 -- 172.21.15.116:0/2191541315 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fee2c038f20 msgr2=0x7fee2c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.913+0000 7fee49b76700 1 --2- 172.21.15.116:0/2191541315 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fee2c038f20 0x7fee2c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fee38006fd0 tx=0x7fee38006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.913+0000 7fee49b76700 1 -- 172.21.15.116:0/2191541315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 msgr2=0x7fee4419bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.913+0000 7fee49b76700 1 --2- 172.21.15.116:0/2191541315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 0x7fee4419bc40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee34011850 tx=0x7fee34011c10 comp rx=0 tx=0).stop 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.913+0000 7fee49b76700 1 -- 172.21.15.116:0/2191541315 shutdown_connections 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.913+0000 7fee49b76700 1 --2- 172.21.15.116:0/2191541315 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fee2c038f20 0x7fee2c03b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.913+0000 7fee49b76700 1 --2- 172.21.15.116:0/2191541315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee44107c80 0x7fee4419bc40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.913+0000 7fee49b76700 1 -- 172.21.15.116:0/2191541315 >> 172.21.15.116:0/2191541315 conn(0x7fee44103200 msgr2=0x7fee44103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.913+0000 7fee49b76700 1 -- 172.21.15.116:0/2191541315 shutdown_connections 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:03.913+0000 7fee49b76700 1 -- 172.21.15.116:0/2191541315 wait complete. 2024-03-19T01:03:04.416 INFO:teuthology.orchestra.run.smithi116.stdout:Generating ssh key... 2024-03-19T01:03:05.035 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:04 smithi116 ceph-mon[62701]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:03:05.968 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: Generating public/private rsa key pair. 2024-03-19T01:03:05.969 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: Your identification has been saved in /tmp/tmp6o06ulgn/key. 2024-03-19T01:03:05.969 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: Your public key has been saved in /tmp/tmp6o06ulgn/key.pub. 2024-03-19T01:03:05.969 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: The key fingerprint is: 2024-03-19T01:03:05.969 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: SHA256:fFBWlX7+Rx0/TsTkYF5o+CXwBFnl4SJArTEjnOAa0y4 ceph-35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:03:05.969 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: The key's randomart image is: 2024-03-19T01:03:05.969 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: +---[RSA 3072]----+ 2024-03-19T01:03:05.969 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: | .o oo++B+++ | 2024-03-19T01:03:05.969 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: | o + *.+o*++.| 2024-03-19T01:03:05.970 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: | o o o =.=+Oo | 2024-03-19T01:03:05.970 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: | = . o .oo=.| 2024-03-19T01:03:05.970 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: | E . S . .o+| 2024-03-19T01:03:05.970 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: | . . o=| 2024-03-19T01:03:05.970 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: | o.o| 2024-03-19T01:03:05.970 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: | .o| 2024-03-19T01:03:05.970 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: | .| 2024-03-19T01:03:05.970 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: +----[SHA256]-----+ 2024-03-19T01:03:06.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.829+0000 7fd37afa2700 1 Processor -- start 2024-03-19T01:03:06.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.829+0000 7fd37afa2700 1 -- start start 2024-03-19T01:03:06.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.829+0000 7fd37afa2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 0x7fd374106360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:06.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.829+0000 7fd37afa2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd374106930 con 0x7fd374105f40 2024-03-19T01:03:06.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.829+0000 7fd378d3e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 0x7fd374106360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:06.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.829+0000 7fd378d3e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 0x7fd374106360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35456/0 (socket says 172.21.15.116:35456) 2024-03-19T01:03:06.218 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.829+0000 7fd378d3e700 1 -- 172.21.15.116:0/3490109101 learned_addr learned my addr 172.21.15.116:0/3490109101 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.830+0000 7fd378d3e700 1 -- 172.21.15.116:0/3490109101 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd374108ed0 con 0x7fd374105f40 2024-03-19T01:03:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.830+0000 7fd378d3e700 1 --2- 172.21.15.116:0/3490109101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 0x7fd374106360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd3680098a0 tx=0x7fd368009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15dc20ddf673723f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.830+0000 7fd3737fe700 1 -- 172.21.15.116:0/3490109101 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd368004160 con 0x7fd374105f40 2024-03-19T01:03:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.830+0000 7fd3737fe700 1 -- 172.21.15.116:0/3490109101 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd368004300 con 0x7fd374105f40 2024-03-19T01:03:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.830+0000 7fd3737fe700 1 -- 172.21.15.116:0/3490109101 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd368011b50 con 0x7fd374105f40 2024-03-19T01:03:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.830+0000 7fd37afa2700 1 -- 172.21.15.116:0/3490109101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 msgr2=0x7fd374106360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.830+0000 7fd37afa2700 1 --2- 172.21.15.116:0/3490109101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 0x7fd374106360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd3680098a0 tx=0x7fd368009bb0 comp rx=0 tx=0).stop 2024-03-19T01:03:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.830+0000 7fd37afa2700 1 -- 172.21.15.116:0/3490109101 shutdown_connections 2024-03-19T01:03:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.830+0000 7fd37afa2700 1 --2- 172.21.15.116:0/3490109101 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 0x7fd374106360 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.830+0000 7fd37afa2700 1 -- 172.21.15.116:0/3490109101 >> 172.21.15.116:0/3490109101 conn(0x7fd374101890 msgr2=0x7fd374103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.830+0000 7fd37afa2700 1 -- 172.21.15.116:0/3490109101 shutdown_connections 2024-03-19T01:03:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.830+0000 7fd37afa2700 1 -- 172.21.15.116:0/3490109101 wait complete. 2024-03-19T01:03:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.831+0000 7fd37afa2700 1 Processor -- start 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.831+0000 7fd37afa2700 1 -- start start 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.831+0000 7fd37afa2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 0x7fd37419bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.831+0000 7fd37afa2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd37410c280 con 0x7fd374105f40 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.831+0000 7fd378d3e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 0x7fd37419bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.831+0000 7fd378d3e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 0x7fd37419bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35464/0 (socket says 172.21.15.116:35464) 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.831+0000 7fd378d3e700 1 -- 172.21.15.116:0/3151830641 learned_addr learned my addr 172.21.15.116:0/3151830641 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.831+0000 7fd378d3e700 1 -- 172.21.15.116:0/3151830641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd37419c1b0 con 0x7fd374105f40 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.832+0000 7fd378d3e700 1 --2- 172.21.15.116:0/3151830641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 0x7fd37419bc70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd368009fb0 tx=0x7fd368011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.832+0000 7fd371ffb700 1 -- 172.21.15.116:0/3151830641 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3680038b0 con 0x7fd374105f40 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.832+0000 7fd37afa2700 1 -- 172.21.15.116:0/3151830641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd37419c440 con 0x7fd374105f40 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.832+0000 7fd371ffb700 1 -- 172.21.15.116:0/3151830641 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd368011e50 con 0x7fd374105f40 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.832+0000 7fd37afa2700 1 -- 172.21.15.116:0/3151830641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd37419f130 con 0x7fd374105f40 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.832+0000 7fd371ffb700 1 -- 172.21.15.116:0/3151830641 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd368004300 con 0x7fd374105f40 2024-03-19T01:03:06.220 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.833+0000 7fd371ffb700 1 -- 172.21.15.116:0/3151830641 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fd36802d440 con 0x7fd374105f40 2024-03-19T01:03:06.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.833+0000 7fd37afa2700 1 -- 172.21.15.116:0/3151830641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd374063940 con 0x7fd374105f40 2024-03-19T01:03:06.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.833+0000 7fd371ffb700 1 --2- 172.21.15.116:0/3151830641 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fd364039070 0x7fd36403b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:06.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.833+0000 7fd371ffb700 1 -- 172.21.15.116:0/3151830641 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd36804d340 con 0x7fd374105f40 2024-03-19T01:03:06.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.833+0000 7fd373fff700 1 --2- 172.21.15.116:0/3151830641 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fd364039070 0x7fd36403b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:06.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.833+0000 7fd373fff700 1 --2- 172.21.15.116:0/3151830641 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fd364039070 0x7fd36403b530 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd360006fd0 tx=0x7fd360006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:06.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:04.838+0000 7fd371ffb700 1 -- 172.21.15.116:0/3151830641 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd36802c9a0 con 0x7fd374105f40 2024-03-19T01:03:06.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:05.033+0000 7fd37afa2700 1 -- 172.21.15.116:0/3151830641 --> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fd374102ba0 con 0x7fd364039070 2024-03-19T01:03:06.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:05.667+0000 7fd371ffb700 1 -- 172.21.15.116:0/3151830641 <== mgr.14118 v2:172.21.15.116:6800/1107242393 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd374102ba0 con 0x7fd364039070 2024-03-19T01:03:06.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:05.669+0000 7fd37afa2700 1 -- 172.21.15.116:0/3151830641 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fd364039070 msgr2=0x7fd36403b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:06.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:05.669+0000 7fd37afa2700 1 --2- 172.21.15.116:0/3151830641 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fd364039070 0x7fd36403b530 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd360006fd0 tx=0x7fd360006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:06.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:05.669+0000 7fd37afa2700 1 -- 172.21.15.116:0/3151830641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 msgr2=0x7fd37419bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:06.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:05.669+0000 7fd37afa2700 1 --2- 172.21.15.116:0/3151830641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 0x7fd37419bc70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd368009fb0 tx=0x7fd368011bf0 comp rx=0 tx=0).stop 2024-03-19T01:03:06.221 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:05.669+0000 7fd37afa2700 1 -- 172.21.15.116:0/3151830641 shutdown_connections 2024-03-19T01:03:06.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:05.669+0000 7fd37afa2700 1 --2- 172.21.15.116:0/3151830641 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fd364039070 0x7fd36403b530 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:06.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:05.669+0000 7fd37afa2700 1 --2- 172.21.15.116:0/3151830641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd374105f40 0x7fd37419bc70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:06.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:05.669+0000 7fd37afa2700 1 -- 172.21.15.116:0/3151830641 >> 172.21.15.116:0/3151830641 conn(0x7fd374101890 msgr2=0x7fd374102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:06.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:05.669+0000 7fd37afa2700 1 -- 172.21.15.116:0/3151830641 shutdown_connections 2024-03-19T01:03:06.222 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:05.669+0000 7fd37afa2700 1 -- 172.21.15.116:0/3151830641 wait complete. 2024-03-19T01:03:06.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:06 smithi116 ceph-mon[62701]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:03:06.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:06 smithi116 ceph-mon[62701]: Generating ssh key... 2024-03-19T01:03:06.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:06 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:06.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:06 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:07.390 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJY3KYz2wnSSqrFonV6RJdXsHgY4a3hB6m7FZxA66KYCPdxecd/RqcnTtrbRUkzm5qD4xDn4kMGqZQbuhIowEYzwAkd6mReX8p2jciUITHe9n141cSKMjKc6QhNU0qpOAk4uq3HTH/8++w4FS9UFULFTfwOkazXwCrY7Tq/dtGp8dx0RefhUpzQTgLrhzlVB2OJkcPtqhqrQe/jUYjmbPVhDSZ3tQzb53+icbF8ixvPQFBqES00FAnTMNV/NYg9/NtzzAbzpZ3yLE1MIC3HwYtLHF/4gNIK6KUhMnwc92HVC3qrrDVBg9r9F7pvgSXAgo4P5d0lYPAdhIplbSzBb0gQpW1ZKvlMPU/bf6p/37z8J7FSOscZBaCoPZugYEj1WW+Z8M1Xf0IM6L6o/OPfHXoMFgoKaL/NLN7pILmhGMujCH2hyr2Nt+96KIo3JY5Fy6rIRpEQwM+P8MxseZMciOvgn1CfvTqRYXAYdt7bKIpjLnd8Jd6VIz5JBoMk5s6zJk= ceph-35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:03:07.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.594+0000 7fb1be342700 1 Processor -- start 2024-03-19T01:03:07.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.594+0000 7fb1be342700 1 -- start start 2024-03-19T01:03:07.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.594+0000 7fb1be342700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 0x7fb1b810abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:07.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.594+0000 7fb1be342700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b807bd50 con 0x7fb1b8108800 2024-03-19T01:03:07.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.594+0000 7fb1b7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 0x7fb1b810abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:07.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.594+0000 7fb1b7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 0x7fb1b810abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44144/0 (socket says 172.21.15.116:44144) 2024-03-19T01:03:07.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.594+0000 7fb1b7fff700 1 -- 172.21.15.116:0/1848688676 learned_addr learned my addr 172.21.15.116:0/1848688676 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:07.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.594+0000 7fb1b7fff700 1 -- 172.21.15.116:0/1848688676 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b810b130 con 0x7fb1b8108800 2024-03-19T01:03:07.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.594+0000 7fb1b7fff700 1 --2- 172.21.15.116:0/1848688676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 0x7fb1b810abf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb1a80098a0 tx=0x7fb1a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74581b9d379b539d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:07.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.594+0000 7fb1b6ffd700 1 -- 172.21.15.116:0/1848688676 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1a8004160 con 0x7fb1b8108800 2024-03-19T01:03:07.391 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.594+0000 7fb1b6ffd700 1 -- 172.21.15.116:0/1848688676 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb1a8004300 con 0x7fb1b8108800 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.594+0000 7fb1b6ffd700 1 -- 172.21.15.116:0/1848688676 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1a8011b50 con 0x7fb1b8108800 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.595+0000 7fb1be342700 1 -- 172.21.15.116:0/1848688676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 msgr2=0x7fb1b810abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.595+0000 7fb1be342700 1 --2- 172.21.15.116:0/1848688676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 0x7fb1b810abf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb1a80098a0 tx=0x7fb1a8009bb0 comp rx=0 tx=0).stop 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.595+0000 7fb1be342700 1 -- 172.21.15.116:0/1848688676 shutdown_connections 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.595+0000 7fb1be342700 1 --2- 172.21.15.116:0/1848688676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 0x7fb1b810abf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.595+0000 7fb1be342700 1 -- 172.21.15.116:0/1848688676 >> 172.21.15.116:0/1848688676 conn(0x7fb1b8102150 msgr2=0x7fb1b8104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.595+0000 7fb1be342700 1 -- 172.21.15.116:0/1848688676 shutdown_connections 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.595+0000 7fb1be342700 1 -- 172.21.15.116:0/1848688676 wait complete. 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.596+0000 7fb1be342700 1 Processor -- start 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.596+0000 7fb1be342700 1 -- start start 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.596+0000 7fb1be342700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 0x7fb1b819bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.596+0000 7fb1be342700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b810c020 con 0x7fb1b8108800 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.596+0000 7fb1b7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 0x7fb1b819bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.596+0000 7fb1b7fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 0x7fb1b819bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44154/0 (socket says 172.21.15.116:44154) 2024-03-19T01:03:07.392 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.596+0000 7fb1b7fff700 1 -- 172.21.15.116:0/2767976421 learned_addr learned my addr 172.21.15.116:0/2767976421 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.596+0000 7fb1b7fff700 1 -- 172.21.15.116:0/2767976421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b819c120 con 0x7fb1b8108800 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.596+0000 7fb1b7fff700 1 --2- 172.21.15.116:0/2767976421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 0x7fb1b819bbe0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8009fb0 tx=0x7fb1a8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.597+0000 7fb1b57fa700 1 -- 172.21.15.116:0/2767976421 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1a80038b0 con 0x7fb1b8108800 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.597+0000 7fb1b57fa700 1 -- 172.21.15.116:0/2767976421 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb1a8011e50 con 0x7fb1b8108800 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.597+0000 7fb1be342700 1 -- 172.21.15.116:0/2767976421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b819c3b0 con 0x7fb1b8108800 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.597+0000 7fb1b57fa700 1 -- 172.21.15.116:0/2767976421 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb1a8019420 con 0x7fb1b8108800 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.597+0000 7fb1be342700 1 -- 172.21.15.116:0/2767976421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1b819f0a0 con 0x7fb1b8108800 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.598+0000 7fb1b57fa700 1 -- 172.21.15.116:0/2767976421 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fb1a8019720 con 0x7fb1b8108800 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.598+0000 7fb1be342700 1 -- 172.21.15.116:0/2767976421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1b8063940 con 0x7fb1b8108800 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.598+0000 7fb1b57fa700 1 --2- 172.21.15.116:0/2767976421 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fb1a0038ed0 0x7fb1a003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.598+0000 7fb1b57fa700 1 -- 172.21.15.116:0/2767976421 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb1a804d2c0 con 0x7fb1b8108800 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.598+0000 7fb1b77fe700 1 --2- 172.21.15.116:0/2767976421 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fb1a0038ed0 0x7fb1a003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.598+0000 7fb1b77fe700 1 --2- 172.21.15.116:0/2767976421 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fb1a0038ed0 0x7fb1a003b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac006fd0 tx=0x7fb1ac006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.603+0000 7fb1b57fa700 1 -- 172.21.15.116:0/2767976421 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb1a8003ac0 con 0x7fb1b8108800 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.799+0000 7fb1be342700 1 -- 172.21.15.116:0/2767976421 --> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fb1b8103380 con 0x7fb1a0038ed0 2024-03-19T01:03:07.393 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.800+0000 7fb1b57fa700 1 -- 172.21.15.116:0/2767976421 <== mgr.14118 v2:172.21.15.116:6800/1107242393 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fb1b8103380 con 0x7fb1a0038ed0 2024-03-19T01:03:07.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.802+0000 7fb1be342700 1 -- 172.21.15.116:0/2767976421 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fb1a0038ed0 msgr2=0x7fb1a003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:07.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.802+0000 7fb1be342700 1 --2- 172.21.15.116:0/2767976421 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fb1a0038ed0 0x7fb1a003b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac006fd0 tx=0x7fb1ac006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:07.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.802+0000 7fb1be342700 1 -- 172.21.15.116:0/2767976421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 msgr2=0x7fb1b819bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:07.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.802+0000 7fb1be342700 1 --2- 172.21.15.116:0/2767976421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 0x7fb1b819bbe0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8009fb0 tx=0x7fb1a8011bf0 comp rx=0 tx=0).stop 2024-03-19T01:03:07.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.802+0000 7fb1be342700 1 -- 172.21.15.116:0/2767976421 shutdown_connections 2024-03-19T01:03:07.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.802+0000 7fb1be342700 1 --2- 172.21.15.116:0/2767976421 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fb1a0038ed0 0x7fb1a003b390 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:07.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.802+0000 7fb1be342700 1 --2- 172.21.15.116:0/2767976421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1b8108800 0x7fb1b819bbe0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:07.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.802+0000 7fb1be342700 1 -- 172.21.15.116:0/2767976421 >> 172.21.15.116:0/2767976421 conn(0x7fb1b8102150 msgr2=0x7fb1b8102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:07.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.802+0000 7fb1be342700 1 -- 172.21.15.116:0/2767976421 shutdown_connections 2024-03-19T01:03:07.394 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:06.802+0000 7fb1be342700 1 -- 172.21.15.116:0/2767976421 wait complete. 2024-03-19T01:03:07.394 INFO:teuthology.orchestra.run.smithi116.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-19T01:03:07.394 INFO:teuthology.orchestra.run.smithi116.stdout:Adding key to root@localhost authorized_keys... 2024-03-19T01:03:07.394 INFO:teuthology.orchestra.run.smithi116.stdout:Adding host smithi116... 2024-03-19T01:03:07.760 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:07 smithi116 ceph-mon[62701]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:03:09.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:08 smithi116 ceph-mon[62701]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi116", "addr": "172.21.15.116", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:03:09.145 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:08 smithi116 ceph-mon[62701]: Deploying cephadm binary to smithi116 2024-03-19T01:03:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Added host 'smithi116' with addr '172.21.15.116' 2024-03-19T01:03:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.745+0000 7fe197164700 1 Processor -- start 2024-03-19T01:03:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.745+0000 7fe197164700 1 -- start start 2024-03-19T01:03:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.745+0000 7fe197164700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 0x7fe190107490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.746+0000 7fe197164700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe190107a60 con 0x7fe190107070 2024-03-19T01:03:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.746+0000 7fe194f00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 0x7fe190107490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:12.765 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.746+0000 7fe194f00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 0x7fe190107490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44156/0 (socket says 172.21.15.116:44156) 2024-03-19T01:03:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.746+0000 7fe194f00700 1 -- 172.21.15.116:0/453819822 learned_addr learned my addr 172.21.15.116:0/453819822 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.746+0000 7fe194f00700 1 -- 172.21.15.116:0/453819822 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe190107d90 con 0x7fe190107070 2024-03-19T01:03:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.746+0000 7fe194f00700 1 --2- 172.21.15.116:0/453819822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 0x7fe190107490 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe184009ab0 tx=0x7fe184009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2ced3a5862e688 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.746+0000 7fe18f7fe700 1 -- 172.21.15.116:0/453819822 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe184004160 con 0x7fe190107070 2024-03-19T01:03:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.746+0000 7fe18f7fe700 1 -- 172.21.15.116:0/453819822 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe18400bca0 con 0x7fe190107070 2024-03-19T01:03:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.746+0000 7fe18f7fe700 1 -- 172.21.15.116:0/453819822 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe184003d60 con 0x7fe190107070 2024-03-19T01:03:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.747+0000 7fe197164700 1 -- 172.21.15.116:0/453819822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 msgr2=0x7fe190107490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.747+0000 7fe197164700 1 --2- 172.21.15.116:0/453819822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 0x7fe190107490 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe184009ab0 tx=0x7fe184009dc0 comp rx=0 tx=0).stop 2024-03-19T01:03:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.747+0000 7fe197164700 1 -- 172.21.15.116:0/453819822 shutdown_connections 2024-03-19T01:03:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.747+0000 7fe197164700 1 --2- 172.21.15.116:0/453819822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 0x7fe190107490 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.747+0000 7fe197164700 1 -- 172.21.15.116:0/453819822 >> 172.21.15.116:0/453819822 conn(0x7fe1901018d0 msgr2=0x7fe190103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.747+0000 7fe197164700 1 -- 172.21.15.116:0/453819822 shutdown_connections 2024-03-19T01:03:12.766 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.747+0000 7fe197164700 1 -- 172.21.15.116:0/453819822 wait complete. 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.747+0000 7fe197164700 1 Processor -- start 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.747+0000 7fe197164700 1 -- start start 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.748+0000 7fe197164700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 0x7fe190106650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.748+0000 7fe197164700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe19010b140 con 0x7fe190107070 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.748+0000 7fe194f00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 0x7fe190106650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.748+0000 7fe194f00700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 0x7fe190106650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44170/0 (socket says 172.21.15.116:44170) 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.748+0000 7fe194f00700 1 -- 172.21.15.116:0/91860804 learned_addr learned my addr 172.21.15.116:0/91860804 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.748+0000 7fe194f00700 1 -- 172.21.15.116:0/91860804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19019e930 con 0x7fe190107070 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.748+0000 7fe194f00700 1 --2- 172.21.15.116:0/91860804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 0x7fe190106650 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe184003aa0 tx=0x7fe184003e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.748+0000 7fe18dffb700 1 -- 172.21.15.116:0/91860804 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe18400be50 con 0x7fe190107070 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.748+0000 7fe18dffb700 1 -- 172.21.15.116:0/91860804 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe184004340 con 0x7fe190107070 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.748+0000 7fe197164700 1 -- 172.21.15.116:0/91860804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe19019eb00 con 0x7fe190107070 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.748+0000 7fe18dffb700 1 -- 172.21.15.116:0/91860804 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe18401a780 con 0x7fe190107070 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.748+0000 7fe197164700 1 -- 172.21.15.116:0/91860804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe19019eea0 con 0x7fe190107070 2024-03-19T01:03:12.767 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.749+0000 7fe18dffb700 1 -- 172.21.15.116:0/91860804 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fe184022020 con 0x7fe190107070 2024-03-19T01:03:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.749+0000 7fe18dffb700 1 --2- 172.21.15.116:0/91860804 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fe180038f20 0x7fe18003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.749+0000 7fe18dffb700 1 -- 172.21.15.116:0/91860804 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe18404cea0 con 0x7fe190107070 2024-03-19T01:03:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.749+0000 7fe197164700 1 -- 172.21.15.116:0/91860804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe190063940 con 0x7fe190107070 2024-03-19T01:03:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.750+0000 7fe18ffff700 1 --2- 172.21.15.116:0/91860804 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fe180038f20 0x7fe18003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.750+0000 7fe18ffff700 1 --2- 172.21.15.116:0/91860804 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fe180038f20 0x7fe18003b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe17c006fd0 tx=0x7fe17c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.754+0000 7fe18dffb700 1 -- 172.21.15.116:0/91860804 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe18402c3a0 con 0x7fe190107070 2024-03-19T01:03:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:07.954+0000 7fe197164700 1 -- 172.21.15.116:0/91860804 --> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi116", "addr": "172.21.15.116", "target": ["mon-mgr", ""]}) -- 0x7fe190102b90 con 0x7fe180038f20 2024-03-19T01:03:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:12.163+0000 7fe18dffb700 1 -- 172.21.15.116:0/91860804 <== mgr.14118 v2:172.21.15.116:6800/1107242393 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fe190102b90 con 0x7fe180038f20 2024-03-19T01:03:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:12.166+0000 7fe197164700 1 -- 172.21.15.116:0/91860804 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fe180038f20 msgr2=0x7fe18003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:12.166+0000 7fe197164700 1 --2- 172.21.15.116:0/91860804 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fe180038f20 0x7fe18003b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe17c006fd0 tx=0x7fe17c006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:12.166+0000 7fe197164700 1 -- 172.21.15.116:0/91860804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 msgr2=0x7fe190106650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:12.166+0000 7fe197164700 1 --2- 172.21.15.116:0/91860804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 0x7fe190106650 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe184003aa0 tx=0x7fe184003e60 comp rx=0 tx=0).stop 2024-03-19T01:03:12.768 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:12.166+0000 7fe197164700 1 -- 172.21.15.116:0/91860804 shutdown_connections 2024-03-19T01:03:12.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:12.166+0000 7fe197164700 1 --2- 172.21.15.116:0/91860804 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7fe180038f20 0x7fe18003b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:12.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:12.166+0000 7fe197164700 1 --2- 172.21.15.116:0/91860804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe190107070 0x7fe190106650 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:12.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:12.166+0000 7fe197164700 1 -- 172.21.15.116:0/91860804 >> 172.21.15.116:0/91860804 conn(0x7fe1901018d0 msgr2=0x7fe190102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:12.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:12.166+0000 7fe197164700 1 -- 172.21.15.116:0/91860804 shutdown_connections 2024-03-19T01:03:12.769 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:12.166+0000 7fe197164700 1 -- 172.21.15.116:0/91860804 wait complete. 2024-03-19T01:03:12.769 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying unmanaged mon service... 2024-03-19T01:03:13.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:13 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:13.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:13 smithi116 ceph-mon[62701]: Added host smithi116 2024-03-19T01:03:13.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:13 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:03:14.089 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-19T01:03:14.089 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.156+0000 7f6a86ef7700 1 Processor -- start 2024-03-19T01:03:14.089 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.156+0000 7f6a86ef7700 1 -- start start 2024-03-19T01:03:14.089 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.156+0000 7f6a86ef7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 0x7f6a80106920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:14.089 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.156+0000 7f6a86ef7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8007be30 con 0x7f6a80106500 2024-03-19T01:03:14.089 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.156+0000 7f6a84c93700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 0x7f6a80106920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:14.089 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.156+0000 7f6a84c93700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 0x7f6a80106920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44186/0 (socket says 172.21.15.116:44186) 2024-03-19T01:03:14.090 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.156+0000 7f6a84c93700 1 -- 172.21.15.116:0/2296841074 learned_addr learned my addr 172.21.15.116:0/2296841074 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:14.090 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.157+0000 7f6a84c93700 1 -- 172.21.15.116:0/2296841074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a80106e60 con 0x7f6a80106500 2024-03-19T01:03:14.090 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.157+0000 7f6a84c93700 1 --2- 172.21.15.116:0/2296841074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 0x7f6a80106920 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6a74009ab0 tx=0x7f6a74009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b3a85f8517a4782 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:14.090 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.157+0000 7f6a7f7fe700 1 -- 172.21.15.116:0/2296841074 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a74004160 con 0x7f6a80106500 2024-03-19T01:03:14.090 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.157+0000 7f6a7f7fe700 1 -- 172.21.15.116:0/2296841074 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6a7400bca0 con 0x7f6a80106500 2024-03-19T01:03:14.090 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.157+0000 7f6a7f7fe700 1 -- 172.21.15.116:0/2296841074 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a74003d60 con 0x7f6a80106500 2024-03-19T01:03:14.090 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.157+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2296841074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 msgr2=0x7f6a80106920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:14.090 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.157+0000 7f6a86ef7700 1 --2- 172.21.15.116:0/2296841074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 0x7f6a80106920 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6a74009ab0 tx=0x7f6a74009dc0 comp rx=0 tx=0).stop 2024-03-19T01:03:14.090 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.158+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2296841074 shutdown_connections 2024-03-19T01:03:14.090 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.158+0000 7f6a86ef7700 1 --2- 172.21.15.116:0/2296841074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 0x7f6a80106920 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:14.090 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.158+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2296841074 >> 172.21.15.116:0/2296841074 conn(0x7f6a80102140 msgr2=0x7f6a80104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:14.090 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.158+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2296841074 shutdown_connections 2024-03-19T01:03:14.091 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.158+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2296841074 wait complete. 2024-03-19T01:03:14.091 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.158+0000 7f6a86ef7700 1 Processor -- start 2024-03-19T01:03:14.091 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.158+0000 7f6a86ef7700 1 -- start start 2024-03-19T01:03:14.091 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.158+0000 7f6a86ef7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 0x7f6a8019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:14.091 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.158+0000 7f6a86ef7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a80107e80 con 0x7f6a80106500 2024-03-19T01:03:14.091 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.158+0000 7f6a84c93700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 0x7f6a8019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:14.091 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.158+0000 7f6a84c93700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 0x7f6a8019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44198/0 (socket says 172.21.15.116:44198) 2024-03-19T01:03:14.091 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.158+0000 7f6a84c93700 1 -- 172.21.15.116:0/2661474228 learned_addr learned my addr 172.21.15.116:0/2661474228 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:14.091 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.159+0000 7f6a84c93700 1 -- 172.21.15.116:0/2661474228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a8019e360 con 0x7f6a80106500 2024-03-19T01:03:14.091 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.159+0000 7f6a84c93700 1 --2- 172.21.15.116:0/2661474228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 0x7f6a8019de20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a74003aa0 tx=0x7f6a74003e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:14.091 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.159+0000 7f6a7dffb700 1 -- 172.21.15.116:0/2661474228 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a7400be50 con 0x7f6a80106500 2024-03-19T01:03:14.091 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.159+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2661474228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a8019e5f0 con 0x7f6a80106500 2024-03-19T01:03:14.091 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.159+0000 7f6a7dffb700 1 -- 172.21.15.116:0/2661474228 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f6a74004340 con 0x7f6a80106500 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.159+0000 7f6a7dffb700 1 -- 172.21.15.116:0/2661474228 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a7401a780 con 0x7f6a80106500 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.159+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2661474228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a801a12e0 con 0x7f6a80106500 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.160+0000 7f6a7dffb700 1 -- 172.21.15.116:0/2661474228 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f6a74022020 con 0x7f6a80106500 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.160+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2661474228 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a80063940 con 0x7f6a80106500 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.160+0000 7f6a7dffb700 1 --2- 172.21.15.116:0/2661474228 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f6a70038ed0 0x7f6a7003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.160+0000 7f6a7dffb700 1 -- 172.21.15.116:0/2661474228 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6a7404cec0 con 0x7f6a80106500 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.160+0000 7f6a7ffff700 1 --2- 172.21.15.116:0/2661474228 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f6a70038ed0 0x7f6a7003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.161+0000 7f6a7ffff700 1 --2- 172.21.15.116:0/2661474228 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f6a70038ed0 0x7f6a7003b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c006fd0 tx=0x7f6a6c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.165+0000 7f6a7dffb700 1 -- 172.21.15.116:0/2661474228 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6a7402ddb0 con 0x7f6a80106500 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.373+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2661474228 --> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f6a800008d0 con 0x7f6a70038ed0 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.381+0000 7f6a7dffb700 1 -- 172.21.15.116:0/2661474228 <== mgr.14118 v2:172.21.15.116:6800/1107242393 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6a800008d0 con 0x7f6a70038ed0 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.383+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2661474228 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f6a70038ed0 msgr2=0x7f6a7003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.383+0000 7f6a86ef7700 1 --2- 172.21.15.116:0/2661474228 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f6a70038ed0 0x7f6a7003b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c006fd0 tx=0x7f6a6c006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:14.092 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.383+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2661474228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 msgr2=0x7f6a8019de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:14.093 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.383+0000 7f6a86ef7700 1 --2- 172.21.15.116:0/2661474228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 0x7f6a8019de20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6a74003aa0 tx=0x7f6a74003e60 comp rx=0 tx=0).stop 2024-03-19T01:03:14.093 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.383+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2661474228 shutdown_connections 2024-03-19T01:03:14.093 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.383+0000 7f6a86ef7700 1 --2- 172.21.15.116:0/2661474228 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f6a70038ed0 0x7f6a7003b390 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:14.093 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.383+0000 7f6a86ef7700 1 --2- 172.21.15.116:0/2661474228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a80106500 0x7f6a8019de20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:14.093 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.383+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2661474228 >> 172.21.15.116:0/2661474228 conn(0x7f6a80102140 msgr2=0x7f6a80194a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:14.093 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.383+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2661474228 shutdown_connections 2024-03-19T01:03:14.093 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:13.383+0000 7f6a86ef7700 1 -- 172.21.15.116:0/2661474228 wait complete. 2024-03-19T01:03:14.093 INFO:teuthology.orchestra.run.smithi116.stdout:Deploying unmanaged mgr service... 2024-03-19T01:03:14.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:14 smithi116 ceph-mon[62701]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:03:14.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:14 smithi116 ceph-mon[62701]: Saving service mon spec with placement count:5 2024-03-19T01:03:14.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:14 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:15.475 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:15 smithi116 ceph-mon[62701]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:03:15.475 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:15 smithi116 ceph-mon[62701]: Saving service mgr spec with placement count:2 2024-03-19T01:03:15.475 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:15 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:16.111 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.597+0000 7f32e0560700 1 Processor -- start 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.597+0000 7f32e0560700 1 -- start start 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.598+0000 7f32e0560700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 0x7f32d810abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.598+0000 7f32e0560700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f32d807bdb0 con 0x7f32d81087c0 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.598+0000 7f32de2fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 0x7f32d810abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.598+0000 7f32de2fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 0x7f32d810abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44210/0 (socket says 172.21.15.116:44210) 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.598+0000 7f32de2fc700 1 -- 172.21.15.116:0/240967935 learned_addr learned my addr 172.21.15.116:0/240967935 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.598+0000 7f32de2fc700 1 -- 172.21.15.116:0/240967935 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d810b0f0 con 0x7f32d81087c0 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.598+0000 7f32de2fc700 1 --2- 172.21.15.116:0/240967935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 0x7f32d810abb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f32d40098a0 tx=0x7f32d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3358375cfbd54fd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.599+0000 7f32dd2fa700 1 -- 172.21.15.116:0/240967935 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32d4004160 con 0x7f32d81087c0 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.599+0000 7f32dd2fa700 1 -- 172.21.15.116:0/240967935 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f32d4004300 con 0x7f32d81087c0 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.599+0000 7f32dd2fa700 1 -- 172.21.15.116:0/240967935 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32d4011b50 con 0x7f32d81087c0 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.599+0000 7f32e0560700 1 -- 172.21.15.116:0/240967935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 msgr2=0x7f32d810abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:16.112 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.599+0000 7f32e0560700 1 --2- 172.21.15.116:0/240967935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 0x7f32d810abb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f32d40098a0 tx=0x7f32d4009bb0 comp rx=0 tx=0).stop 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.599+0000 7f32e0560700 1 -- 172.21.15.116:0/240967935 shutdown_connections 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.599+0000 7f32e0560700 1 --2- 172.21.15.116:0/240967935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 0x7f32d810abb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.599+0000 7f32e0560700 1 -- 172.21.15.116:0/240967935 >> 172.21.15.116:0/240967935 conn(0x7f32d8102190 msgr2=0x7f32d81045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.599+0000 7f32e0560700 1 -- 172.21.15.116:0/240967935 shutdown_connections 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.599+0000 7f32e0560700 1 -- 172.21.15.116:0/240967935 wait complete. 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.600+0000 7f32e0560700 1 Processor -- start 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.600+0000 7f32e0560700 1 -- start start 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.600+0000 7f32e0560700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 0x7f32d819bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.600+0000 7f32e0560700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f32d810c040 con 0x7f32d81087c0 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.600+0000 7f32de2fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 0x7f32d819bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.600+0000 7f32de2fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 0x7f32d819bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44226/0 (socket says 172.21.15.116:44226) 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.600+0000 7f32de2fc700 1 -- 172.21.15.116:0/3930468436 learned_addr learned my addr 172.21.15.116:0/3930468436 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.601+0000 7f32de2fc700 1 -- 172.21.15.116:0/3930468436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d819c160 con 0x7f32d81087c0 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.601+0000 7f32de2fc700 1 --2- 172.21.15.116:0/3930468436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 0x7f32d819bc20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f32d4011b60 tx=0x7f32d4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:16.113 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.601+0000 7f32cf7fe700 1 -- 172.21.15.116:0/3930468436 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32d4003970 con 0x7f32d81087c0 2024-03-19T01:03:16.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.601+0000 7f32cf7fe700 1 -- 172.21.15.116:0/3930468436 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f32d4003b10 con 0x7f32d81087c0 2024-03-19T01:03:16.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.601+0000 7f32e0560700 1 -- 172.21.15.116:0/3930468436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32d819c3f0 con 0x7f32d81087c0 2024-03-19T01:03:16.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.601+0000 7f32cf7fe700 1 -- 172.21.15.116:0/3930468436 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f32d4024610 con 0x7f32d81087c0 2024-03-19T01:03:16.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.601+0000 7f32e0560700 1 -- 172.21.15.116:0/3930468436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32d819f0e0 con 0x7f32d81087c0 2024-03-19T01:03:16.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.602+0000 7f32cf7fe700 1 -- 172.21.15.116:0/3930468436 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f32d40248c0 con 0x7f32d81087c0 2024-03-19T01:03:16.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.602+0000 7f32e0560700 1 -- 172.21.15.116:0/3930468436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32d8063940 con 0x7f32d81087c0 2024-03-19T01:03:16.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.602+0000 7f32cf7fe700 1 --2- 172.21.15.116:0/3930468436 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f32c4038f70 0x7f32c403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:16.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.602+0000 7f32cf7fe700 1 -- 172.21.15.116:0/3930468436 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f32d4011e00 con 0x7f32d81087c0 2024-03-19T01:03:16.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.602+0000 7f32ddafb700 1 --2- 172.21.15.116:0/3930468436 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f32c4038f70 0x7f32c403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:16.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.602+0000 7f32ddafb700 1 --2- 172.21.15.116:0/3930468436 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f32c4038f70 0x7f32c403b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f32c8006fd0 tx=0x7f32c8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:16.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.607+0000 7f32cf7fe700 1 -- 172.21.15.116:0/3930468436 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f32d400ba00 con 0x7f32d81087c0 2024-03-19T01:03:16.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.830+0000 7f32e0560700 1 -- 172.21.15.116:0/3930468436 --> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f32d81033e0 con 0x7f32c4038f70 2024-03-19T01:03:16.114 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.837+0000 7f32cf7fe700 1 -- 172.21.15.116:0/3930468436 <== mgr.14118 v2:172.21.15.116:6800/1107242393 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f32d81033e0 con 0x7f32c4038f70 2024-03-19T01:03:16.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.839+0000 7f32e0560700 1 -- 172.21.15.116:0/3930468436 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f32c4038f70 msgr2=0x7f32c403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:16.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.839+0000 7f32e0560700 1 --2- 172.21.15.116:0/3930468436 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f32c4038f70 0x7f32c403b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f32c8006fd0 tx=0x7f32c8006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:16.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.839+0000 7f32e0560700 1 -- 172.21.15.116:0/3930468436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 msgr2=0x7f32d819bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:16.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.839+0000 7f32e0560700 1 --2- 172.21.15.116:0/3930468436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 0x7f32d819bc20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f32d4011b60 tx=0x7f32d4011c40 comp rx=0 tx=0).stop 2024-03-19T01:03:16.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.839+0000 7f32e0560700 1 -- 172.21.15.116:0/3930468436 shutdown_connections 2024-03-19T01:03:16.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.839+0000 7f32e0560700 1 --2- 172.21.15.116:0/3930468436 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f32c4038f70 0x7f32c403b430 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:16.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.839+0000 7f32e0560700 1 --2- 172.21.15.116:0/3930468436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32d81087c0 0x7f32d819bc20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:16.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.839+0000 7f32e0560700 1 -- 172.21.15.116:0/3930468436 >> 172.21.15.116:0/3930468436 conn(0x7f32d8102190 msgr2=0x7f32d8102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:16.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.839+0000 7f32e0560700 1 -- 172.21.15.116:0/3930468436 shutdown_connections 2024-03-19T01:03:16.115 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:14.839+0000 7f32e0560700 1 -- 172.21.15.116:0/3930468436 wait complete. 2024-03-19T01:03:17.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:16 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:17.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:16 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:17.400 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.635+0000 7f1a6dd77700 1 Processor -- start 2024-03-19T01:03:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.635+0000 7f1a6dd77700 1 -- start start 2024-03-19T01:03:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.635+0000 7f1a6dd77700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 0x7f1a68106350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.635+0000 7f1a6dd77700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a68106920 con 0x7f1a68105f30 2024-03-19T01:03:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.635+0000 7f1a677fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 0x7f1a68106350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.635+0000 7f1a677fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 0x7f1a68106350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43934/0 (socket says 172.21.15.116:43934) 2024-03-19T01:03:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.635+0000 7f1a677fe700 1 -- 172.21.15.116:0/3024634010 learned_addr learned my addr 172.21.15.116:0/3024634010 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.635+0000 7f1a677fe700 1 -- 172.21.15.116:0/3024634010 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a68106aa0 con 0x7f1a68105f30 2024-03-19T01:03:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.635+0000 7f1a677fe700 1 --2- 172.21.15.116:0/3024634010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 0x7f1a68106350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1a58009ab0 tx=0x7f1a58009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eead9ef7d68deaf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.636+0000 7f1a667fc700 1 -- 172.21.15.116:0/3024634010 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a58004160 con 0x7f1a68105f30 2024-03-19T01:03:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.636+0000 7f1a667fc700 1 -- 172.21.15.116:0/3024634010 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a5800bcc0 con 0x7f1a68105f30 2024-03-19T01:03:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.636+0000 7f1a667fc700 1 -- 172.21.15.116:0/3024634010 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a58003d00 con 0x7f1a68105f30 2024-03-19T01:03:17.401 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.636+0000 7f1a6dd77700 1 -- 172.21.15.116:0/3024634010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 msgr2=0x7f1a68106350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.636+0000 7f1a6dd77700 1 --2- 172.21.15.116:0/3024634010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 0x7f1a68106350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1a58009ab0 tx=0x7f1a58009dc0 comp rx=0 tx=0).stop 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.636+0000 7f1a6dd77700 1 -- 172.21.15.116:0/3024634010 shutdown_connections 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.636+0000 7f1a6dd77700 1 --2- 172.21.15.116:0/3024634010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 0x7f1a68106350 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.636+0000 7f1a6dd77700 1 -- 172.21.15.116:0/3024634010 >> 172.21.15.116:0/3024634010 conn(0x7f1a681018d0 msgr2=0x7f1a68103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.636+0000 7f1a6dd77700 1 -- 172.21.15.116:0/3024634010 shutdown_connections 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.636+0000 7f1a6dd77700 1 -- 172.21.15.116:0/3024634010 wait complete. 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.637+0000 7f1a6dd77700 1 Processor -- start 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.637+0000 7f1a6dd77700 1 -- start start 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.637+0000 7f1a6dd77700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 0x7f1a68108830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.637+0000 7f1a6dd77700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a6810c1e0 con 0x7f1a68105f30 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.637+0000 7f1a677fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 0x7f1a68108830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.637+0000 7f1a677fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 0x7f1a68108830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43946/0 (socket says 172.21.15.116:43946) 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.637+0000 7f1a677fe700 1 -- 172.21.15.116:0/2966296382 learned_addr learned my addr 172.21.15.116:0/2966296382 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:17.402 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.638+0000 7f1a677fe700 1 -- 172.21.15.116:0/2966296382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a68108d70 con 0x7f1a68105f30 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.638+0000 7f1a677fe700 1 --2- 172.21.15.116:0/2966296382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 0x7f1a68108830 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1a5800ba10 tx=0x7f1a5800baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.638+0000 7f1a64ff9700 1 -- 172.21.15.116:0/2966296382 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a5800be70 con 0x7f1a68105f30 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.638+0000 7f1a64ff9700 1 -- 172.21.15.116:0/2966296382 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a5801a430 con 0x7f1a68105f30 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.638+0000 7f1a64ff9700 1 -- 172.21.15.116:0/2966296382 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a580115b0 con 0x7f1a68105f30 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.638+0000 7f1a6dd77700 1 -- 172.21.15.116:0/2966296382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a68106f60 con 0x7f1a68105f30 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.638+0000 7f1a6dd77700 1 -- 172.21.15.116:0/2966296382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a681074a0 con 0x7f1a68105f30 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.639+0000 7f1a64ff9700 1 -- 172.21.15.116:0/2966296382 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f1a580117a0 con 0x7f1a68105f30 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.639+0000 7f1a64ff9700 1 --2- 172.21.15.116:0/2966296382 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f1a50038f70 0x7f1a5003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.639+0000 7f1a64ff9700 1 -- 172.21.15.116:0/2966296382 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1a5804e090 con 0x7f1a68105f30 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.639+0000 7f1a6dd77700 1 -- 172.21.15.116:0/2966296382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a68063940 con 0x7f1a68105f30 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.639+0000 7f1a66ffd700 1 --2- 172.21.15.116:0/2966296382 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f1a50038f70 0x7f1a5003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.639+0000 7f1a66ffd700 1 --2- 172.21.15.116:0/2966296382 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f1a50038f70 0x7f1a5003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c006fd0 tx=0x7f1a5c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.644+0000 7f1a64ff9700 1 -- 172.21.15.116:0/2966296382 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1a5802cc30 con 0x7f1a68105f30 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.846+0000 7f1a6dd77700 1 -- 172.21.15.116:0/2966296382 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f1a68107db0 con 0x7f1a68105f30 2024-03-19T01:03:17.403 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.849+0000 7f1a64ff9700 1 -- 172.21.15.116:0/2966296382 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f1a5802d7c0 con 0x7f1a68105f30 2024-03-19T01:03:17.404 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.852+0000 7f1a6dd77700 1 -- 172.21.15.116:0/2966296382 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f1a50038f70 msgr2=0x7f1a5003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:17.404 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.852+0000 7f1a6dd77700 1 --2- 172.21.15.116:0/2966296382 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f1a50038f70 0x7f1a5003b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c006fd0 tx=0x7f1a5c006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:17.404 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.852+0000 7f1a6dd77700 1 -- 172.21.15.116:0/2966296382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 msgr2=0x7f1a68108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:17.404 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.852+0000 7f1a6dd77700 1 --2- 172.21.15.116:0/2966296382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 0x7f1a68108830 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1a5800ba10 tx=0x7f1a5800baf0 comp rx=0 tx=0).stop 2024-03-19T01:03:17.404 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.852+0000 7f1a6dd77700 1 -- 172.21.15.116:0/2966296382 shutdown_connections 2024-03-19T01:03:17.404 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.852+0000 7f1a6dd77700 1 --2- 172.21.15.116:0/2966296382 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f1a50038f70 0x7f1a5003b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:17.404 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.852+0000 7f1a6dd77700 1 --2- 172.21.15.116:0/2966296382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a68105f30 0x7f1a68108830 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:17.404 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.852+0000 7f1a6dd77700 1 -- 172.21.15.116:0/2966296382 >> 172.21.15.116:0/2966296382 conn(0x7f1a681018d0 msgr2=0x7f1a68102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:17.404 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.852+0000 7f1a6dd77700 1 -- 172.21.15.116:0/2966296382 shutdown_connections 2024-03-19T01:03:17.404 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:16.852+0000 7f1a6dd77700 1 -- 172.21.15.116:0/2966296382 wait complete. 2024-03-19T01:03:18.002 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:17 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2966296382' entity='client.admin' 2024-03-19T01:03:19.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.793+0000 7f5dc5006700 1 Processor -- start 2024-03-19T01:03:19.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.793+0000 7f5dc5006700 1 -- start start 2024-03-19T01:03:19.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.794+0000 7f5dc5006700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 0x7f5dc010ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:19.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.794+0000 7f5dc5006700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc007bdd0 con 0x7f5dc0108780 2024-03-19T01:03:19.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.794+0000 7f5dbed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 0x7f5dc010ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:19.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.794+0000 7f5dbed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 0x7f5dc010ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43960/0 (socket says 172.21.15.116:43960) 2024-03-19T01:03:19.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.794+0000 7f5dbed9d700 1 -- 172.21.15.116:0/2152590512 learned_addr learned my addr 172.21.15.116:0/2152590512 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:19.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.794+0000 7f5dbed9d700 1 -- 172.21.15.116:0/2152590512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc010b0b0 con 0x7f5dc0108780 2024-03-19T01:03:19.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.794+0000 7f5dbed9d700 1 --2- 172.21.15.116:0/2152590512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 0x7f5dc010ab70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5db00098a0 tx=0x7f5db0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c40eb4bc36b0a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:19.017 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.794+0000 7f5dbdd9b700 1 -- 172.21.15.116:0/2152590512 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5db0004160 con 0x7f5dc0108780 2024-03-19T01:03:19.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.794+0000 7f5dbdd9b700 1 -- 172.21.15.116:0/2152590512 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5db0004300 con 0x7f5dc0108780 2024-03-19T01:03:19.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.794+0000 7f5dbdd9b700 1 -- 172.21.15.116:0/2152590512 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5db0011b20 con 0x7f5dc0108780 2024-03-19T01:03:19.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.795+0000 7f5dc5006700 1 -- 172.21.15.116:0/2152590512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 msgr2=0x7f5dc010ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:19.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.795+0000 7f5dc5006700 1 --2- 172.21.15.116:0/2152590512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 0x7f5dc010ab70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5db00098a0 tx=0x7f5db0009bb0 comp rx=0 tx=0).stop 2024-03-19T01:03:19.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.795+0000 7f5dc5006700 1 -- 172.21.15.116:0/2152590512 shutdown_connections 2024-03-19T01:03:19.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.795+0000 7f5dc5006700 1 --2- 172.21.15.116:0/2152590512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 0x7f5dc010ab70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:19.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.795+0000 7f5dc5006700 1 -- 172.21.15.116:0/2152590512 >> 172.21.15.116:0/2152590512 conn(0x7f5dc0102150 msgr2=0x7f5dc0104590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:19.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.795+0000 7f5dc5006700 1 -- 172.21.15.116:0/2152590512 shutdown_connections 2024-03-19T01:03:19.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.795+0000 7f5dc5006700 1 -- 172.21.15.116:0/2152590512 wait complete. 2024-03-19T01:03:19.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.796+0000 7f5dc5006700 1 Processor -- start 2024-03-19T01:03:19.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.796+0000 7f5dc5006700 1 -- start start 2024-03-19T01:03:19.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.796+0000 7f5dc5006700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 0x7f5dc019bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:19.018 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.796+0000 7f5dc5006700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc010c0c0 con 0x7f5dc0108780 2024-03-19T01:03:19.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.796+0000 7f5dbed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 0x7f5dc019bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:19.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.796+0000 7f5dbed9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 0x7f5dc019bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43968/0 (socket says 172.21.15.116:43968) 2024-03-19T01:03:19.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.796+0000 7f5dbed9d700 1 -- 172.21.15.116:0/3778680418 learned_addr learned my addr 172.21.15.116:0/3778680418 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:19.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.796+0000 7f5dbed9d700 1 -- 172.21.15.116:0/3778680418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc019c0e0 con 0x7f5dc0108780 2024-03-19T01:03:19.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.796+0000 7f5dbed9d700 1 --2- 172.21.15.116:0/3778680418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 0x7f5dc019bba0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5db0011b60 tx=0x7f5db0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:19.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.797+0000 7f5da7fff700 1 -- 172.21.15.116:0/3778680418 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5db0003b60 con 0x7f5dc0108780 2024-03-19T01:03:19.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.797+0000 7f5dc5006700 1 -- 172.21.15.116:0/3778680418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc019c370 con 0x7f5dc0108780 2024-03-19T01:03:19.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.797+0000 7f5da7fff700 1 -- 172.21.15.116:0/3778680418 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5db001a910 con 0x7f5dc0108780 2024-03-19T01:03:19.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.797+0000 7f5da7fff700 1 -- 172.21.15.116:0/3778680418 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5db0024930 con 0x7f5dc0108780 2024-03-19T01:03:19.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.797+0000 7f5dc5006700 1 -- 172.21.15.116:0/3778680418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc019c7d0 con 0x7f5dc0108780 2024-03-19T01:03:19.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.798+0000 7f5da7fff700 1 -- 172.21.15.116:0/3778680418 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f5db0022020 con 0x7f5dc0108780 2024-03-19T01:03:19.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.798+0000 7f5dc5006700 1 -- 172.21.15.116:0/3778680418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dc0063940 con 0x7f5dc0108780 2024-03-19T01:03:19.019 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.798+0000 7f5da7fff700 1 --2- 172.21.15.116:0/3778680418 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f5da8038f20 0x7f5da803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:19.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.798+0000 7f5da7fff700 1 -- 172.21.15.116:0/3778680418 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5db004d1b0 con 0x7f5dc0108780 2024-03-19T01:03:19.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.798+0000 7f5dbe59c700 1 --2- 172.21.15.116:0/3778680418 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f5da8038f20 0x7f5da803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:19.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.798+0000 7f5dbe59c700 1 --2- 172.21.15.116:0/3778680418 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f5da8038f20 0x7f5da803b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5db4006fd0 tx=0x7f5db4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:19.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:17.803+0000 7f5da7fff700 1 -- 172.21.15.116:0/3778680418 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5db001ab30 con 0x7f5dc0108780 2024-03-19T01:03:19.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:18.001+0000 7f5dc5006700 1 -- 172.21.15.116:0/3778680418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f5dc019f1a0 con 0x7f5dc0108780 2024-03-19T01:03:19.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:18.004+0000 7f5da7fff700 1 -- 172.21.15.116:0/3778680418 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f5db002d910 con 0x7f5dc0108780 2024-03-19T01:03:19.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:18.006+0000 7f5dc5006700 1 -- 172.21.15.116:0/3778680418 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f5da8038f20 msgr2=0x7f5da803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:19.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:18.006+0000 7f5dc5006700 1 --2- 172.21.15.116:0/3778680418 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f5da8038f20 0x7f5da803b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5db4006fd0 tx=0x7f5db4006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:19.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:18.006+0000 7f5dc5006700 1 -- 172.21.15.116:0/3778680418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 msgr2=0x7f5dc019bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:19.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:18.006+0000 7f5dc5006700 1 --2- 172.21.15.116:0/3778680418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 0x7f5dc019bba0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5db0011b60 tx=0x7f5db0011c40 comp rx=0 tx=0).stop 2024-03-19T01:03:19.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:18.007+0000 7f5dc5006700 1 -- 172.21.15.116:0/3778680418 shutdown_connections 2024-03-19T01:03:19.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:18.007+0000 7f5dc5006700 1 --2- 172.21.15.116:0/3778680418 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f5da8038f20 0x7f5da803b3e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:19.020 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:18.007+0000 7f5dc5006700 1 --2- 172.21.15.116:0/3778680418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5dc0108780 0x7f5dc019bba0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:19.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:18.007+0000 7f5dc5006700 1 -- 172.21.15.116:0/3778680418 >> 172.21.15.116:0/3778680418 conn(0x7f5dc0102150 msgr2=0x7f5dc0102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:19.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:18.007+0000 7f5dc5006700 1 -- 172.21.15.116:0/3778680418 shutdown_connections 2024-03-19T01:03:19.021 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:18.007+0000 7f5dc5006700 1 -- 172.21.15.116:0/3778680418 wait complete. 2024-03-19T01:03:19.021 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling the dashboard module... 2024-03-19T01:03:19.126 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:19 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3778680418' entity='client.admin' 2024-03-19T01:03:19.127 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:19 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:20.299 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:20 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4082776632' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-19T01:03:20.299 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:20 smithi116 ceph-mon[62701]: from='mgr.14118 172.21.15.116:0/964155970' entity='mgr.x' 2024-03-19T01:03:20.300 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:20 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: ignoring --setuser ceph since I am not root 2024-03-19T01:03:20.300 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:20 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: ignoring --setgroup ceph since I am not root 2024-03-19T01:03:20.760 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.391+0000 7f53cc2d9700 1 Processor -- start 2024-03-19T01:03:20.760 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.391+0000 7f53cc2d9700 1 -- start start 2024-03-19T01:03:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.391+0000 7f53cc2d9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 0x7f53c410abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.391+0000 7f53cc2d9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f53c407bd70 con 0x7f53c41087d0 2024-03-19T01:03:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.391+0000 7f53ca075700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 0x7f53c410abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.392+0000 7f53ca075700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 0x7f53c410abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43978/0 (socket says 172.21.15.116:43978) 2024-03-19T01:03:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.392+0000 7f53ca075700 1 -- 172.21.15.116:0/2943577223 learned_addr learned my addr 172.21.15.116:0/2943577223 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.392+0000 7f53ca075700 1 -- 172.21.15.116:0/2943577223 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c410b100 con 0x7f53c41087d0 2024-03-19T01:03:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.392+0000 7f53ca075700 1 --2- 172.21.15.116:0/2943577223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 0x7f53c410abc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f53c00098a0 tx=0x7f53c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1bcd4a0617fae3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.392+0000 7f53c9073700 1 -- 172.21.15.116:0/2943577223 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53c0004160 con 0x7f53c41087d0 2024-03-19T01:03:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.392+0000 7f53c9073700 1 -- 172.21.15.116:0/2943577223 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53c0004300 con 0x7f53c41087d0 2024-03-19T01:03:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.392+0000 7f53c9073700 1 -- 172.21.15.116:0/2943577223 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53c0011b20 con 0x7f53c41087d0 2024-03-19T01:03:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.393+0000 7f53cc2d9700 1 -- 172.21.15.116:0/2943577223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 msgr2=0x7f53c410abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.393+0000 7f53cc2d9700 1 --2- 172.21.15.116:0/2943577223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 0x7f53c410abc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f53c00098a0 tx=0x7f53c0009bb0 comp rx=0 tx=0).stop 2024-03-19T01:03:20.761 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.393+0000 7f53cc2d9700 1 -- 172.21.15.116:0/2943577223 shutdown_connections 2024-03-19T01:03:20.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.393+0000 7f53cc2d9700 1 --2- 172.21.15.116:0/2943577223 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 0x7f53c410abc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:20.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.393+0000 7f53cc2d9700 1 -- 172.21.15.116:0/2943577223 >> 172.21.15.116:0/2943577223 conn(0x7f53c4102140 msgr2=0x7f53c4104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:20.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.393+0000 7f53cc2d9700 1 -- 172.21.15.116:0/2943577223 shutdown_connections 2024-03-19T01:03:20.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.393+0000 7f53cc2d9700 1 -- 172.21.15.116:0/2943577223 wait complete. 2024-03-19T01:03:20.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.394+0000 7f53cc2d9700 1 Processor -- start 2024-03-19T01:03:20.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.394+0000 7f53cc2d9700 1 -- start start 2024-03-19T01:03:20.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.394+0000 7f53cc2d9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 0x7f53c419ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:20.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.394+0000 7f53cc2d9700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f53c410c0b0 con 0x7f53c41087d0 2024-03-19T01:03:20.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.394+0000 7f53ca075700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 0x7f53c419ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:20.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.394+0000 7f53ca075700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 0x7f53c419ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43994/0 (socket says 172.21.15.116:43994) 2024-03-19T01:03:20.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.394+0000 7f53ca075700 1 -- 172.21.15.116:0/4082776632 learned_addr learned my addr 172.21.15.116:0/4082776632 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:20.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.394+0000 7f53ca075700 1 -- 172.21.15.116:0/4082776632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c419e300 con 0x7f53c41087d0 2024-03-19T01:03:20.762 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.395+0000 7f53ca075700 1 --2- 172.21.15.116:0/4082776632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 0x7f53c419ddc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f53c0011b60 tx=0x7f53c0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:20.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.395+0000 7f53bb7fe700 1 -- 172.21.15.116:0/4082776632 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53c0003b60 con 0x7f53c41087d0 2024-03-19T01:03:20.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.395+0000 7f53bb7fe700 1 -- 172.21.15.116:0/4082776632 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53c001a910 con 0x7f53c41087d0 2024-03-19T01:03:20.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.395+0000 7f53bb7fe700 1 -- 172.21.15.116:0/4082776632 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f53c0024930 con 0x7f53c41087d0 2024-03-19T01:03:20.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.395+0000 7f53cc2d9700 1 -- 172.21.15.116:0/4082776632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c419e590 con 0x7f53c41087d0 2024-03-19T01:03:20.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.395+0000 7f53cc2d9700 1 -- 172.21.15.116:0/4082776632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c419e9f0 con 0x7f53c41087d0 2024-03-19T01:03:20.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.396+0000 7f53cc2d9700 1 -- 172.21.15.116:0/4082776632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53c4063940 con 0x7f53c41087d0 2024-03-19T01:03:20.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.396+0000 7f53bb7fe700 1 -- 172.21.15.116:0/4082776632 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f53c0022020 con 0x7f53c41087d0 2024-03-19T01:03:20.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.396+0000 7f53bb7fe700 1 --2- 172.21.15.116:0/4082776632 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f53b0038f20 0x7f53b003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:20.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.396+0000 7f53bb7fe700 1 -- 172.21.15.116:0/4082776632 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f53c004d030 con 0x7f53c41087d0 2024-03-19T01:03:20.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.396+0000 7f53c9874700 1 --2- 172.21.15.116:0/4082776632 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f53b0038f20 0x7f53b003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:20.763 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.397+0000 7f53c9874700 1 --2- 172.21.15.116:0/4082776632 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f53b0038f20 0x7f53b003b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f53b4006fd0 tx=0x7f53b4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.401+0000 7f53bb7fe700 1 -- 172.21.15.116:0/4082776632 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f53c002d910 con 0x7f53c41087d0 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:19.671+0000 7f53cc2d9700 1 -- 172.21.15.116:0/4082776632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f53c41a13c0 con 0x7f53c41087d0 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:20.009+0000 7f53bb7fe700 1 -- 172.21.15.116:0/4082776632 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f53c004f130 con 0x7f53c41087d0 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:20.011+0000 7f53bb7fe700 1 -- 172.21.15.116:0/4082776632 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f53c002d3f0 con 0x7f53c41087d0 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:20.012+0000 7f53cc2d9700 1 -- 172.21.15.116:0/4082776632 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f53b0038f20 msgr2=0x7f53b003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:20.012+0000 7f53cc2d9700 1 --2- 172.21.15.116:0/4082776632 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f53b0038f20 0x7f53b003b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f53b4006fd0 tx=0x7f53b4006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:20.012+0000 7f53cc2d9700 1 -- 172.21.15.116:0/4082776632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 msgr2=0x7f53c419ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:20.012+0000 7f53cc2d9700 1 --2- 172.21.15.116:0/4082776632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 0x7f53c419ddc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f53c0011b60 tx=0x7f53c0011c40 comp rx=0 tx=0).stop 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:20.012+0000 7f53cc2d9700 1 -- 172.21.15.116:0/4082776632 shutdown_connections 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:20.012+0000 7f53cc2d9700 1 --2- 172.21.15.116:0/4082776632 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f53b0038f20 0x7f53b003b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:20.012+0000 7f53cc2d9700 1 --2- 172.21.15.116:0/4082776632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53c41087d0 0x7f53c419ddc0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:20.012+0000 7f53cc2d9700 1 -- 172.21.15.116:0/4082776632 >> 172.21.15.116:0/4082776632 conn(0x7f53c4102140 msgr2=0x7f53c4102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:20.012+0000 7f53cc2d9700 1 -- 172.21.15.116:0/4082776632 shutdown_connections 2024-03-19T01:03:20.764 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:20.012+0000 7f53cc2d9700 1 -- 172.21.15.116:0/4082776632 wait complete. 2024-03-19T01:03:20.905 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:20 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:20.652+0000 7f5d199d4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T01:03:20.905 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:20 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:20.749+0000 7f5d199d4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T01:03:21.288 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:21 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4082776632' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-19T01:03:21.289 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:21 smithi116 ceph-mon[62701]: mgrmap e9: x(active, since 19s) 2024-03-19T01:03:21.550 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:21 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:21.439+0000 7f5d199d4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T01:03:21.550 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:21 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:21.548+0000 7f5d199d4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T01:03:21.841 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:21 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:21.698+0000 7f5d199d4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T01:03:21.841 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:21 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:21.792+0000 7f5d199d4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T01:03:22.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:22 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/206655900' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-19T01:03:22.144 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:21 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:21.875+0000 7f5d199d4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T01:03:22.145 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:22 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:22.144+0000 7f5d199d4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T01:03:22.405 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:22 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:22.253+0000 7f5d199d4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T01:03:22.405 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:22 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:22.340+0000 7f5d199d4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T01:03:22.730 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:22 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:22.437+0000 7f5d199d4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T01:03:22.730 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:22 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:22.614+0000 7f5d199d4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T01:03:22.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T01:03:22.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-19T01:03:22.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-19T01:03:22.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-19T01:03:22.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-19T01:03:22.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T01:03:22.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.219+0000 7f36b1854700 1 Processor -- start 2024-03-19T01:03:22.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.219+0000 7f36b1854700 1 -- start start 2024-03-19T01:03:22.739 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.220+0000 7f36b1854700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 0x7f36ac106fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.220+0000 7f36b1854700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac107590 con 0x7f36ac106ba0 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.220+0000 7f36aaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 0x7f36ac106fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.220+0000 7f36aaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 0x7f36ac106fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44010/0 (socket says 172.21.15.116:44010) 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.221+0000 7f36aaffd700 1 -- 172.21.15.116:0/3933425385 learned_addr learned my addr 172.21.15.116:0/3933425385 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.221+0000 7f36aaffd700 1 -- 172.21.15.116:0/3933425385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36ac107da0 con 0x7f36ac106ba0 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.221+0000 7f36aaffd700 1 --2- 172.21.15.116:0/3933425385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 0x7f36ac106fc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f369c009ab0 tx=0x7f369c009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e76e00ee341b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.221+0000 7f36a9ffb700 1 -- 172.21.15.116:0/3933425385 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f369c004160 con 0x7f36ac106ba0 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.221+0000 7f36a9ffb700 1 -- 172.21.15.116:0/3933425385 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f369c00bcc0 con 0x7f36ac106ba0 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.222+0000 7f36a9ffb700 1 -- 172.21.15.116:0/3933425385 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f369c003d00 con 0x7f36ac106ba0 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.222+0000 7f36b1854700 1 -- 172.21.15.116:0/3933425385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 msgr2=0x7f36ac106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.222+0000 7f36b1854700 1 --2- 172.21.15.116:0/3933425385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 0x7f36ac106fc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f369c009ab0 tx=0x7f369c009dc0 comp rx=0 tx=0).stop 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.222+0000 7f36b1854700 1 -- 172.21.15.116:0/3933425385 shutdown_connections 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.222+0000 7f36b1854700 1 --2- 172.21.15.116:0/3933425385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 0x7f36ac106fc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:22.740 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.222+0000 7f36b1854700 1 -- 172.21.15.116:0/3933425385 >> 172.21.15.116:0/3933425385 conn(0x7f36ac102140 msgr2=0x7f36ac104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:22.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.222+0000 7f36b1854700 1 -- 172.21.15.116:0/3933425385 shutdown_connections 2024-03-19T01:03:22.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.222+0000 7f36b1854700 1 -- 172.21.15.116:0/3933425385 wait complete. 2024-03-19T01:03:22.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.223+0000 7f36b1854700 1 Processor -- start 2024-03-19T01:03:22.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.223+0000 7f36b1854700 1 -- start start 2024-03-19T01:03:22.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.223+0000 7f36b1854700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 0x7f36ac19bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:22.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.223+0000 7f36b1854700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36ac07af50 con 0x7f36ac106ba0 2024-03-19T01:03:22.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.224+0000 7f36aaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 0x7f36ac19bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:22.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.224+0000 7f36aaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 0x7f36ac19bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44026/0 (socket says 172.21.15.116:44026) 2024-03-19T01:03:22.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.224+0000 7f36aaffd700 1 -- 172.21.15.116:0/206655900 learned_addr learned my addr 172.21.15.116:0/206655900 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:22.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.224+0000 7f36aaffd700 1 -- 172.21.15.116:0/206655900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36ac19c190 con 0x7f36ac106ba0 2024-03-19T01:03:22.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.224+0000 7f36aaffd700 1 --2- 172.21.15.116:0/206655900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 0x7f36ac19bc50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f369c003b40 tx=0x7f369c0042a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:22.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.224+0000 7f36b0852700 1 -- 172.21.15.116:0/206655900 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f369c003d00 con 0x7f36ac106ba0 2024-03-19T01:03:22.741 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.225+0000 7f36b0852700 1 -- 172.21.15.116:0/206655900 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f369c00be70 con 0x7f36ac106ba0 2024-03-19T01:03:22.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.225+0000 7f36b1854700 1 -- 172.21.15.116:0/206655900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36ac19c420 con 0x7f36ac106ba0 2024-03-19T01:03:22.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.225+0000 7f36b0852700 1 -- 172.21.15.116:0/206655900 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f369c011640 con 0x7f36ac106ba0 2024-03-19T01:03:22.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.225+0000 7f36b1854700 1 -- 172.21.15.116:0/206655900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36ac19c840 con 0x7f36ac106ba0 2024-03-19T01:03:22.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.226+0000 7f36b0852700 1 -- 172.21.15.116:0/206655900 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f369c022020 con 0x7f36ac106ba0 2024-03-19T01:03:22.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.226+0000 7f36b1854700 1 -- 172.21.15.116:0/206655900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36ac063940 con 0x7f36ac106ba0 2024-03-19T01:03:22.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.226+0000 7f36b0852700 1 --2- 172.21.15.116:0/206655900 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f3694038fd0 0x7f369403b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:22.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.227+0000 7f36aa7fc700 1 -- 172.21.15.116:0/206655900 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f3694038fd0 msgr2=0x7f369403b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1107242393 2024-03-19T01:03:22.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.227+0000 7f36aa7fc700 1 --2- 172.21.15.116:0/206655900 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f3694038fd0 0x7f369403b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:03:22.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.227+0000 7f36b0852700 1 -- 172.21.15.116:0/206655900 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f369c04cdf0 con 0x7f36ac106ba0 2024-03-19T01:03:22.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.234+0000 7f36b0852700 1 -- 172.21.15.116:0/206655900 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f369c024410 con 0x7f36ac106ba0 2024-03-19T01:03:22.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.428+0000 7f36aa7fc700 1 -- 172.21.15.116:0/206655900 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f3694038fd0 msgr2=0x7f369403b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1107242393 2024-03-19T01:03:22.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.428+0000 7f36aa7fc700 1 --2- 172.21.15.116:0/206655900 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f3694038fd0 0x7f369403b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:03:22.742 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.497+0000 7f36b1854700 1 -- 172.21.15.116:0/206655900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f36ac0008d0 con 0x7f36ac106ba0 2024-03-19T01:03:22.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.497+0000 7f36b0852700 1 -- 172.21.15.116:0/206655900 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f369c02cc20 con 0x7f36ac106ba0 2024-03-19T01:03:22.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.499+0000 7f36b1854700 1 -- 172.21.15.116:0/206655900 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f3694038fd0 msgr2=0x7f369403b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:03:22.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.499+0000 7f36b1854700 1 --2- 172.21.15.116:0/206655900 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f3694038fd0 0x7f369403b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:22.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.499+0000 7f36b1854700 1 -- 172.21.15.116:0/206655900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 msgr2=0x7f36ac19bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:22.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.499+0000 7f36b1854700 1 --2- 172.21.15.116:0/206655900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 0x7f36ac19bc50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f369c003b40 tx=0x7f369c0042a0 comp rx=0 tx=0).stop 2024-03-19T01:03:22.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.499+0000 7f36b1854700 1 -- 172.21.15.116:0/206655900 shutdown_connections 2024-03-19T01:03:22.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.499+0000 7f36b1854700 1 --2- 172.21.15.116:0/206655900 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f3694038fd0 0x7f369403b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:22.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.499+0000 7f36b1854700 1 --2- 172.21.15.116:0/206655900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36ac106ba0 0x7f36ac19bc50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:22.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.499+0000 7f36b1854700 1 -- 172.21.15.116:0/206655900 >> 172.21.15.116:0/206655900 conn(0x7f36ac102140 msgr2=0x7f36ac102950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:22.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.500+0000 7f36b1854700 1 -- 172.21.15.116:0/206655900 shutdown_connections 2024-03-19T01:03:22.743 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:21.500+0000 7f36b1854700 1 -- 172.21.15.116:0/206655900 wait complete. 2024-03-19T01:03:22.743 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for the mgr to restart... 2024-03-19T01:03:22.743 INFO:teuthology.orchestra.run.smithi116.stdout:Waiting for mgr epoch 9... 2024-03-19T01:03:23.715 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:23 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:23.585+0000 7f5d199d4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T01:03:24.044 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:23 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:23.714+0000 7f5d199d4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T01:03:24.045 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:24 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:24.044+0000 7f5d199d4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T01:03:24.328 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:24 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:24.132+0000 7f5d199d4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T01:03:24.328 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:24 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:24.327+0000 7f5d199d4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:03:24.611 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:24 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:24.512+0000 7f5d199d4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T01:03:24.905 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:24 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:24.797+0000 7f5d199d4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:03:25.405 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:25 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:25.052+0000 7f5d199d4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T01:03:25.405 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:25 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:25.285+0000 7f5d199d4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T01:03:26.405 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:26 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:26.075+0000 7f5d199d4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T01:03:26.905 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:26 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:26.646+0000 7f5d199d4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T01:03:26.905 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:26 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:26.744+0000 7f5d199d4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T01:03:27.405 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:03:26 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:03:26.960+0000 7f5d199d4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T01:03:27.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:27 smithi116 ceph-mon[62701]: Active manager daemon x restarted 2024-03-19T01:03:27.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:27 smithi116 ceph-mon[62701]: Activating manager daemon x 2024-03-19T01:03:27.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:27 smithi116 ceph-mon[62701]: osdmap e3: 0 total, 0 up, 0 in 2024-03-19T01:03:27.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:27 smithi116 ceph-mon[62701]: mgrmap e10: x(active, starting, since 0.00331458s) 2024-03-19T01:03:27.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:03:27.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-19T01:03:27.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-19T01:03:27.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-19T01:03:27.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-19T01:03:27.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:27 smithi116 ceph-mon[62701]: Manager daemon x is now available 2024-03-19T01:03:27.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:03:27.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:03:28.271 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-19T01:03:28.271 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-19T01:03:28.271 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:28 smithi116 ceph-mon[62701]: mgrmap e11: x(active, since 1.00568s) 2024-03-19T01:03:28.523 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout { 2024-03-19T01:03:28.523 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-19T01:03:28.523 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-19T01:03:28.523 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout } 2024-03-19T01:03:28.523 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.203+0000 7f9524ccc700 1 Processor -- start 2024-03-19T01:03:28.523 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.203+0000 7f9524ccc700 1 -- start start 2024-03-19T01:03:28.523 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.204+0000 7f9524ccc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 0x7f95201080a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:28.524 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.204+0000 7f9524ccc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9520108670 con 0x7f9520107c80 2024-03-19T01:03:28.524 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.204+0000 7f951e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 0x7f95201080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:28.524 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.204+0000 7f951e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 0x7f95201080a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44036/0 (socket says 172.21.15.116:44036) 2024-03-19T01:03:28.524 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.204+0000 7f951e59c700 1 -- 172.21.15.116:0/1797039481 learned_addr learned my addr 172.21.15.116:0/1797039481 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:28.524 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.204+0000 7f951e59c700 1 -- 172.21.15.116:0/1797039481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9520108e80 con 0x7f9520107c80 2024-03-19T01:03:28.524 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.204+0000 7f951e59c700 1 --2- 172.21.15.116:0/1797039481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 0x7f95201080a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f951000f960 tx=0x7f951000fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf7a69e2fda5831 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:28.524 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.204+0000 7f951d59a700 1 -- 172.21.15.116:0/1797039481 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f951001c070 con 0x7f9520107c80 2024-03-19T01:03:28.524 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.205+0000 7f951d59a700 1 -- 172.21.15.116:0/1797039481 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9510004510 con 0x7f9520107c80 2024-03-19T01:03:28.524 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.205+0000 7f951d59a700 1 -- 172.21.15.116:0/1797039481 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9510003d10 con 0x7f9520107c80 2024-03-19T01:03:28.524 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.205+0000 7f9524ccc700 1 -- 172.21.15.116:0/1797039481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 msgr2=0x7f95201080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:28.524 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.205+0000 7f9524ccc700 1 --2- 172.21.15.116:0/1797039481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 0x7f95201080a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f951000f960 tx=0x7f951000fc70 comp rx=0 tx=0).stop 2024-03-19T01:03:28.524 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.205+0000 7f9524ccc700 1 -- 172.21.15.116:0/1797039481 shutdown_connections 2024-03-19T01:03:28.524 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.205+0000 7f9524ccc700 1 --2- 172.21.15.116:0/1797039481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 0x7f95201080a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:28.525 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.205+0000 7f9524ccc700 1 -- 172.21.15.116:0/1797039481 >> 172.21.15.116:0/1797039481 conn(0x7f9520103200 msgr2=0x7f9520105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:28.525 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.205+0000 7f9524ccc700 1 -- 172.21.15.116:0/1797039481 shutdown_connections 2024-03-19T01:03:28.525 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.205+0000 7f9524ccc700 1 -- 172.21.15.116:0/1797039481 wait complete. 2024-03-19T01:03:28.525 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.206+0000 7f9524ccc700 1 Processor -- start 2024-03-19T01:03:28.525 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.206+0000 7f9524ccc700 1 -- start start 2024-03-19T01:03:28.525 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.206+0000 7f9524ccc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 0x7f9520199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:28.525 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.206+0000 7f9524ccc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f952007c060 con 0x7f9520107c80 2024-03-19T01:03:28.525 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.206+0000 7f951e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 0x7f9520199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:28.525 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.206+0000 7f951e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 0x7f9520199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44038/0 (socket says 172.21.15.116:44038) 2024-03-19T01:03:28.525 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.206+0000 7f951e59c700 1 -- 172.21.15.116:0/1131701663 learned_addr learned my addr 172.21.15.116:0/1131701663 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:28.525 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.207+0000 7f951e59c700 1 -- 172.21.15.116:0/1131701663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9520199f80 con 0x7f9520107c80 2024-03-19T01:03:28.525 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.207+0000 7f951e59c700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 0x7f9520199a40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9510004120 tx=0x7f9510004200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:28.525 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.207+0000 7f950f7fe700 1 -- 172.21.15.116:0/1131701663 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9510025070 con 0x7f9520107c80 2024-03-19T01:03:28.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.207+0000 7f9524ccc700 1 -- 172.21.15.116:0/1131701663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f952019a210 con 0x7f9520107c80 2024-03-19T01:03:28.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.207+0000 7f9524ccc700 1 -- 172.21.15.116:0/1131701663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f952019cf00 con 0x7f9520107c80 2024-03-19T01:03:28.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.207+0000 7f950f7fe700 1 -- 172.21.15.116:0/1131701663 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95100179d0 con 0x7f9520107c80 2024-03-19T01:03:28.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.207+0000 7f950f7fe700 1 -- 172.21.15.116:0/1131701663 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f951001c040 con 0x7f9520107c80 2024-03-19T01:03:28.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.208+0000 7f950f7fe700 1 -- 172.21.15.116:0/1131701663 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f9510028070 con 0x7f9520107c80 2024-03-19T01:03:28.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.208+0000 7f950f7fe700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 0x7f950803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:28.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.208+0000 7f950f7fe700 1 -- 172.21.15.116:0/1131701663 --> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f950803bba0 con 0x7f9508038fd0 2024-03-19T01:03:28.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.208+0000 7f950f7fe700 1 -- 172.21.15.116:0/1131701663 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9510014070 con 0x7f9520107c80 2024-03-19T01:03:28.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.208+0000 7f951dd9b700 1 -- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 msgr2=0x7f950803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1107242393 2024-03-19T01:03:28.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.208+0000 7f951dd9b700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 0x7f950803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:03:28.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.409+0000 7f951dd9b700 1 -- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 msgr2=0x7f950803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1107242393 2024-03-19T01:03:28.526 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.409+0000 7f951dd9b700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 0x7f950803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-19T01:03:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.810+0000 7f951dd9b700 1 -- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 msgr2=0x7f950803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1107242393 2024-03-19T01:03:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:23.810+0000 7f951dd9b700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 0x7f950803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-19T01:03:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:24.611+0000 7f951dd9b700 1 -- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 msgr2=0x7f950803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1107242393 2024-03-19T01:03:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:24.611+0000 7f951dd9b700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 0x7f950803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-19T01:03:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:26.212+0000 7f951dd9b700 1 -- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 msgr2=0x7f950803b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.116:6800/1107242393 2024-03-19T01:03:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:26.212+0000 7f951dd9b700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 0x7f950803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-19T01:03:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:26.976+0000 7f950f7fe700 1 -- 172.21.15.116:0/1131701663 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7f951002ace0 con 0x7f9520107c80 2024-03-19T01:03:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:26.976+0000 7f950f7fe700 1 -- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 msgr2=0x7f950803b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:03:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:26.976+0000 7f950f7fe700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 0x7f950803b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.977+0000 7f950f7fe700 1 -- 172.21.15.116:0/1131701663 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f9510051930 con 0x7f9520107c80 2024-03-19T01:03:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.978+0000 7f950f7fe700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f950803ca80 0x7f950803ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.978+0000 7f950f7fe700 1 -- 172.21.15.116:0/1131701663 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9510050630 con 0x7f950803ca80 2024-03-19T01:03:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.978+0000 7f951dd9b700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f950803ca80 0x7f950803ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.979+0000 7f951dd9b700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f950803ca80 0x7f950803ee70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9514003a10 tx=0x7f9514009270 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.979+0000 7f950f7fe700 1 -- 172.21.15.116:0/1131701663 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f9510050630 con 0x7f950803ca80 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.985+0000 7f9524ccc700 1 -- 172.21.15.116:0/1131701663 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9520063940 con 0x7f950803ca80 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.986+0000 7f950f7fe700 1 -- 172.21.15.116:0/1131701663 <== mgr.14150 v2:172.21.15.116:6800/2389438183 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f9520063940 con 0x7f950803ca80 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.986+0000 7f9524ccc700 1 -- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f950803ca80 msgr2=0x7f950803ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.986+0000 7f9524ccc700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f950803ca80 0x7f950803ee70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9514003a10 tx=0x7f9514009270 comp rx=0 tx=0).stop 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.986+0000 7f9524ccc700 1 -- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 msgr2=0x7f9520199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.986+0000 7f9524ccc700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 0x7f9520199a40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9510004120 tx=0x7f9510004200 comp rx=0 tx=0).stop 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.986+0000 7f9524ccc700 1 -- 172.21.15.116:0/1131701663 shutdown_connections 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.986+0000 7f9524ccc700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f950803ca80 0x7f950803ee70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.986+0000 7f9524ccc700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9520107c80 0x7f9520199a40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.986+0000 7f9524ccc700 1 --2- 172.21.15.116:0/1131701663 >> [v2:172.21.15.116:6800/1107242393,v1:172.21.15.116:6801/1107242393] conn(0x7f9508038fd0 0x7f950803b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.986+0000 7f9524ccc700 1 -- 172.21.15.116:0/1131701663 >> 172.21.15.116:0/1131701663 conn(0x7f9520103200 msgr2=0x7f9520104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:28.528 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.986+0000 7f9524ccc700 1 -- 172.21.15.116:0/1131701663 shutdown_connections 2024-03-19T01:03:28.529 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:27.986+0000 7f9524ccc700 1 -- 172.21.15.116:0/1131701663 wait complete. 2024-03-19T01:03:28.529 INFO:teuthology.orchestra.run.smithi116.stdout:mgr epoch 9 is available 2024-03-19T01:03:28.529 INFO:teuthology.orchestra.run.smithi116.stdout:Generating a dashboard self-signed certificate... 2024-03-19T01:03:29.296 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:29 smithi116 ceph-mon[62701]: [19/Mar/2024:01:03:27] ENGINE Bus STARTING 2024-03-19T01:03:29.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:29 smithi116 ceph-mon[62701]: [19/Mar/2024:01:03:27] ENGINE Serving on http://172.21.15.116:8765 2024-03-19T01:03:29.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:29 smithi116 ceph-mon[62701]: [19/Mar/2024:01:03:27] ENGINE Serving on https://172.21.15.116:7150 2024-03-19T01:03:29.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:29 smithi116 ceph-mon[62701]: [19/Mar/2024:01:03:27] ENGINE Bus STARTED 2024-03-19T01:03:29.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:29 smithi116 ceph-mon[62701]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-19T01:03:29.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:29 smithi116 ceph-mon[62701]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-19T01:03:29.852 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-19T01:03:29.852 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.940+0000 7fcfc076a700 1 Processor -- start 2024-03-19T01:03:29.852 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.940+0000 7fcfc076a700 1 -- start start 2024-03-19T01:03:29.852 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.940+0000 7fcfc076a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 0x7fcfb807c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:29.852 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.940+0000 7fcfc076a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb807c870 con 0x7fcfb807de40 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.940+0000 7fcfbe506700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 0x7fcfb807c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.940+0000 7fcfbe506700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 0x7fcfb807c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44228/0 (socket says 172.21.15.116:44228) 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.940+0000 7fcfbe506700 1 -- 172.21.15.116:0/3145020716 learned_addr learned my addr 172.21.15.116:0/3145020716 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.941+0000 7fcfbe506700 1 -- 172.21.15.116:0/3145020716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb807c9f0 con 0x7fcfb807de40 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.941+0000 7fcfbe506700 1 --2- 172.21.15.116:0/3145020716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 0x7fcfb807c2a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40098a0 tx=0x7fcfb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be29a281c6344fd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.941+0000 7fcfbd504700 1 -- 172.21.15.116:0/3145020716 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfb4004160 con 0x7fcfb807de40 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.941+0000 7fcfbd504700 1 -- 172.21.15.116:0/3145020716 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb4004300 con 0x7fcfb807de40 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.941+0000 7fcfbd504700 1 -- 172.21.15.116:0/3145020716 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfb4011b20 con 0x7fcfb807de40 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.941+0000 7fcfc076a700 1 -- 172.21.15.116:0/3145020716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 msgr2=0x7fcfb807c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.942+0000 7fcfc076a700 1 --2- 172.21.15.116:0/3145020716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 0x7fcfb807c2a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcfb40098a0 tx=0x7fcfb4009bb0 comp rx=0 tx=0).stop 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.942+0000 7fcfc076a700 1 -- 172.21.15.116:0/3145020716 shutdown_connections 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.942+0000 7fcfc076a700 1 --2- 172.21.15.116:0/3145020716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 0x7fcfb807c2a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.942+0000 7fcfc076a700 1 -- 172.21.15.116:0/3145020716 >> 172.21.15.116:0/3145020716 conn(0x7fcfb8103210 msgr2=0x7fcfb8105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.942+0000 7fcfc076a700 1 -- 172.21.15.116:0/3145020716 shutdown_connections 2024-03-19T01:03:29.853 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.942+0000 7fcfc076a700 1 -- 172.21.15.116:0/3145020716 wait complete. 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.942+0000 7fcfc076a700 1 Processor -- start 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.942+0000 7fcfc076a700 1 -- start start 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.942+0000 7fcfc076a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 0x7fcfb819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.942+0000 7fcfc076a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb810e290 con 0x7fcfb807de40 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.943+0000 7fcfbe506700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 0x7fcfb819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.943+0000 7fcfbe506700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 0x7fcfb819ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44236/0 (socket says 172.21.15.116:44236) 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.943+0000 7fcfbe506700 1 -- 172.21.15.116:0/4106039094 learned_addr learned my addr 172.21.15.116:0/4106039094 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.943+0000 7fcfbe506700 1 -- 172.21.15.116:0/4106039094 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb819e2f0 con 0x7fcfb807de40 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.943+0000 7fcfbe506700 1 --2- 172.21.15.116:0/4106039094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 0x7fcfb819ddb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4000c00 tx=0x7fcfb4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.943+0000 7fcfaf7fe700 1 -- 172.21.15.116:0/4106039094 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfb4003b60 con 0x7fcfb807de40 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.943+0000 7fcfaf7fe700 1 -- 172.21.15.116:0/4106039094 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfb401a910 con 0x7fcfb807de40 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.943+0000 7fcfaf7fe700 1 -- 172.21.15.116:0/4106039094 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcfb4024930 con 0x7fcfb807de40 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.943+0000 7fcfc076a700 1 -- 172.21.15.116:0/4106039094 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb819e580 con 0x7fcfb807de40 2024-03-19T01:03:29.854 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.943+0000 7fcfc076a700 1 -- 172.21.15.116:0/4106039094 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb819e9e0 con 0x7fcfb807de40 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.944+0000 7fcfaf7fe700 1 -- 172.21.15.116:0/4106039094 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7fcfb4022020 con 0x7fcfb807de40 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.944+0000 7fcfaf7fe700 1 --2- 172.21.15.116:0/4106039094 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fcfa4038ec0 0x7fcfa403b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.944+0000 7fcfc076a700 1 -- 172.21.15.116:0/4106039094 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfb8063940 con 0x7fcfb807de40 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.944+0000 7fcfaf7fe700 1 -- 172.21.15.116:0/4106039094 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fcfb404cf10 con 0x7fcfb807de40 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.944+0000 7fcfbdd05700 1 --2- 172.21.15.116:0/4106039094 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fcfa4038ec0 0x7fcfa403b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.945+0000 7fcfbdd05700 1 --2- 172.21.15.116:0/4106039094 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fcfa4038ec0 0x7fcfa403b380 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8006fd0 tx=0x7fcfa8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:28.949+0000 7fcfaf7fe700 1 -- 172.21.15.116:0/4106039094 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcfb4016260 con 0x7fcfb807de40 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:29.031+0000 7fcfaf7fe700 1 -- 172.21.15.116:0/4106039094 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fcfb402d3f0 con 0x7fcfb807de40 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:29.153+0000 7fcfc076a700 1 -- 172.21.15.116:0/4106039094 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fcfb81044b0 con 0x7fcfa4038ec0 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:29.296+0000 7fcfaf7fe700 1 -- 172.21.15.116:0/4106039094 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fcfb81044b0 con 0x7fcfa4038ec0 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:29.299+0000 7fcfc076a700 1 -- 172.21.15.116:0/4106039094 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fcfa4038ec0 msgr2=0x7fcfa403b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:29.299+0000 7fcfc076a700 1 --2- 172.21.15.116:0/4106039094 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fcfa4038ec0 0x7fcfa403b380 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8006fd0 tx=0x7fcfa8006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:29.299+0000 7fcfc076a700 1 -- 172.21.15.116:0/4106039094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 msgr2=0x7fcfb819ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:29.299+0000 7fcfc076a700 1 --2- 172.21.15.116:0/4106039094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 0x7fcfb819ddb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcfb4000c00 tx=0x7fcfb4011c10 comp rx=0 tx=0).stop 2024-03-19T01:03:29.855 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:29.300+0000 7fcfc076a700 1 -- 172.21.15.116:0/4106039094 shutdown_connections 2024-03-19T01:03:29.856 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:29.300+0000 7fcfc076a700 1 --2- 172.21.15.116:0/4106039094 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fcfa4038ec0 0x7fcfa403b380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:29.856 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:29.300+0000 7fcfc076a700 1 --2- 172.21.15.116:0/4106039094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfb807de40 0x7fcfb819ddb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:29.856 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:29.300+0000 7fcfc076a700 1 -- 172.21.15.116:0/4106039094 >> 172.21.15.116:0/4106039094 conn(0x7fcfb8103210 msgr2=0x7fcfb8103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:29.856 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:29.300+0000 7fcfc076a700 1 -- 172.21.15.116:0/4106039094 shutdown_connections 2024-03-19T01:03:29.856 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:29.300+0000 7fcfc076a700 1 -- 172.21.15.116:0/4106039094 wait complete. 2024-03-19T01:03:29.856 INFO:teuthology.orchestra.run.smithi116.stdout:Creating initial admin user... 2024-03-19T01:03:30.105 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:30 smithi116 ceph-mon[62701]: mgrmap e12: x(active, since 2s) 2024-03-19T01:03:30.105 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:30.106 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:30.106 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:31.276 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$jFUr7gytpnr0I5vn3QuFFuAEeD8ZRoA.vrrz9S3gHF0BfGjSdccmG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710810210, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.320+0000 7f28ff367700 1 Processor -- start 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.320+0000 7f28ff367700 1 -- start start 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.320+0000 7f28ff367700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 0x7f28f81069a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.320+0000 7f28ff367700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f28f807bdb0 con 0x7f28f8106580 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.321+0000 7f28fd103700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 0x7f28f81069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.321+0000 7f28fd103700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 0x7f28f81069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44244/0 (socket says 172.21.15.116:44244) 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.321+0000 7f28fd103700 1 -- 172.21.15.116:0/1147798079 learned_addr learned my addr 172.21.15.116:0/1147798079 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.321+0000 7f28fd103700 1 -- 172.21.15.116:0/1147798079 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f8106ee0 con 0x7f28f8106580 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.321+0000 7f28fd103700 1 --2- 172.21.15.116:0/1147798079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 0x7f28f81069a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f28e80098a0 tx=0x7f28e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a60238411ff40d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.321+0000 7f28effff700 1 -- 172.21.15.116:0/1147798079 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28e8004160 con 0x7f28f8106580 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.321+0000 7f28effff700 1 -- 172.21.15.116:0/1147798079 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28e8004300 con 0x7f28f8106580 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.321+0000 7f28effff700 1 -- 172.21.15.116:0/1147798079 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28e8011b20 con 0x7f28f8106580 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.322+0000 7f28ff367700 1 -- 172.21.15.116:0/1147798079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 msgr2=0x7f28f81069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.322+0000 7f28ff367700 1 --2- 172.21.15.116:0/1147798079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 0x7f28f81069a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f28e80098a0 tx=0x7f28e8009bb0 comp rx=0 tx=0).stop 2024-03-19T01:03:31.277 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.322+0000 7f28ff367700 1 -- 172.21.15.116:0/1147798079 shutdown_connections 2024-03-19T01:03:31.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.322+0000 7f28ff367700 1 --2- 172.21.15.116:0/1147798079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 0x7f28f81069a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:31.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.322+0000 7f28ff367700 1 -- 172.21.15.116:0/1147798079 >> 172.21.15.116:0/1147798079 conn(0x7f28f8102140 msgr2=0x7f28f8104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:31.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.322+0000 7f28ff367700 1 -- 172.21.15.116:0/1147798079 shutdown_connections 2024-03-19T01:03:31.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.322+0000 7f28ff367700 1 -- 172.21.15.116:0/1147798079 wait complete. 2024-03-19T01:03:31.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.323+0000 7f28ff367700 1 Processor -- start 2024-03-19T01:03:31.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.323+0000 7f28ff367700 1 -- start start 2024-03-19T01:03:31.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.323+0000 7f28ff367700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 0x7f28f819ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:31.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.323+0000 7f28ff367700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f28f8107fb0 con 0x7f28f8106580 2024-03-19T01:03:31.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.323+0000 7f28fd103700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 0x7f28f819ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:31.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.323+0000 7f28fd103700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 0x7f28f819ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44252/0 (socket says 172.21.15.116:44252) 2024-03-19T01:03:31.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.323+0000 7f28fd103700 1 -- 172.21.15.116:0/3740018836 learned_addr learned my addr 172.21.15.116:0/3740018836 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:31.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.323+0000 7f28fd103700 1 -- 172.21.15.116:0/3740018836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f819e310 con 0x7f28f8106580 2024-03-19T01:03:31.278 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.323+0000 7f28fd103700 1 --2- 172.21.15.116:0/3740018836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 0x7f28f819ddd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f28e8011b60 tx=0x7f28e8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.323+0000 7f28ee7fc700 1 -- 172.21.15.116:0/3740018836 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28e8003b60 con 0x7f28f8106580 2024-03-19T01:03:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.323+0000 7f28ee7fc700 1 -- 172.21.15.116:0/3740018836 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28e801a910 con 0x7f28f8106580 2024-03-19T01:03:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.323+0000 7f28ee7fc700 1 -- 172.21.15.116:0/3740018836 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28e8024930 con 0x7f28f8106580 2024-03-19T01:03:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.324+0000 7f28ff367700 1 -- 172.21.15.116:0/3740018836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f819e5a0 con 0x7f28f8106580 2024-03-19T01:03:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.324+0000 7f28ff367700 1 -- 172.21.15.116:0/3740018836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f819ea00 con 0x7f28f8106580 2024-03-19T01:03:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.324+0000 7f28ee7fc700 1 -- 172.21.15.116:0/3740018836 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f28e8022020 con 0x7f28f8106580 2024-03-19T01:03:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.325+0000 7f28ff367700 1 -- 172.21.15.116:0/3740018836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28f8063940 con 0x7f28f8106580 2024-03-19T01:03:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.325+0000 7f28ee7fc700 1 --2- 172.21.15.116:0/3740018836 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f28e4038f70 0x7f28e403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.325+0000 7f28ee7fc700 1 -- 172.21.15.116:0/3740018836 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f28e804d1d0 con 0x7f28f8106580 2024-03-19T01:03:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.325+0000 7f28fc902700 1 --2- 172.21.15.116:0/3740018836 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f28e4038f70 0x7f28e403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.325+0000 7f28fc902700 1 --2- 172.21.15.116:0/3740018836 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f28e4038f70 0x7f28e403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f28f4006fd0 tx=0x7f28f4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.330+0000 7f28ee7fc700 1 -- 172.21.15.116:0/3740018836 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f28e8024c50 con 0x7f28f8106580 2024-03-19T01:03:31.279 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.558+0000 7f28ff367700 1 -- 172.21.15.116:0/3740018836 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f28f8197900 con 0x7f28e4038f70 2024-03-19T01:03:31.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.782+0000 7f28ee7fc700 1 -- 172.21.15.116:0/3740018836 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f28f8197900 con 0x7f28e4038f70 2024-03-19T01:03:31.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.784+0000 7f28ff367700 1 -- 172.21.15.116:0/3740018836 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f28e4038f70 msgr2=0x7f28e403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:31.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.784+0000 7f28ff367700 1 --2- 172.21.15.116:0/3740018836 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f28e4038f70 0x7f28e403b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f28f4006fd0 tx=0x7f28f4006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:31.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.784+0000 7f28ff367700 1 -- 172.21.15.116:0/3740018836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 msgr2=0x7f28f819ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:31.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.784+0000 7f28ff367700 1 --2- 172.21.15.116:0/3740018836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 0x7f28f819ddd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f28e8011b60 tx=0x7f28e8011c40 comp rx=0 tx=0).stop 2024-03-19T01:03:31.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.784+0000 7f28ff367700 1 -- 172.21.15.116:0/3740018836 shutdown_connections 2024-03-19T01:03:31.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.784+0000 7f28ff367700 1 --2- 172.21.15.116:0/3740018836 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f28e4038f70 0x7f28e403b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:31.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.784+0000 7f28ff367700 1 --2- 172.21.15.116:0/3740018836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28f8106580 0x7f28f819ddd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:31.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.784+0000 7f28ff367700 1 -- 172.21.15.116:0/3740018836 >> 172.21.15.116:0/3740018836 conn(0x7f28f8102140 msgr2=0x7f28f8194a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:31.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.784+0000 7f28ff367700 1 -- 172.21.15.116:0/3740018836 shutdown_connections 2024-03-19T01:03:31.280 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:30.784+0000 7f28ff367700 1 -- 172.21.15.116:0/3740018836 wait complete. 2024-03-19T01:03:31.280 INFO:teuthology.orchestra.run.smithi116.stdout:Fetching dashboard port number... 2024-03-19T01:03:31.355 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:31 smithi116 ceph-mon[62701]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:03:31.355 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:31.355 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:32 smithi116 ceph-mon[62701]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:03:32.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:32 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1680264815' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-19T01:03:32.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stdout 8443 2024-03-19T01:03:32.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.745+0000 7fa9290e8700 1 Processor -- start 2024-03-19T01:03:32.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.745+0000 7fa9290e8700 1 -- start start 2024-03-19T01:03:32.642 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.745+0000 7fa9290e8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 0x7fa92407c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.745+0000 7fa9290e8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa92407c850 con 0x7fa92407de20 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.745+0000 7fa922d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 0x7fa92407c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.746+0000 7fa922d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 0x7fa92407c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44264/0 (socket says 172.21.15.116:44264) 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.746+0000 7fa922d9d700 1 -- 172.21.15.116:0/828448423 learned_addr learned my addr 172.21.15.116:0/828448423 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.746+0000 7fa922d9d700 1 -- 172.21.15.116:0/828448423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa92407c9d0 con 0x7fa92407de20 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.746+0000 7fa922d9d700 1 --2- 172.21.15.116:0/828448423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 0x7fa92407c280 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa9140098a0 tx=0x7fa914009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b188f17267f3ec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.746+0000 7fa921d9b700 1 -- 172.21.15.116:0/828448423 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa914004160 con 0x7fa92407de20 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.746+0000 7fa921d9b700 1 -- 172.21.15.116:0/828448423 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa914004300 con 0x7fa92407de20 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.746+0000 7fa921d9b700 1 -- 172.21.15.116:0/828448423 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa914011b20 con 0x7fa92407de20 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.747+0000 7fa9290e8700 1 -- 172.21.15.116:0/828448423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 msgr2=0x7fa92407c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.747+0000 7fa9290e8700 1 --2- 172.21.15.116:0/828448423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 0x7fa92407c280 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa9140098a0 tx=0x7fa914009bb0 comp rx=0 tx=0).stop 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.747+0000 7fa9290e8700 1 -- 172.21.15.116:0/828448423 shutdown_connections 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.747+0000 7fa9290e8700 1 --2- 172.21.15.116:0/828448423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 0x7fa92407c280 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:32.643 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.747+0000 7fa9290e8700 1 -- 172.21.15.116:0/828448423 >> 172.21.15.116:0/828448423 conn(0x7fa9241031b0 msgr2=0x7fa924105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.747+0000 7fa9290e8700 1 -- 172.21.15.116:0/828448423 shutdown_connections 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.747+0000 7fa9290e8700 1 -- 172.21.15.116:0/828448423 wait complete. 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.747+0000 7fa9290e8700 1 Processor -- start 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.748+0000 7fa9290e8700 1 -- start start 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.748+0000 7fa9290e8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 0x7fa92419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.748+0000 7fa9290e8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa92410e290 con 0x7fa92407de20 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.748+0000 7fa922d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 0x7fa92419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.748+0000 7fa922d9d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 0x7fa92419dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44272/0 (socket says 172.21.15.116:44272) 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.748+0000 7fa922d9d700 1 -- 172.21.15.116:0/1680264815 learned_addr learned my addr 172.21.15.116:0/1680264815 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.748+0000 7fa922d9d700 1 -- 172.21.15.116:0/1680264815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa92419e320 con 0x7fa92407de20 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.748+0000 7fa922d9d700 1 --2- 172.21.15.116:0/1680264815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 0x7fa92419dde0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa9140116a0 tx=0x7fa914011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.748+0000 7fa90bfff700 1 -- 172.21.15.116:0/1680264815 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa914003a40 con 0x7fa92407de20 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.748+0000 7fa90bfff700 1 -- 172.21.15.116:0/1680264815 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa914003be0 con 0x7fa92407de20 2024-03-19T01:03:32.644 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.748+0000 7fa90bfff700 1 -- 172.21.15.116:0/1680264815 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa91401a720 con 0x7fa92407de20 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.748+0000 7fa9290e8700 1 -- 172.21.15.116:0/1680264815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa92419e5b0 con 0x7fa92407de20 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.749+0000 7fa9290e8700 1 -- 172.21.15.116:0/1680264815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa92419ea10 con 0x7fa92407de20 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.749+0000 7fa90bfff700 1 -- 172.21.15.116:0/1680264815 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fa91401aa20 con 0x7fa92407de20 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.749+0000 7fa9290e8700 1 -- 172.21.15.116:0/1680264815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa924063940 con 0x7fa92407de20 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.749+0000 7fa90bfff700 1 --2- 172.21.15.116:0/1680264815 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa90c038f20 0x7fa90c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.750+0000 7fa90bfff700 1 -- 172.21.15.116:0/1680264815 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa91404d750 con 0x7fa92407de20 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.750+0000 7fa92259c700 1 --2- 172.21.15.116:0/1680264815 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa90c038f20 0x7fa90c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.750+0000 7fa92259c700 1 --2- 172.21.15.116:0/1680264815 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa90c038f20 0x7fa90c03b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa918006fd0 tx=0x7fa918006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.754+0000 7fa90bfff700 1 -- 172.21.15.116:0/1680264815 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa914011e20 con 0x7fa92407de20 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.954+0000 7fa9290e8700 1 -- 172.21.15.116:0/1680264815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7fa9241a13e0 con 0x7fa92407de20 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.954+0000 7fa90bfff700 1 -- 172.21.15.116:0/1680264815 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7fa914019c70 con 0x7fa92407de20 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.956+0000 7fa9290e8700 1 -- 172.21.15.116:0/1680264815 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa90c038f20 msgr2=0x7fa90c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.956+0000 7fa9290e8700 1 --2- 172.21.15.116:0/1680264815 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa90c038f20 0x7fa90c03b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa918006fd0 tx=0x7fa918006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:32.645 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.956+0000 7fa9290e8700 1 -- 172.21.15.116:0/1680264815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 msgr2=0x7fa92419dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:32.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.956+0000 7fa9290e8700 1 --2- 172.21.15.116:0/1680264815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 0x7fa92419dde0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa9140116a0 tx=0x7fa914011ba0 comp rx=0 tx=0).stop 2024-03-19T01:03:32.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.956+0000 7fa9290e8700 1 -- 172.21.15.116:0/1680264815 shutdown_connections 2024-03-19T01:03:32.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.956+0000 7fa9290e8700 1 --2- 172.21.15.116:0/1680264815 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa90c038f20 0x7fa90c03b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:32.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.956+0000 7fa9290e8700 1 --2- 172.21.15.116:0/1680264815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa92407de20 0x7fa92419dde0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:32.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.956+0000 7fa9290e8700 1 -- 172.21.15.116:0/1680264815 >> 172.21.15.116:0/1680264815 conn(0x7fa9241031b0 msgr2=0x7fa924103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:32.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.957+0000 7fa9290e8700 1 -- 172.21.15.116:0/1680264815 shutdown_connections 2024-03-19T01:03:32.646 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:31.957+0000 7fa9290e8700 1 -- 172.21.15.116:0/1680264815 wait complete. 2024-03-19T01:03:32.648 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-19T01:03:32.648 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout disabled 2024-03-19T01:03:32.655 INFO:teuthology.orchestra.run.smithi116.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-19T01:03:32.655 INFO:teuthology.orchestra.run.smithi116.stdout:systemctl: stdout inactive 2024-03-19T01:03:32.655 INFO:teuthology.orchestra.run.smithi116.stdout:firewalld.service is not enabled 2024-03-19T01:03:32.655 INFO:teuthology.orchestra.run.smithi116.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-19T01:03:32.657 INFO:teuthology.orchestra.run.smithi116.stdout:Ceph Dashboard is now available at: 2024-03-19T01:03:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:03:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: URL: https://smithi116.front.sepia.ceph.com:8443/ 2024-03-19T01:03:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: User: admin 2024-03-19T01:03:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: Password: 9n7gzpy3k1 2024-03-19T01:03:32.657 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:03:32.658 INFO:teuthology.orchestra.run.smithi116.stdout:Saving cluster configuration to /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config directory 2024-03-19T01:03:32.658 INFO:teuthology.orchestra.run.smithi116.stdout:Enabling autotune for osd_memory_target 2024-03-19T01:03:34.403 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:34 smithi116 ceph-mon[62701]: mgrmap e13: x(active, since 6s) 2024-03-19T01:03:34.411 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.065+0000 7fba4099e700 1 Processor -- start 2024-03-19T01:03:34.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.065+0000 7fba4099e700 1 -- start start 2024-03-19T01:03:34.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.065+0000 7fba4099e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 0x7fba3c10a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:34.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.065+0000 7fba4099e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c07bdd0 con 0x7fba3c108570 2024-03-19T01:03:34.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.065+0000 7fba3b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 0x7fba3c10a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:34.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.065+0000 7fba3b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 0x7fba3c10a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44274/0 (socket says 172.21.15.116:44274) 2024-03-19T01:03:34.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.065+0000 7fba3b7fe700 1 -- 172.21.15.116:0/4170772631 learned_addr learned my addr 172.21.15.116:0/4170772631 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:34.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.065+0000 7fba3b7fe700 1 -- 172.21.15.116:0/4170772631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba3c10aea0 con 0x7fba3c108570 2024-03-19T01:03:34.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.065+0000 7fba3b7fe700 1 --2- 172.21.15.116:0/4170772631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 0x7fba3c10a960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0098a0 tx=0x7fba2c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dcbc73ee3df429d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:34.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.065+0000 7fba3a7fc700 1 -- 172.21.15.116:0/4170772631 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba2c004160 con 0x7fba3c108570 2024-03-19T01:03:34.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.065+0000 7fba3a7fc700 1 -- 172.21.15.116:0/4170772631 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba2c004300 con 0x7fba3c108570 2024-03-19T01:03:34.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.066+0000 7fba3a7fc700 1 -- 172.21.15.116:0/4170772631 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba2c011b20 con 0x7fba3c108570 2024-03-19T01:03:34.412 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.066+0000 7fba4099e700 1 -- 172.21.15.116:0/4170772631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 msgr2=0x7fba3c10a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.066+0000 7fba4099e700 1 --2- 172.21.15.116:0/4170772631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 0x7fba3c10a960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0098a0 tx=0x7fba2c009bb0 comp rx=0 tx=0).stop 2024-03-19T01:03:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.066+0000 7fba4099e700 1 -- 172.21.15.116:0/4170772631 shutdown_connections 2024-03-19T01:03:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.066+0000 7fba4099e700 1 --2- 172.21.15.116:0/4170772631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 0x7fba3c10a960 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.066+0000 7fba4099e700 1 -- 172.21.15.116:0/4170772631 >> 172.21.15.116:0/4170772631 conn(0x7fba3c101fa0 msgr2=0x7fba3c1043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.066+0000 7fba4099e700 1 -- 172.21.15.116:0/4170772631 shutdown_connections 2024-03-19T01:03:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.066+0000 7fba4099e700 1 -- 172.21.15.116:0/4170772631 wait complete. 2024-03-19T01:03:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.067+0000 7fba4099e700 1 Processor -- start 2024-03-19T01:03:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.067+0000 7fba4099e700 1 -- start start 2024-03-19T01:03:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.067+0000 7fba4099e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 0x7fba3c19b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.067+0000 7fba4099e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fba3c10beb0 con 0x7fba3c108570 2024-03-19T01:03:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.067+0000 7fba3b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 0x7fba3c19b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.067+0000 7fba3b7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 0x7fba3c19b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44284/0 (socket says 172.21.15.116:44284) 2024-03-19T01:03:34.413 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.067+0000 7fba3b7fe700 1 -- 172.21.15.116:0/2871879027 learned_addr learned my addr 172.21.15.116:0/2871879027 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:34.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.068+0000 7fba3b7fe700 1 -- 172.21.15.116:0/2871879027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba3c19bf20 con 0x7fba3c108570 2024-03-19T01:03:34.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.068+0000 7fba3b7fe700 1 --2- 172.21.15.116:0/2871879027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 0x7fba3c19b9e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0116a0 tx=0x7fba2c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:34.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.068+0000 7fba38ff9700 1 -- 172.21.15.116:0/2871879027 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba2c003a40 con 0x7fba3c108570 2024-03-19T01:03:34.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.068+0000 7fba38ff9700 1 -- 172.21.15.116:0/2871879027 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba2c003be0 con 0x7fba3c108570 2024-03-19T01:03:34.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.068+0000 7fba4099e700 1 -- 172.21.15.116:0/2871879027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba3c19c1b0 con 0x7fba3c108570 2024-03-19T01:03:34.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.068+0000 7fba38ff9700 1 -- 172.21.15.116:0/2871879027 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fba2c01a750 con 0x7fba3c108570 2024-03-19T01:03:34.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.068+0000 7fba4099e700 1 -- 172.21.15.116:0/2871879027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba3c19c610 con 0x7fba3c108570 2024-03-19T01:03:34.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.069+0000 7fba38ff9700 1 -- 172.21.15.116:0/2871879027 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fba2c01aa00 con 0x7fba3c108570 2024-03-19T01:03:34.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.069+0000 7fba38ff9700 1 --2- 172.21.15.116:0/2871879027 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fba24039020 0x7fba2403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:34.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.069+0000 7fba4099e700 1 -- 172.21.15.116:0/2871879027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba3c063940 con 0x7fba3c108570 2024-03-19T01:03:34.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.069+0000 7fba38ff9700 1 -- 172.21.15.116:0/2871879027 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fba2c04d630 con 0x7fba3c108570 2024-03-19T01:03:34.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.069+0000 7fba3affd700 1 --2- 172.21.15.116:0/2871879027 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fba24039020 0x7fba2403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:34.414 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.069+0000 7fba3affd700 1 --2- 172.21.15.116:0/2871879027 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fba24039020 0x7fba2403b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fba30006fd0 tx=0x7fba30006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:34.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.074+0000 7fba38ff9700 1 -- 172.21.15.116:0/2871879027 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fba2c011e20 con 0x7fba3c108570 2024-03-19T01:03:34.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.273+0000 7fba4099e700 1 -- 172.21.15.116:0/2871879027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fba3c19efe0 con 0x7fba3c108570 2024-03-19T01:03:34.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.273+0000 7fba38ff9700 1 -- 172.21.15.116:0/2871879027 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fba2c032020 con 0x7fba3c108570 2024-03-19T01:03:34.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.275+0000 7fba4099e700 1 -- 172.21.15.116:0/2871879027 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fba24039020 msgr2=0x7fba2403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:34.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.275+0000 7fba4099e700 1 --2- 172.21.15.116:0/2871879027 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fba24039020 0x7fba2403b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fba30006fd0 tx=0x7fba30006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:34.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.275+0000 7fba4099e700 1 -- 172.21.15.116:0/2871879027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 msgr2=0x7fba3c19b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:34.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.275+0000 7fba4099e700 1 --2- 172.21.15.116:0/2871879027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 0x7fba3c19b9e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0116a0 tx=0x7fba2c011ba0 comp rx=0 tx=0).stop 2024-03-19T01:03:34.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.275+0000 7fba4099e700 1 -- 172.21.15.116:0/2871879027 shutdown_connections 2024-03-19T01:03:34.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.275+0000 7fba4099e700 1 --2- 172.21.15.116:0/2871879027 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fba24039020 0x7fba2403b4e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:34.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.275+0000 7fba4099e700 1 --2- 172.21.15.116:0/2871879027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba3c108570 0x7fba3c19b9e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:34.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.275+0000 7fba4099e700 1 -- 172.21.15.116:0/2871879027 >> 172.21.15.116:0/2871879027 conn(0x7fba3c101fa0 msgr2=0x7fba3c102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:34.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.275+0000 7fba4099e700 1 -- 172.21.15.116:0/2871879027 shutdown_connections 2024-03-19T01:03:34.415 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:33.275+0000 7fba4099e700 1 -- 172.21.15.116:0/2871879027 wait complete. 2024-03-19T01:03:35.783 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.872+0000 7f99e4ace700 1 Processor -- start 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.872+0000 7f99e4ace700 1 -- start start 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.873+0000 7f99e4ace700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 0x7f99e007c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.873+0000 7f99e4ace700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99e007c850 con 0x7f99e007de20 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.873+0000 7f99de59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 0x7f99e007c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.873+0000 7f99de59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 0x7f99e007c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44286/0 (socket says 172.21.15.116:44286) 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.873+0000 7f99de59c700 1 -- 172.21.15.116:0/1954815964 learned_addr learned my addr 172.21.15.116:0/1954815964 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.873+0000 7f99de59c700 1 -- 172.21.15.116:0/1954815964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e007c9d0 con 0x7f99e007de20 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.873+0000 7f99de59c700 1 --2- 172.21.15.116:0/1954815964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 0x7f99e007c280 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f99d00098a0 tx=0x7f99d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9826887fb34b87 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.873+0000 7f99dd59a700 1 -- 172.21.15.116:0/1954815964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99d0004160 con 0x7f99e007de20 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.873+0000 7f99dd59a700 1 -- 172.21.15.116:0/1954815964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99d0004300 con 0x7f99e007de20 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.873+0000 7f99dd59a700 1 -- 172.21.15.116:0/1954815964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99d0011b20 con 0x7f99e007de20 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.874+0000 7f99e4ace700 1 -- 172.21.15.116:0/1954815964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 msgr2=0x7f99e007c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.874+0000 7f99e4ace700 1 --2- 172.21.15.116:0/1954815964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 0x7f99e007c280 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f99d00098a0 tx=0x7f99d0009bb0 comp rx=0 tx=0).stop 2024-03-19T01:03:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.874+0000 7f99e4ace700 1 -- 172.21.15.116:0/1954815964 shutdown_connections 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.874+0000 7f99e4ace700 1 --2- 172.21.15.116:0/1954815964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 0x7f99e007c280 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.874+0000 7f99e4ace700 1 -- 172.21.15.116:0/1954815964 >> 172.21.15.116:0/1954815964 conn(0x7f99e0103200 msgr2=0x7f99e0105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.874+0000 7f99e4ace700 1 -- 172.21.15.116:0/1954815964 shutdown_connections 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.874+0000 7f99e4ace700 1 -- 172.21.15.116:0/1954815964 wait complete. 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.875+0000 7f99e4ace700 1 Processor -- start 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.875+0000 7f99e4ace700 1 -- start start 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.875+0000 7f99e4ace700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 0x7f99e019de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.875+0000 7f99e4ace700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99e010e2e0 con 0x7f99e007de20 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.875+0000 7f99de59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 0x7f99e019de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.875+0000 7f99de59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 0x7f99e019de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44290/0 (socket says 172.21.15.116:44290) 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.875+0000 7f99de59c700 1 -- 172.21.15.116:0/991738156 learned_addr learned my addr 172.21.15.116:0/991738156 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.875+0000 7f99de59c700 1 -- 172.21.15.116:0/991738156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99e019e340 con 0x7f99e007de20 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.876+0000 7f99de59c700 1 --2- 172.21.15.116:0/991738156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 0x7f99e019de00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f99d0000c00 tx=0x7f99d0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.876+0000 7f99cf7fe700 1 -- 172.21.15.116:0/991738156 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99d0003b60 con 0x7f99e007de20 2024-03-19T01:03:35.785 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.876+0000 7f99cf7fe700 1 -- 172.21.15.116:0/991738156 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f99d001a910 con 0x7f99e007de20 2024-03-19T01:03:35.786 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.876+0000 7f99e4ace700 1 -- 172.21.15.116:0/991738156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99e019e5d0 con 0x7f99e007de20 2024-03-19T01:03:35.786 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.876+0000 7f99cf7fe700 1 -- 172.21.15.116:0/991738156 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99d0024930 con 0x7f99e007de20 2024-03-19T01:03:35.786 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.876+0000 7f99e4ace700 1 -- 172.21.15.116:0/991738156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99e019ea30 con 0x7f99e007de20 2024-03-19T01:03:35.786 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.877+0000 7f99cf7fe700 1 -- 172.21.15.116:0/991738156 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f99d0022020 con 0x7f99e007de20 2024-03-19T01:03:35.786 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.877+0000 7f99e4ace700 1 -- 172.21.15.116:0/991738156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99e0063940 con 0x7f99e007de20 2024-03-19T01:03:35.786 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.877+0000 7f99cf7fe700 1 --2- 172.21.15.116:0/991738156 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f99c8039010 0x7f99c803b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:35.786 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.877+0000 7f99cf7fe700 1 -- 172.21.15.116:0/991738156 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f99d004d290 con 0x7f99e007de20 2024-03-19T01:03:35.786 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.877+0000 7f99ddd9b700 1 --2- 172.21.15.116:0/991738156 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f99c8039010 0x7f99c803b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:35.786 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.877+0000 7f99ddd9b700 1 --2- 172.21.15.116:0/991738156 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f99c8039010 0x7f99c803b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f99d4006fd0 tx=0x7f99d4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:35.786 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:34.882+0000 7f99cf7fe700 1 -- 172.21.15.116:0/991738156 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f99d000b3e0 con 0x7f99e007de20 2024-03-19T01:03:35.786 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:35.187+0000 7f99e4ace700 1 -- 172.21.15.116:0/991738156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f99e01a14c0 con 0x7f99e007de20 2024-03-19T01:03:35.786 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:35.189+0000 7f99cf7fe700 1 -- 172.21.15.116:0/991738156 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f99d00172b0 con 0x7f99e007de20 2024-03-19T01:03:35.786 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:35.191+0000 7f99e4ace700 1 -- 172.21.15.116:0/991738156 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f99c8039010 msgr2=0x7f99c803b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:35.191+0000 7f99e4ace700 1 --2- 172.21.15.116:0/991738156 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f99c8039010 0x7f99c803b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f99d4006fd0 tx=0x7f99d4006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:35.191+0000 7f99e4ace700 1 -- 172.21.15.116:0/991738156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 msgr2=0x7f99e019de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:35.191+0000 7f99e4ace700 1 --2- 172.21.15.116:0/991738156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 0x7f99e019de00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f99d0000c00 tx=0x7f99d0011c10 comp rx=0 tx=0).stop 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:35.191+0000 7f99e4ace700 1 -- 172.21.15.116:0/991738156 shutdown_connections 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:35.191+0000 7f99e4ace700 1 --2- 172.21.15.116:0/991738156 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f99c8039010 0x7f99c803b4d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:35.191+0000 7f99e4ace700 1 --2- 172.21.15.116:0/991738156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99e007de20 0x7f99e019de00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:35.191+0000 7f99e4ace700 1 -- 172.21.15.116:0/991738156 >> 172.21.15.116:0/991738156 conn(0x7f99e0103200 msgr2=0x7f99e0103da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:35.191+0000 7f99e4ace700 1 -- 172.21.15.116:0/991738156 shutdown_connections 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr 2024-03-19T01:03:35.192+0000 7f99e4ace700 1 -- 172.21.15.116:0/991738156 wait complete. 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:03:35.787 INFO:teuthology.orchestra.run.smithi116.stdout:Or, if you are only running a single cluster on this host: 2024-03-19T01:03:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:03:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-19T01:03:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:03:35.788 INFO:teuthology.orchestra.run.smithi116.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-19T01:03:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:03:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: ceph telemetry on 2024-03-19T01:03:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:03:35.788 INFO:teuthology.orchestra.run.smithi116.stdout:For more information see: 2024-03-19T01:03:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:03:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-19T01:03:35.788 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:03:35.788 INFO:teuthology.orchestra.run.smithi116.stdout:Bootstrap complete. 2024-03-19T01:03:35.828 INFO:tasks.cephadm:Fetching config... 2024-03-19T01:03:35.829 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:03:35.829 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-19T01:03:35.844 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-19T01:03:35.844 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:03:35.844 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-19T01:03:35.900 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-19T01:03:35.900 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:03:35.900 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-19T01:03:35.968 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-19T01:03:35.968 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:03:35.968 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-19T01:03:36.023 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-19T01:03:36.023 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJY3KYz2wnSSqrFonV6RJdXsHgY4a3hB6m7FZxA66KYCPdxecd/RqcnTtrbRUkzm5qD4xDn4kMGqZQbuhIowEYzwAkd6mReX8p2jciUITHe9n141cSKMjKc6QhNU0qpOAk4uq3HTH/8++w4FS9UFULFTfwOkazXwCrY7Tq/dtGp8dx0RefhUpzQTgLrhzlVB2OJkcPtqhqrQe/jUYjmbPVhDSZ3tQzb53+icbF8ixvPQFBqES00FAnTMNV/NYg9/NtzzAbzpZ3yLE1MIC3HwYtLHF/4gNIK6KUhMnwc92HVC3qrrDVBg9r9F7pvgSXAgo4P5d0lYPAdhIplbSzBb0gQpW1ZKvlMPU/bf6p/37z8J7FSOscZBaCoPZugYEj1WW+Z8M1Xf0IM6L6o/OPfHXoMFgoKaL/NLN7pILmhGMujCH2hyr2Nt+96KIo3JY5Fy6rIRpEQwM+P8MxseZMciOvgn1CfvTqRYXAYdt7bKIpjLnd8Jd6VIz5JBoMk5s6zJk= ceph-35912c14-e58c-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T01:03:36.102 INFO:teuthology.orchestra.run.smithi116.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJY3KYz2wnSSqrFonV6RJdXsHgY4a3hB6m7FZxA66KYCPdxecd/RqcnTtrbRUkzm5qD4xDn4kMGqZQbuhIowEYzwAkd6mReX8p2jciUITHe9n141cSKMjKc6QhNU0qpOAk4uq3HTH/8++w4FS9UFULFTfwOkazXwCrY7Tq/dtGp8dx0RefhUpzQTgLrhzlVB2OJkcPtqhqrQe/jUYjmbPVhDSZ3tQzb53+icbF8ixvPQFBqES00FAnTMNV/NYg9/NtzzAbzpZ3yLE1MIC3HwYtLHF/4gNIK6KUhMnwc92HVC3qrrDVBg9r9F7pvgSXAgo4P5d0lYPAdhIplbSzBb0gQpW1ZKvlMPU/bf6p/37z8J7FSOscZBaCoPZugYEj1WW+Z8M1Xf0IM6L6o/OPfHXoMFgoKaL/NLN7pILmhGMujCH2hyr2Nt+96KIo3JY5Fy6rIRpEQwM+P8MxseZMciOvgn1CfvTqRYXAYdt7bKIpjLnd8Jd6VIz5JBoMk5s6zJk= ceph-35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:03:36.116 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJY3KYz2wnSSqrFonV6RJdXsHgY4a3hB6m7FZxA66KYCPdxecd/RqcnTtrbRUkzm5qD4xDn4kMGqZQbuhIowEYzwAkd6mReX8p2jciUITHe9n141cSKMjKc6QhNU0qpOAk4uq3HTH/8++w4FS9UFULFTfwOkazXwCrY7Tq/dtGp8dx0RefhUpzQTgLrhzlVB2OJkcPtqhqrQe/jUYjmbPVhDSZ3tQzb53+icbF8ixvPQFBqES00FAnTMNV/NYg9/NtzzAbzpZ3yLE1MIC3HwYtLHF/4gNIK6KUhMnwc92HVC3qrrDVBg9r9F7pvgSXAgo4P5d0lYPAdhIplbSzBb0gQpW1ZKvlMPU/bf6p/37z8J7FSOscZBaCoPZugYEj1WW+Z8M1Xf0IM6L6o/OPfHXoMFgoKaL/NLN7pILmhGMujCH2hyr2Nt+96KIo3JY5Fy6rIRpEQwM+P8MxseZMciOvgn1CfvTqRYXAYdt7bKIpjLnd8Jd6VIz5JBoMk5s6zJk= ceph-35912c14-e58c-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T01:03:36.161 INFO:teuthology.orchestra.run.smithi156.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJY3KYz2wnSSqrFonV6RJdXsHgY4a3hB6m7FZxA66KYCPdxecd/RqcnTtrbRUkzm5qD4xDn4kMGqZQbuhIowEYzwAkd6mReX8p2jciUITHe9n141cSKMjKc6QhNU0qpOAk4uq3HTH/8++w4FS9UFULFTfwOkazXwCrY7Tq/dtGp8dx0RefhUpzQTgLrhzlVB2OJkcPtqhqrQe/jUYjmbPVhDSZ3tQzb53+icbF8ixvPQFBqES00FAnTMNV/NYg9/NtzzAbzpZ3yLE1MIC3HwYtLHF/4gNIK6KUhMnwc92HVC3qrrDVBg9r9F7pvgSXAgo4P5d0lYPAdhIplbSzBb0gQpW1ZKvlMPU/bf6p/37z8J7FSOscZBaCoPZugYEj1WW+Z8M1Xf0IM6L6o/OPfHXoMFgoKaL/NLN7pILmhGMujCH2hyr2Nt+96KIo3JY5Fy6rIRpEQwM+P8MxseZMciOvgn1CfvTqRYXAYdt7bKIpjLnd8Jd6VIz5JBoMk5s6zJk= ceph-35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:03:36.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJY3KYz2wnSSqrFonV6RJdXsHgY4a3hB6m7FZxA66KYCPdxecd/RqcnTtrbRUkzm5qD4xDn4kMGqZQbuhIowEYzwAkd6mReX8p2jciUITHe9n141cSKMjKc6QhNU0qpOAk4uq3HTH/8++w4FS9UFULFTfwOkazXwCrY7Tq/dtGp8dx0RefhUpzQTgLrhzlVB2OJkcPtqhqrQe/jUYjmbPVhDSZ3tQzb53+icbF8ixvPQFBqES00FAnTMNV/NYg9/NtzzAbzpZ3yLE1MIC3HwYtLHF/4gNIK6KUhMnwc92HVC3qrrDVBg9r9F7pvgSXAgo4P5d0lYPAdhIplbSzBb0gQpW1ZKvlMPU/bf6p/37z8J7FSOscZBaCoPZugYEj1WW+Z8M1Xf0IM6L6o/OPfHXoMFgoKaL/NLN7pILmhGMujCH2hyr2Nt+96KIo3JY5Fy6rIRpEQwM+P8MxseZMciOvgn1CfvTqRYXAYdt7bKIpjLnd8Jd6VIz5JBoMk5s6zJk= ceph-35912c14-e58c-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-19T01:03:36.222 INFO:teuthology.orchestra.run.smithi162.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJY3KYz2wnSSqrFonV6RJdXsHgY4a3hB6m7FZxA66KYCPdxecd/RqcnTtrbRUkzm5qD4xDn4kMGqZQbuhIowEYzwAkd6mReX8p2jciUITHe9n141cSKMjKc6QhNU0qpOAk4uq3HTH/8++w4FS9UFULFTfwOkazXwCrY7Tq/dtGp8dx0RefhUpzQTgLrhzlVB2OJkcPtqhqrQe/jUYjmbPVhDSZ3tQzb53+icbF8ixvPQFBqES00FAnTMNV/NYg9/NtzzAbzpZ3yLE1MIC3HwYtLHF/4gNIK6KUhMnwc92HVC3qrrDVBg9r9F7pvgSXAgo4P5d0lYPAdhIplbSzBb0gQpW1ZKvlMPU/bf6p/37z8J7FSOscZBaCoPZugYEj1WW+Z8M1Xf0IM6L6o/OPfHXoMFgoKaL/NLN7pILmhGMujCH2hyr2Nt+96KIo3JY5Fy6rIRpEQwM+P8MxseZMciOvgn1CfvTqRYXAYdt7bKIpjLnd8Jd6VIz5JBoMk5s6zJk= ceph-35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:03:36.237 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-19T01:03:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:36 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/991738156' entity='client.admin' 2024-03-19T01:03:36.477 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:03:38.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.727+0000 7efe8981a700 1 -- 172.21.15.116:0/1991765814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe84100cb0 msgr2=0x7efe841010d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:38.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.727+0000 7efe8981a700 1 --2- 172.21.15.116:0/1991765814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe84100cb0 0x7efe841010d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efe74009910 tx=0x7efe74009c20 comp rx=0 tx=0).stop 2024-03-19T01:03:38.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.727+0000 7efe8981a700 1 -- 172.21.15.116:0/1991765814 shutdown_connections 2024-03-19T01:03:38.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.727+0000 7efe8981a700 1 --2- 172.21.15.116:0/1991765814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe84100cb0 0x7efe841010d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:38.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.727+0000 7efe8981a700 1 -- 172.21.15.116:0/1991765814 >> 172.21.15.116:0/1991765814 conn(0x7efe840fc270 msgr2=0x7efe840fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:38.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.727+0000 7efe8981a700 1 -- 172.21.15.116:0/1991765814 shutdown_connections 2024-03-19T01:03:38.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.727+0000 7efe8981a700 1 -- 172.21.15.116:0/1991765814 wait complete. 2024-03-19T01:03:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.727+0000 7efe8981a700 1 Processor -- start 2024-03-19T01:03:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.728+0000 7efe8981a700 1 -- start start 2024-03-19T01:03:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.728+0000 7efe8981a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe84100cb0 0x7efe8406a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.728+0000 7efe8981a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efe84106cd0 con 0x7efe84100cb0 2024-03-19T01:03:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.728+0000 7efe82ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe84100cb0 0x7efe8406a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.728+0000 7efe82ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe84100cb0 0x7efe8406a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38200/0 (socket says 172.21.15.116:38200) 2024-03-19T01:03:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.728+0000 7efe82ffd700 1 -- 172.21.15.116:0/671870499 learned_addr learned my addr 172.21.15.116:0/671870499 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.728+0000 7efe82ffd700 1 -- 172.21.15.116:0/671870499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe8419a170 con 0x7efe84100cb0 2024-03-19T01:03:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.729+0000 7efe82ffd700 1 --2- 172.21.15.116:0/671870499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe84100cb0 0x7efe8406a1b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe74006010 tx=0x7efe74005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:38.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.729+0000 7efe88818700 1 -- 172.21.15.116:0/671870499 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe74027070 con 0x7efe84100cb0 2024-03-19T01:03:38.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.729+0000 7efe88818700 1 -- 172.21.15.116:0/671870499 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe7400be40 con 0x7efe84100cb0 2024-03-19T01:03:38.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.729+0000 7efe88818700 1 -- 172.21.15.116:0/671870499 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efe7400f460 con 0x7efe84100cb0 2024-03-19T01:03:38.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.729+0000 7efe8981a700 1 -- 172.21.15.116:0/671870499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe8419a340 con 0x7efe84100cb0 2024-03-19T01:03:38.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.729+0000 7efe8981a700 1 -- 172.21.15.116:0/671870499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe8419a760 con 0x7efe84100cb0 2024-03-19T01:03:38.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.730+0000 7efe88818700 1 -- 172.21.15.116:0/671870499 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7efe7402a070 con 0x7efe84100cb0 2024-03-19T01:03:38.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.730+0000 7efe88818700 1 --2- 172.21.15.116:0/671870499 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efe6c038fc0 0x7efe6c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:38.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.730+0000 7efe88818700 1 -- 172.21.15.116:0/671870499 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7efe7401e030 con 0x7efe84100cb0 2024-03-19T01:03:38.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.730+0000 7efe827fc700 1 --2- 172.21.15.116:0/671870499 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efe6c038fc0 0x7efe6c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:38.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.731+0000 7efe8981a700 1 -- 172.21.15.116:0/671870499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe84107450 con 0x7efe84100cb0 2024-03-19T01:03:38.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.731+0000 7efe827fc700 1 --2- 172.21.15.116:0/671870499 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efe6c038fc0 0x7efe6c03b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efe78006fd0 tx=0x7efe78006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:38.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.736+0000 7efe88818700 1 -- 172.21.15.116:0/671870499 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efe74031070 con 0x7efe84100cb0 2024-03-19T01:03:38.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.940+0000 7efe8981a700 1 -- 172.21.15.116:0/671870499 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7efe8402dcc0 con 0x7efe84100cb0 2024-03-19T01:03:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.943+0000 7efe88818700 1 -- 172.21.15.116:0/671870499 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7efe74013070 con 0x7efe84100cb0 2024-03-19T01:03:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.947+0000 7efe8981a700 1 -- 172.21.15.116:0/671870499 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efe6c038fc0 msgr2=0x7efe6c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.947+0000 7efe8981a700 1 --2- 172.21.15.116:0/671870499 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efe6c038fc0 0x7efe6c03b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efe78006fd0 tx=0x7efe78006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.947+0000 7efe8981a700 1 -- 172.21.15.116:0/671870499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe84100cb0 msgr2=0x7efe8406a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.947+0000 7efe8981a700 1 --2- 172.21.15.116:0/671870499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe84100cb0 0x7efe8406a1b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe74006010 tx=0x7efe74005050 comp rx=0 tx=0).stop 2024-03-19T01:03:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.947+0000 7efe8981a700 1 -- 172.21.15.116:0/671870499 shutdown_connections 2024-03-19T01:03:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.947+0000 7efe8981a700 1 --2- 172.21.15.116:0/671870499 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efe6c038fc0 0x7efe6c03b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.947+0000 7efe8981a700 1 --2- 172.21.15.116:0/671870499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe84100cb0 0x7efe8406a1b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.947+0000 7efe8981a700 1 -- 172.21.15.116:0/671870499 >> 172.21.15.116:0/671870499 conn(0x7efe840fc270 msgr2=0x7efe840fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.947+0000 7efe8981a700 1 -- 172.21.15.116:0/671870499 shutdown_connections 2024-03-19T01:03:38.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:38.947+0000 7efe8981a700 1 -- 172.21.15.116:0/671870499 wait complete. 2024-03-19T01:03:39.784 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-19T01:03:39.784 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-19T01:03:39.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:39.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:39.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi116", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:03:39.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:39.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:03:39.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:39.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:39 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/671870499' entity='client.admin' 2024-03-19T01:03:39.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:03:39.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:40.638 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:03:42.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.209+0000 7f50dcda4700 1 -- 172.21.15.116:0/2305640156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8100c70 msgr2=0x7f50d8101090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:42.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.209+0000 7f50dcda4700 1 --2- 172.21.15.116:0/2305640156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8100c70 0x7f50d8101090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f50c8009910 tx=0x7f50c8009c20 comp rx=0 tx=0).stop 2024-03-19T01:03:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.210+0000 7f50dcda4700 1 -- 172.21.15.116:0/2305640156 shutdown_connections 2024-03-19T01:03:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.210+0000 7f50dcda4700 1 --2- 172.21.15.116:0/2305640156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8100c70 0x7f50d8101090 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.210+0000 7f50dcda4700 1 -- 172.21.15.116:0/2305640156 >> 172.21.15.116:0/2305640156 conn(0x7f50d80fc250 msgr2=0x7f50d80fe670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.210+0000 7f50dcda4700 1 -- 172.21.15.116:0/2305640156 shutdown_connections 2024-03-19T01:03:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.210+0000 7f50dcda4700 1 -- 172.21.15.116:0/2305640156 wait complete. 2024-03-19T01:03:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.210+0000 7f50dcda4700 1 Processor -- start 2024-03-19T01:03:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.210+0000 7f50dcda4700 1 -- start start 2024-03-19T01:03:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.210+0000 7f50dcda4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8100c70 0x7f50d81974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.211+0000 7f50dcda4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f50d8106d00 con 0x7f50d8100c70 2024-03-19T01:03:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.211+0000 7f50d659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8100c70 0x7f50d81974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:42.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.211+0000 7f50d659c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8100c70 0x7f50d81974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38222/0 (socket says 172.21.15.116:38222) 2024-03-19T01:03:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.211+0000 7f50d659c700 1 -- 172.21.15.116:0/824920314 learned_addr learned my addr 172.21.15.116:0/824920314 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.211+0000 7f50d659c700 1 -- 172.21.15.116:0/824920314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d81979f0 con 0x7f50d8100c70 2024-03-19T01:03:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.211+0000 7f50d659c700 1 --2- 172.21.15.116:0/824920314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8100c70 0x7f50d81974b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f50c8009fd0 tx=0x7f50c8005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.211+0000 7f50c77fe700 1 -- 172.21.15.116:0/824920314 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50c8027070 con 0x7f50d8100c70 2024-03-19T01:03:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.211+0000 7f50c77fe700 1 -- 172.21.15.116:0/824920314 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50c800be40 con 0x7f50d8100c70 2024-03-19T01:03:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.211+0000 7f50c77fe700 1 -- 172.21.15.116:0/824920314 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50c800f460 con 0x7f50d8100c70 2024-03-19T01:03:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.211+0000 7f50dcda4700 1 -- 172.21.15.116:0/824920314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d8197c80 con 0x7f50d8100c70 2024-03-19T01:03:42.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.211+0000 7f50dcda4700 1 -- 172.21.15.116:0/824920314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d819a970 con 0x7f50d8100c70 2024-03-19T01:03:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.212+0000 7f50c77fe700 1 -- 172.21.15.116:0/824920314 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f50c802a070 con 0x7f50d8100c70 2024-03-19T01:03:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.212+0000 7f50dcda4700 1 -- 172.21.15.116:0/824920314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d8107480 con 0x7f50d8100c70 2024-03-19T01:03:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.212+0000 7f50c77fe700 1 --2- 172.21.15.116:0/824920314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f50c0038fc0 0x7f50c003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.212+0000 7f50c77fe700 1 -- 172.21.15.116:0/824920314 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f50c801e030 con 0x7f50d8100c70 2024-03-19T01:03:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.212+0000 7f50d5d9b700 1 --2- 172.21.15.116:0/824920314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f50c0038fc0 0x7f50c003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.213+0000 7f50d5d9b700 1 --2- 172.21.15.116:0/824920314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f50c0038fc0 0x7f50c003b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f50cc006fd0 tx=0x7f50cc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.217+0000 7f50c77fe700 1 -- 172.21.15.116:0/824920314 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f50c8013070 con 0x7f50d8100c70 2024-03-19T01:03:42.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.424+0000 7f50dcda4700 1 -- 172.21.15.116:0/824920314 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f50d802dcc0 con 0x7f50c0038fc0 2024-03-19T01:03:42.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.427+0000 7f50c77fe700 1 -- 172.21.15.116:0/824920314 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f50d802dcc0 con 0x7f50c0038fc0 2024-03-19T01:03:42.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.429+0000 7f50dcda4700 1 -- 172.21.15.116:0/824920314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f50c0038fc0 msgr2=0x7f50c003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:42.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.429+0000 7f50dcda4700 1 --2- 172.21.15.116:0/824920314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f50c0038fc0 0x7f50c003b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f50cc006fd0 tx=0x7f50cc006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:42.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.429+0000 7f50dcda4700 1 -- 172.21.15.116:0/824920314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8100c70 msgr2=0x7f50d81974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:42.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.429+0000 7f50dcda4700 1 --2- 172.21.15.116:0/824920314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8100c70 0x7f50d81974b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f50c8009fd0 tx=0x7f50c8005050 comp rx=0 tx=0).stop 2024-03-19T01:03:42.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.429+0000 7f50dcda4700 1 -- 172.21.15.116:0/824920314 shutdown_connections 2024-03-19T01:03:42.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.429+0000 7f50dcda4700 1 --2- 172.21.15.116:0/824920314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f50c0038fc0 0x7f50c003b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:42.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.429+0000 7f50dcda4700 1 --2- 172.21.15.116:0/824920314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8100c70 0x7f50d81974b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:42.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.429+0000 7f50dcda4700 1 -- 172.21.15.116:0/824920314 >> 172.21.15.116:0/824920314 conn(0x7f50d80fc250 msgr2=0x7f50d806a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:42.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.429+0000 7f50dcda4700 1 -- 172.21.15.116:0/824920314 shutdown_connections 2024-03-19T01:03:42.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:42.430+0000 7f50dcda4700 1 -- 172.21.15.116:0/824920314 wait complete. 2024-03-19T01:03:43.136 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi156 2024-03-19T01:03:43.136 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-19T01:03:43.136 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.conf 2024-03-19T01:03:43.155 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-19T01:03:43.155 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T01:03:43.212 INFO:tasks.cephadm:Adding host smithi156 to orchestrator... 2024-03-19T01:03:43.213 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch host add smithi156 2024-03-19T01:03:43.454 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:03:43.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:43 smithi116 ceph-mon[62701]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:03:43.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:43.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:03:43.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:03:43.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:03:43.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:43 smithi116 ceph-mon[62701]: Updating smithi116:/etc/ceph/ceph.conf 2024-03-19T01:03:43.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:43 smithi116 ceph-mon[62701]: Updating smithi116:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:03:43.685 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:43 smithi116 ceph-mon[62701]: Updating smithi116:/etc/ceph/ceph.client.admin.keyring 2024-03-19T01:03:43.686 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:43.686 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:43.686 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:44.610 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:44 smithi116 ceph-mon[62701]: Updating smithi116:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T01:03:44.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.792+0000 7f702191e700 1 -- 172.21.15.116:0/3940093037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c102ed0 msgr2=0x7f701c1032b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:44.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.792+0000 7f702191e700 1 --2- 172.21.15.116:0/3940093037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c102ed0 0x7f701c1032b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f700c009910 tx=0x7f700c009c20 comp rx=0 tx=0).stop 2024-03-19T01:03:44.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.792+0000 7f702191e700 1 -- 172.21.15.116:0/3940093037 shutdown_connections 2024-03-19T01:03:44.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.792+0000 7f702191e700 1 --2- 172.21.15.116:0/3940093037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c102ed0 0x7f701c1032b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:44.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.792+0000 7f702191e700 1 -- 172.21.15.116:0/3940093037 >> 172.21.15.116:0/3940093037 conn(0x7f701c0779b0 msgr2=0x7f701c0fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:44.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.792+0000 7f702191e700 1 -- 172.21.15.116:0/3940093037 shutdown_connections 2024-03-19T01:03:44.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.792+0000 7f702191e700 1 -- 172.21.15.116:0/3940093037 wait complete. 2024-03-19T01:03:44.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.793+0000 7f702191e700 1 Processor -- start 2024-03-19T01:03:44.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.793+0000 7f702191e700 1 -- start start 2024-03-19T01:03:44.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.793+0000 7f702191e700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c102ed0 0x7f701c19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:44.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.793+0000 7f702191e700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f701c10c370 con 0x7f701c102ed0 2024-03-19T01:03:44.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.793+0000 7f701affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c102ed0 0x7f701c19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:44.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.793+0000 7f701affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c102ed0 0x7f701c19d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38242/0 (socket says 172.21.15.116:38242) 2024-03-19T01:03:44.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.793+0000 7f701affd700 1 -- 172.21.15.116:0/1360491257 learned_addr learned my addr 172.21.15.116:0/1360491257 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:44.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.793+0000 7f701affd700 1 -- 172.21.15.116:0/1360491257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701c19db40 con 0x7f701c102ed0 2024-03-19T01:03:44.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.793+0000 7f701affd700 1 --2- 172.21.15.116:0/1360491257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c102ed0 0x7f701c19d600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f700c006010 tx=0x7f700c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:44.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.794+0000 7f702091c700 1 -- 172.21.15.116:0/1360491257 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f700c026070 con 0x7f701c102ed0 2024-03-19T01:03:44.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.794+0000 7f702091c700 1 -- 172.21.15.116:0/1360491257 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f700c00bc30 con 0x7f701c102ed0 2024-03-19T01:03:44.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.794+0000 7f702091c700 1 -- 172.21.15.116:0/1360491257 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f700c0039a0 con 0x7f701c102ed0 2024-03-19T01:03:44.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.794+0000 7f702191e700 1 -- 172.21.15.116:0/1360491257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f701c19ddd0 con 0x7f701c102ed0 2024-03-19T01:03:44.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.794+0000 7f702191e700 1 -- 172.21.15.116:0/1360491257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f701c1976d0 con 0x7f701c102ed0 2024-03-19T01:03:44.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.794+0000 7f702091c700 1 -- 172.21.15.116:0/1360491257 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f700c029070 con 0x7f701c102ed0 2024-03-19T01:03:44.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.795+0000 7f702191e700 1 -- 172.21.15.116:0/1360491257 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f701c069fa0 con 0x7f701c102ed0 2024-03-19T01:03:44.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.795+0000 7f702091c700 1 --2- 172.21.15.116:0/1360491257 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7008039020 0x7f700803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:44.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.795+0000 7f702091c700 1 -- 172.21.15.116:0/1360491257 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f700c01e070 con 0x7f701c102ed0 2024-03-19T01:03:44.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.795+0000 7f70127fc700 1 --2- 172.21.15.116:0/1360491257 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7008039020 0x7f700803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:44.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.795+0000 7f70127fc700 1 --2- 172.21.15.116:0/1360491257 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7008039020 0x7f700803b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7004006fd0 tx=0x7f7004006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:44.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.800+0000 7f702091c700 1 -- 172.21.15.116:0/1360491257 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f700c013070 con 0x7f701c102ed0 2024-03-19T01:03:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:44.998+0000 7f702191e700 1 -- 172.21.15.116:0/1360491257 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}) -- 0x7f701c063970 con 0x7f7008039020 2024-03-19T01:03:45.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:45 smithi116 ceph-mon[62701]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:03:46.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:46 smithi116 ceph-mon[62701]: Deploying cephadm binary to smithi156 2024-03-19T01:03:48.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:47 smithi116 ceph-mon[62701]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:03:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:49.162+0000 7f702091c700 1 -- 172.21.15.116:0/1360491257 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f701c063970 con 0x7f7008039020 2024-03-19T01:03:49.163 INFO:teuthology.orchestra.run.smithi116.stdout:Added host 'smithi156' with addr '172.21.15.156' 2024-03-19T01:03:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:49.164+0000 7f702191e700 1 -- 172.21.15.116:0/1360491257 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7008039020 msgr2=0x7f700803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:49.164+0000 7f702191e700 1 --2- 172.21.15.116:0/1360491257 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7008039020 0x7f700803b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7004006fd0 tx=0x7f7004006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:49.164+0000 7f702191e700 1 -- 172.21.15.116:0/1360491257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c102ed0 msgr2=0x7f701c19d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:49.164+0000 7f702191e700 1 --2- 172.21.15.116:0/1360491257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c102ed0 0x7f701c19d600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f700c006010 tx=0x7f700c005030 comp rx=0 tx=0).stop 2024-03-19T01:03:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:49.164+0000 7f702191e700 1 -- 172.21.15.116:0/1360491257 shutdown_connections 2024-03-19T01:03:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:49.164+0000 7f702191e700 1 --2- 172.21.15.116:0/1360491257 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7008039020 0x7f700803b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:49.164+0000 7f702191e700 1 --2- 172.21.15.116:0/1360491257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f701c102ed0 0x7f701c19d600 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:49.164+0000 7f702191e700 1 -- 172.21.15.116:0/1360491257 >> 172.21.15.116:0/1360491257 conn(0x7f701c0779b0 msgr2=0x7f701c103c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:49.164+0000 7f702191e700 1 -- 172.21.15.116:0/1360491257 shutdown_connections 2024-03-19T01:03:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:49.164+0000 7f702191e700 1 -- 172.21.15.116:0/1360491257 wait complete. 2024-03-19T01:03:49.816 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-19T01:03:50.055 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:03:50.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:49 smithi116 ceph-mon[62701]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:03:50.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:50.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:03:50.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:51.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:50 smithi116 ceph-mon[62701]: Added host smithi156 2024-03-19T01:03:51.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:50 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.714+0000 7f64d3ce4700 1 -- 172.21.15.116:0/3665382499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64cc100e70 msgr2=0x7f64cc101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.715+0000 7f64d3ce4700 1 --2- 172.21.15.116:0/3665382499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64cc100e70 0x7f64cc101250 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f64c8009910 tx=0x7f64c8009c20 comp rx=0 tx=0).stop 2024-03-19T01:03:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.715+0000 7f64d3ce4700 1 -- 172.21.15.116:0/3665382499 shutdown_connections 2024-03-19T01:03:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.715+0000 7f64d3ce4700 1 --2- 172.21.15.116:0/3665382499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64cc100e70 0x7f64cc101250 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.715+0000 7f64d3ce4700 1 -- 172.21.15.116:0/3665382499 >> 172.21.15.116:0/3665382499 conn(0x7f64cc0fc720 msgr2=0x7f64cc0feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.715+0000 7f64d3ce4700 1 -- 172.21.15.116:0/3665382499 shutdown_connections 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.715+0000 7f64d3ce4700 1 -- 172.21.15.116:0/3665382499 wait complete. 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.715+0000 7f64d3ce4700 1 Processor -- start 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.716+0000 7f64d3ce4700 1 -- start start 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.716+0000 7f64d3ce4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64cc100e70 0x7f64cc19d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.716+0000 7f64d3ce4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f64cc10b240 con 0x7f64cc100e70 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.716+0000 7f64d1a80700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64cc100e70 0x7f64cc19d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.716+0000 7f64d1a80700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64cc100e70 0x7f64cc19d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41044/0 (socket says 172.21.15.116:41044) 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.716+0000 7f64d1a80700 1 -- 172.21.15.116:0/2223173648 learned_addr learned my addr 172.21.15.116:0/2223173648 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.716+0000 7f64d1a80700 1 -- 172.21.15.116:0/2223173648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64cc19dab0 con 0x7f64cc100e70 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.716+0000 7f64d1a80700 1 --2- 172.21.15.116:0/2223173648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64cc100e70 0x7f64cc19d570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f64c8005850 tx=0x7f64c8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.716+0000 7f64c2ffd700 1 -- 172.21.15.116:0/2223173648 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64c8026070 con 0x7f64cc100e70 2024-03-19T01:03:51.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.716+0000 7f64c2ffd700 1 -- 172.21.15.116:0/2223173648 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64c800bbd0 con 0x7f64cc100e70 2024-03-19T01:03:51.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.716+0000 7f64d3ce4700 1 -- 172.21.15.116:0/2223173648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64cc19dda0 con 0x7f64cc100e70 2024-03-19T01:03:51.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.716+0000 7f64c2ffd700 1 -- 172.21.15.116:0/2223173648 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f64c8003970 con 0x7f64cc100e70 2024-03-19T01:03:51.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.717+0000 7f64d3ce4700 1 -- 172.21.15.116:0/2223173648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64cc197860 con 0x7f64cc100e70 2024-03-19T01:03:51.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.717+0000 7f64c2ffd700 1 -- 172.21.15.116:0/2223173648 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f64c8029070 con 0x7f64cc100e70 2024-03-19T01:03:51.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.717+0000 7f64d3ce4700 1 -- 172.21.15.116:0/2223173648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64cc102350 con 0x7f64cc100e70 2024-03-19T01:03:51.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.718+0000 7f64c2ffd700 1 --2- 172.21.15.116:0/2223173648 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f64b8039020 0x7f64b803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:51.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.718+0000 7f64c2ffd700 1 -- 172.21.15.116:0/2223173648 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f64c801e070 con 0x7f64cc100e70 2024-03-19T01:03:51.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.718+0000 7f64d127f700 1 --2- 172.21.15.116:0/2223173648 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f64b8039020 0x7f64b803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:51.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.718+0000 7f64d127f700 1 --2- 172.21.15.116:0/2223173648 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f64b8039020 0x7f64b803b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f64bc006fd0 tx=0x7f64bc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:51.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.723+0000 7f64c2ffd700 1 -- 172.21.15.116:0/2223173648 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f64c8013070 con 0x7f64cc100e70 2024-03-19T01:03:51.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.934+0000 7f64d3ce4700 1 -- 172.21.15.116:0/2223173648 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f64cc198b70 con 0x7f64b8039020 2024-03-19T01:03:51.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.935+0000 7f64c2ffd700 1 -- 172.21.15.116:0/2223173648 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7f64cc198b70 con 0x7f64b8039020 2024-03-19T01:03:51.936 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:03:51.936 INFO:teuthology.orchestra.run.smithi116.stdout:[{"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}] 2024-03-19T01:03:51.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.937+0000 7f64d3ce4700 1 -- 172.21.15.116:0/2223173648 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f64b8039020 msgr2=0x7f64b803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:51.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.937+0000 7f64d3ce4700 1 --2- 172.21.15.116:0/2223173648 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f64b8039020 0x7f64b803b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f64bc006fd0 tx=0x7f64bc006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:51.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.937+0000 7f64d3ce4700 1 -- 172.21.15.116:0/2223173648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64cc100e70 msgr2=0x7f64cc19d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:51.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.937+0000 7f64d3ce4700 1 --2- 172.21.15.116:0/2223173648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64cc100e70 0x7f64cc19d570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f64c8005850 tx=0x7f64c8005030 comp rx=0 tx=0).stop 2024-03-19T01:03:51.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.937+0000 7f64d3ce4700 1 -- 172.21.15.116:0/2223173648 shutdown_connections 2024-03-19T01:03:51.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.937+0000 7f64d3ce4700 1 --2- 172.21.15.116:0/2223173648 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f64b8039020 0x7f64b803b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:51.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.937+0000 7f64d3ce4700 1 --2- 172.21.15.116:0/2223173648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64cc100e70 0x7f64cc19d570 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:51.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.937+0000 7f64d3ce4700 1 -- 172.21.15.116:0/2223173648 >> 172.21.15.116:0/2223173648 conn(0x7f64cc0fc720 msgr2=0x7f64cc06a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:51.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.937+0000 7f64d3ce4700 1 -- 172.21.15.116:0/2223173648 shutdown_connections 2024-03-19T01:03:51.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:51.937+0000 7f64d3ce4700 1 -- 172.21.15.116:0/2223173648 wait complete. 2024-03-19T01:03:52.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:51 smithi116 ceph-mon[62701]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:03:52.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:51 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:52.559 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi162 2024-03-19T01:03:52.559 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T01:03:52.559 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.conf 2024-03-19T01:03:52.577 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T01:03:52.577 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T01:03:52.633 INFO:tasks.cephadm:Adding host smithi162 to orchestrator... 2024-03-19T01:03:52.634 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch host add smithi162 2024-03-19T01:03:52.869 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:03:53.317 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:52 smithi116 ceph-mon[62701]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:03:54.281 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:53 smithi116 ceph-mon[62701]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:03:54.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.299+0000 7f85b1ac0700 1 -- 172.21.15.116:0/2075438026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac105ba0 msgr2=0x7f85ac105f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:54.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.299+0000 7f85b1ac0700 1 --2- 172.21.15.116:0/2075438026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac105ba0 0x7f85ac105f80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f859c009910 tx=0x7f859c009c20 comp rx=0 tx=0).stop 2024-03-19T01:03:54.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.299+0000 7f85b1ac0700 1 -- 172.21.15.116:0/2075438026 shutdown_connections 2024-03-19T01:03:54.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.299+0000 7f85b1ac0700 1 --2- 172.21.15.116:0/2075438026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac105ba0 0x7f85ac105f80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:54.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.299+0000 7f85b1ac0700 1 -- 172.21.15.116:0/2075438026 >> 172.21.15.116:0/2075438026 conn(0x7f85ac0fbe80 msgr2=0x7f85ac0fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:54.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.299+0000 7f85b1ac0700 1 -- 172.21.15.116:0/2075438026 shutdown_connections 2024-03-19T01:03:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.299+0000 7f85b1ac0700 1 -- 172.21.15.116:0/2075438026 wait complete. 2024-03-19T01:03:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.299+0000 7f85b1ac0700 1 Processor -- start 2024-03-19T01:03:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.300+0000 7f85b1ac0700 1 -- start start 2024-03-19T01:03:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.300+0000 7f85b1ac0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac105ba0 0x7f85ac19d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.300+0000 7f85b1ac0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f85ac10b320 con 0x7f85ac105ba0 2024-03-19T01:03:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.300+0000 7f85ab7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac105ba0 0x7f85ac19d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.300+0000 7f85ab7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac105ba0 0x7f85ac19d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41050/0 (socket says 172.21.15.116:41050) 2024-03-19T01:03:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.300+0000 7f85ab7fe700 1 -- 172.21.15.116:0/1617225350 learned_addr learned my addr 172.21.15.116:0/1617225350 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:03:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.300+0000 7f85ab7fe700 1 -- 172.21.15.116:0/1617225350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ac19dac0 con 0x7f85ac105ba0 2024-03-19T01:03:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.300+0000 7f85ab7fe700 1 --2- 172.21.15.116:0/1617225350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac105ba0 0x7f85ac19d580 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f859c005850 tx=0x7f859c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.300+0000 7f85a8ff9700 1 -- 172.21.15.116:0/1617225350 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f859c026070 con 0x7f85ac105ba0 2024-03-19T01:03:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.301+0000 7f85a8ff9700 1 -- 172.21.15.116:0/1617225350 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f859c00bc30 con 0x7f85ac105ba0 2024-03-19T01:03:54.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.301+0000 7f85b1ac0700 1 -- 172.21.15.116:0/1617225350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ac19dd50 con 0x7f85ac105ba0 2024-03-19T01:03:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.301+0000 7f85a8ff9700 1 -- 172.21.15.116:0/1617225350 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f859c003a20 con 0x7f85ac105ba0 2024-03-19T01:03:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.301+0000 7f85b1ac0700 1 -- 172.21.15.116:0/1617225350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ac197650 con 0x7f85ac105ba0 2024-03-19T01:03:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.301+0000 7f85a8ff9700 1 -- 172.21.15.116:0/1617225350 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f859c029070 con 0x7f85ac105ba0 2024-03-19T01:03:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.301+0000 7f85b1ac0700 1 -- 172.21.15.116:0/1617225350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85ac102360 con 0x7f85ac105ba0 2024-03-19T01:03:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.302+0000 7f85a8ff9700 1 --2- 172.21.15.116:0/1617225350 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8598039020 0x7f859803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:03:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.302+0000 7f85a8ff9700 1 -- 172.21.15.116:0/1617225350 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f859c01e070 con 0x7f85ac105ba0 2024-03-19T01:03:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.302+0000 7f85aaffd700 1 --2- 172.21.15.116:0/1617225350 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8598039020 0x7f859803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:03:54.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.302+0000 7f85aaffd700 1 --2- 172.21.15.116:0/1617225350 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8598039020 0x7f859803b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8594006fd0 tx=0x7f8594006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:03:54.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.306+0000 7f85a8ff9700 1 -- 172.21.15.116:0/1617225350 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f859c013070 con 0x7f85ac105ba0 2024-03-19T01:03:54.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:54.507+0000 7f85b1ac0700 1 -- 172.21.15.116:0/1617225350 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}) -- 0x7f85ac198b80 con 0x7f8598039020 2024-03-19T01:03:56.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:56 smithi116 ceph-mon[62701]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:03:56.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:56 smithi116 ceph-mon[62701]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:03:56.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:56.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:56.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:56.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:56.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:03:56.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:03:56.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:03:56.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:56 smithi116 ceph-mon[62701]: Updating smithi156:/etc/ceph/ceph.conf 2024-03-19T01:03:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:57 smithi116 ceph-mon[62701]: Deploying cephadm binary to smithi162 2024-03-19T01:03:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:57 smithi116 ceph-mon[62701]: Updating smithi156:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:03:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:57 smithi116 ceph-mon[62701]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-03-19T01:03:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:58 smithi116 ceph-mon[62701]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:58.596+0000 7f85a8ff9700 1 -- 172.21.15.116:0/1617225350 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f85ac198b80 con 0x7f8598039020 2024-03-19T01:03:58.597 INFO:teuthology.orchestra.run.smithi116.stdout:Added host 'smithi162' with addr '172.21.15.162' 2024-03-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:58.598+0000 7f85b1ac0700 1 -- 172.21.15.116:0/1617225350 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8598039020 msgr2=0x7f859803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:58.598+0000 7f85b1ac0700 1 --2- 172.21.15.116:0/1617225350 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8598039020 0x7f859803b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8594006fd0 tx=0x7f8594006e40 comp rx=0 tx=0).stop 2024-03-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:58.598+0000 7f85b1ac0700 1 -- 172.21.15.116:0/1617225350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac105ba0 msgr2=0x7f85ac19d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:58.598+0000 7f85b1ac0700 1 --2- 172.21.15.116:0/1617225350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac105ba0 0x7f85ac19d580 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f859c005850 tx=0x7f859c005030 comp rx=0 tx=0).stop 2024-03-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:58.598+0000 7f85b1ac0700 1 -- 172.21.15.116:0/1617225350 shutdown_connections 2024-03-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:58.598+0000 7f85b1ac0700 1 --2- 172.21.15.116:0/1617225350 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8598039020 0x7f859803b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:58.598+0000 7f85b1ac0700 1 --2- 172.21.15.116:0/1617225350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ac105ba0 0x7f85ac19d580 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:58.598+0000 7f85b1ac0700 1 -- 172.21.15.116:0/1617225350 >> 172.21.15.116:0/1617225350 conn(0x7f85ac0fbe80 msgr2=0x7f85ac100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:03:58.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:58.598+0000 7f85b1ac0700 1 -- 172.21.15.116:0/1617225350 shutdown_connections 2024-03-19T01:03:58.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:03:58.598+0000 7f85b1ac0700 1 -- 172.21.15.116:0/1617225350 wait complete. 2024-03-19T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:59 smithi116 ceph-mon[62701]: Updating smithi156:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:03:59.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:03:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:03:59.269 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-19T01:03:59.501 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:04:00.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:00 smithi116 ceph-mon[62701]: Added host smithi162 2024-03-19T01:04:00.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:00 smithi116 ceph-mon[62701]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:00.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:00 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:00.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:00 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.010+0000 7f3fc141a700 1 -- 172.21.15.116:0/2388259234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100e30 msgr2=0x7f3fbc101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.010+0000 7f3fc141a700 1 --2- 172.21.15.116:0/2388259234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100e30 0x7f3fbc101210 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3fb400b0a0 tx=0x7f3fb400b3b0 comp rx=0 tx=0).stop 2024-03-19T01:04:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.010+0000 7f3fc141a700 1 -- 172.21.15.116:0/2388259234 shutdown_connections 2024-03-19T01:04:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.010+0000 7f3fc141a700 1 --2- 172.21.15.116:0/2388259234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100e30 0x7f3fbc101210 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.010+0000 7f3fc141a700 1 -- 172.21.15.116:0/2388259234 >> 172.21.15.116:0/2388259234 conn(0x7f3fbc0fc6c0 msgr2=0x7f3fbc0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.010+0000 7f3fc141a700 1 -- 172.21.15.116:0/2388259234 shutdown_connections 2024-03-19T01:04:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.010+0000 7f3fc141a700 1 -- 172.21.15.116:0/2388259234 wait complete. 2024-03-19T01:04:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.010+0000 7f3fc141a700 1 Processor -- start 2024-03-19T01:04:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.011+0000 7f3fc141a700 1 -- start start 2024-03-19T01:04:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.011+0000 7f3fc141a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100e30 0x7f3fbc1028d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.011+0000 7f3fc141a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3fbc10b200 con 0x7f3fbc100e30 2024-03-19T01:04:01.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.011+0000 7f3fbaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100e30 0x7f3fbc1028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:01.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.011+0000 7f3fbaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100e30 0x7f3fbc1028d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60910/0 (socket says 172.21.15.116:60910) 2024-03-19T01:04:01.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.011+0000 7f3fbaffd700 1 -- 172.21.15.116:0/3559998370 learned_addr learned my addr 172.21.15.116:0/3559998370 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:04:01.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.011+0000 7f3fbaffd700 1 -- 172.21.15.116:0/3559998370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fbc1047c0 con 0x7f3fbc100e30 2024-03-19T01:04:01.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.011+0000 7f3fbaffd700 1 --2- 172.21.15.116:0/3559998370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100e30 0x7f3fbc1028d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4015040 tx=0x7f3fb4003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.011+0000 7f3fa3fff700 1 -- 172.21.15.116:0/3559998370 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3fb4017040 con 0x7f3fbc100e30 2024-03-19T01:04:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.011+0000 7f3fc141a700 1 -- 172.21.15.116:0/3559998370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fbc102e70 con 0x7f3fbc100e30 2024-03-19T01:04:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.011+0000 7f3fa3fff700 1 -- 172.21.15.116:0/3559998370 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fb4022070 con 0x7f3fbc100e30 2024-03-19T01:04:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.011+0000 7f3fa3fff700 1 -- 172.21.15.116:0/3559998370 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3fb401d5d0 con 0x7f3fbc100e30 2024-03-19T01:04:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.012+0000 7f3fc141a700 1 -- 172.21.15.116:0/3559998370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fbc103330 con 0x7f3fbc100e30 2024-03-19T01:04:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.012+0000 7f3fa3fff700 1 -- 172.21.15.116:0/3559998370 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f3fb4004000 con 0x7f3fbc100e30 2024-03-19T01:04:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.012+0000 7f3fc141a700 1 -- 172.21.15.116:0/3559998370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fbc102310 con 0x7f3fbc100e30 2024-03-19T01:04:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.013+0000 7f3fa3fff700 1 --2- 172.21.15.116:0/3559998370 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3fa4038fc0 0x7f3fa403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:01.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.013+0000 7f3fa3fff700 1 -- 172.21.15.116:0/3559998370 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3fb400f050 con 0x7f3fbc100e30 2024-03-19T01:04:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.013+0000 7f3fba7fc700 1 --2- 172.21.15.116:0/3559998370 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3fa4038fc0 0x7f3fa403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:01.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.013+0000 7f3fba7fc700 1 --2- 172.21.15.116:0/3559998370 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3fa4038fc0 0x7f3fa403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006fd0 tx=0x7f3fb0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:01.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.018+0000 7f3fa3fff700 1 -- 172.21.15.116:0/3559998370 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3fb4007e20 con 0x7f3fbc100e30 2024-03-19T01:04:01.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.218+0000 7f3fc141a700 1 -- 172.21.15.116:0/3559998370 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3fbc103560 con 0x7f3fa4038fc0 2024-03-19T01:04:01.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.219+0000 7f3fa3fff700 1 -- 172.21.15.116:0/3559998370 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f3fbc103560 con 0x7f3fa4038fc0 2024-03-19T01:04:01.219 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:04:01.220 INFO:teuthology.orchestra.run.smithi116.stdout:[{"addr": "172.21.15.116", "hostname": "smithi116", "labels": [], "status": ""}, {"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}, {"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}] 2024-03-19T01:04:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.221+0000 7f3fc141a700 1 -- 172.21.15.116:0/3559998370 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3fa4038fc0 msgr2=0x7f3fa403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.221+0000 7f3fc141a700 1 --2- 172.21.15.116:0/3559998370 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3fa4038fc0 0x7f3fa403b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0006fd0 tx=0x7f3fb0006e40 comp rx=0 tx=0).stop 2024-03-19T01:04:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.221+0000 7f3fc141a700 1 -- 172.21.15.116:0/3559998370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100e30 msgr2=0x7f3fbc1028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.221+0000 7f3fc141a700 1 --2- 172.21.15.116:0/3559998370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100e30 0x7f3fbc1028d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3fb4015040 tx=0x7f3fb4003d50 comp rx=0 tx=0).stop 2024-03-19T01:04:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.221+0000 7f3fc141a700 1 -- 172.21.15.116:0/3559998370 shutdown_connections 2024-03-19T01:04:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.221+0000 7f3fc141a700 1 --2- 172.21.15.116:0/3559998370 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3fa4038fc0 0x7f3fa403b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.221+0000 7f3fc141a700 1 --2- 172.21.15.116:0/3559998370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3fbc100e30 0x7f3fbc1028d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.221+0000 7f3fc141a700 1 -- 172.21.15.116:0/3559998370 >> 172.21.15.116:0/3559998370 conn(0x7f3fbc0fc6c0 msgr2=0x7f3fbc06a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.221+0000 7f3fc141a700 1 -- 172.21.15.116:0/3559998370 shutdown_connections 2024-03-19T01:04:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:01.221+0000 7f3fc141a700 1 -- 172.21.15.116:0/3559998370 wait complete. 2024-03-19T01:04:01.843 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-19T01:04:01.847 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-19T01:04:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:04:02.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:02.147 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:01 smithi116 ceph-mon[62701]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:03.126 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:02 smithi116 ceph-mon[62701]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:04:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.587+0000 7f3845521700 1 -- 172.21.15.116:0/119066475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3840102e90 msgr2=0x7f3840103270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.587+0000 7f3845521700 1 --2- 172.21.15.116:0/119066475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3840102e90 0x7f3840103270 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f383000b0a0 tx=0x7f383000b3b0 comp rx=0 tx=0).stop 2024-03-19T01:04:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.587+0000 7f3845521700 1 -- 172.21.15.116:0/119066475 shutdown_connections 2024-03-19T01:04:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.587+0000 7f3845521700 1 --2- 172.21.15.116:0/119066475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3840102e90 0x7f3840103270 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.587+0000 7f3845521700 1 -- 172.21.15.116:0/119066475 >> 172.21.15.116:0/119066475 conn(0x7f38400fc700 msgr2=0x7f38400feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.587+0000 7f3845521700 1 -- 172.21.15.116:0/119066475 shutdown_connections 2024-03-19T01:04:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.587+0000 7f3845521700 1 -- 172.21.15.116:0/119066475 wait complete. 2024-03-19T01:04:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.588+0000 7f3845521700 1 Processor -- start 2024-03-19T01:04:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.588+0000 7f3845521700 1 -- start start 2024-03-19T01:04:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.588+0000 7f3845521700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3840102e90 0x7f3840100af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.588+0000 7f3845521700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f384010c380 con 0x7f3840102e90 2024-03-19T01:04:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.588+0000 7f383effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3840102e90 0x7f3840100af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.588+0000 7f383effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3840102e90 0x7f3840100af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60926/0 (socket says 172.21.15.116:60926) 2024-03-19T01:04:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.588+0000 7f383effd700 1 -- 172.21.15.116:0/3112555393 learned_addr learned my addr 172.21.15.116:0/3112555393 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:04:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.588+0000 7f383effd700 1 -- 172.21.15.116:0/3112555393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38401029e0 con 0x7f3840102e90 2024-03-19T01:04:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.588+0000 7f383effd700 1 --2- 172.21.15.116:0/3112555393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3840102e90 0x7f3840100af0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f38300062a0 tx=0x7f383000bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.589+0000 7f3837fff700 1 -- 172.21.15.116:0/3112555393 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3830017050 con 0x7f3840102e90 2024-03-19T01:04:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.589+0000 7f3837fff700 1 -- 172.21.15.116:0/3112555393 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3830023070 con 0x7f3840102e90 2024-03-19T01:04:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.589+0000 7f3845521700 1 -- 172.21.15.116:0/3112555393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3840101030 con 0x7f3840102e90 2024-03-19T01:04:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.589+0000 7f3837fff700 1 -- 172.21.15.116:0/3112555393 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3830004000 con 0x7f3840102e90 2024-03-19T01:04:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.589+0000 7f3845521700 1 -- 172.21.15.116:0/3112555393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38401014f0 con 0x7f3840102e90 2024-03-19T01:04:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.589+0000 7f3837fff700 1 -- 172.21.15.116:0/3112555393 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f38300042c0 con 0x7f3840102e90 2024-03-19T01:04:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.590+0000 7f3837fff700 1 --2- 172.21.15.116:0/3112555393 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f382c038fc0 0x7f382c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.590+0000 7f3837fff700 1 -- 172.21.15.116:0/3112555393 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f383000f050 con 0x7f3840102e90 2024-03-19T01:04:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.590+0000 7f383e7fc700 1 --2- 172.21.15.116:0/3112555393 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f382c038fc0 0x7f382c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.590+0000 7f3845521700 1 -- 172.21.15.116:0/3112555393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f384006a880 con 0x7f3840102e90 2024-03-19T01:04:03.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.590+0000 7f383e7fc700 1 --2- 172.21.15.116:0/3112555393 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f382c038fc0 0x7f382c03b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3828006fd0 tx=0x7f3828006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:03.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.595+0000 7f3837fff700 1 -- 172.21.15.116:0/3112555393 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3830026df0 con 0x7f3840102e90 2024-03-19T01:04:03.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:03.799+0000 7f3845521700 1 -- 172.21.15.116:0/3112555393 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f3840063940 con 0x7f3840102e90 2024-03-19T01:04:04.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:03 smithi116 ceph-mon[62701]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:04.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:04.714+0000 7f3837fff700 1 -- 172.21.15.116:0/3112555393 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f383001cc90 con 0x7f3840102e90 2024-03-19T01:04:04.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:04.717+0000 7f3845521700 1 -- 172.21.15.116:0/3112555393 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f382c038fc0 msgr2=0x7f382c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:04.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:04.717+0000 7f3845521700 1 --2- 172.21.15.116:0/3112555393 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f382c038fc0 0x7f382c03b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3828006fd0 tx=0x7f3828006e40 comp rx=0 tx=0).stop 2024-03-19T01:04:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:04.717+0000 7f3845521700 1 -- 172.21.15.116:0/3112555393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3840102e90 msgr2=0x7f3840100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:04.717+0000 7f3845521700 1 --2- 172.21.15.116:0/3112555393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3840102e90 0x7f3840100af0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f38300062a0 tx=0x7f383000bdd0 comp rx=0 tx=0).stop 2024-03-19T01:04:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:04.717+0000 7f3845521700 1 -- 172.21.15.116:0/3112555393 shutdown_connections 2024-03-19T01:04:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:04.717+0000 7f3845521700 1 --2- 172.21.15.116:0/3112555393 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f382c038fc0 0x7f382c03b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:04.717+0000 7f3845521700 1 --2- 172.21.15.116:0/3112555393 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3840102e90 0x7f3840100af0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:04.717+0000 7f3845521700 1 -- 172.21.15.116:0/3112555393 >> 172.21.15.116:0/3112555393 conn(0x7f38400fc700 msgr2=0x7f3840103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:04.717+0000 7f3845521700 1 -- 172.21.15.116:0/3112555393 shutdown_connections 2024-03-19T01:04:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:04.717+0000 7f3845521700 1 -- 172.21.15.116:0/3112555393 wait complete. 2024-03-19T01:04:04.719 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2024-03-19T01:04:05.099 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:04 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3112555393' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-19T01:04:05.099 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:05.099 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:05.099 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:05.099 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:05.100 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:04:05.100 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:05.100 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:05.100 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:04 smithi116 ceph-mon[62701]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-19T01:04:05.400 INFO:tasks.cephadm:Adding mon.a on smithi116 2024-03-19T01:04:05.400 INFO:tasks.cephadm:Adding mon.b on smithi156 2024-03-19T01:04:05.401 INFO:tasks.cephadm:Adding mon.c on smithi162 2024-03-19T01:04:05.401 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi116:172.21.15.116=a;smithi156:172.21.15.156=b;smithi162:172.21.15.162=c' 2024-03-19T01:04:05.629 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:06.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:05 smithi116 ceph-mon[62701]: Updating smithi162:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:06.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:05 smithi116 ceph-mon[62701]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-03-19T01:04:06.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:05 smithi116 ceph-mon[62701]: Updating smithi162:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T01:04:06.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:05 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3112555393' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-19T01:04:06.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:05 smithi116 ceph-mon[62701]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T01:04:06.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:06.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:06.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:06.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:05 smithi116 ceph-mon[62701]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:07.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.134+0000 7f071d4a2700 1 -- 172.21.15.162:0/172092324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718101f30 msgr2=0x7f0718102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:07.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.134+0000 7f071d4a2700 1 --2- 172.21.15.162:0/172092324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718101f30 0x7f0718102310 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0708009910 tx=0x7f0708009c20 comp rx=0 tx=0).stop 2024-03-19T01:04:07.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.134+0000 7f071d4a2700 1 -- 172.21.15.162:0/172092324 shutdown_connections 2024-03-19T01:04:07.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.134+0000 7f071d4a2700 1 --2- 172.21.15.162:0/172092324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718101f30 0x7f0718102310 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:07.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.134+0000 7f071d4a2700 1 -- 172.21.15.162:0/172092324 >> 172.21.15.162:0/172092324 conn(0x7f07180fd7e0 msgr2=0x7f07180ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:07.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.134+0000 7f071d4a2700 1 -- 172.21.15.162:0/172092324 shutdown_connections 2024-03-19T01:04:07.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.134+0000 7f071d4a2700 1 -- 172.21.15.162:0/172092324 wait complete. 2024-03-19T01:04:07.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.135+0000 7f071d4a2700 1 Processor -- start 2024-03-19T01:04:07.136 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.135+0000 7f071d4a2700 1 -- start start 2024-03-19T01:04:07.137 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.135+0000 7f071d4a2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718101f30 0x7f071819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:07.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.135+0000 7f071d4a2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f071810c300 con 0x7f0718101f30 2024-03-19T01:04:07.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.136+0000 7f0716ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718101f30 0x7f071819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:07.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.136+0000 7f0716ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718101f30 0x7f071819d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.162:53836/0 (socket says 172.21.15.162:53836) 2024-03-19T01:04:07.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.136+0000 7f0716ffd700 1 -- 172.21.15.162:0/263848765 learned_addr learned my addr 172.21.15.162:0/263848765 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T01:04:07.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.136+0000 7f0716ffd700 1 -- 172.21.15.162:0/263848765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f071819db30 con 0x7f0718101f30 2024-03-19T01:04:07.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.136+0000 7f0716ffd700 1 --2- 172.21.15.162:0/263848765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718101f30 0x7f071819d5f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0708006010 tx=0x7f0708005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:07.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.136+0000 7f06fffff700 1 -- 172.21.15.162:0/263848765 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0708026070 con 0x7f0718101f30 2024-03-19T01:04:07.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.136+0000 7f06fffff700 1 -- 172.21.15.162:0/263848765 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f070800bbd0 con 0x7f0718101f30 2024-03-19T01:04:07.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.136+0000 7f071d4a2700 1 -- 172.21.15.162:0/263848765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f071819de20 con 0x7f0718101f30 2024-03-19T01:04:07.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.136+0000 7f06fffff700 1 -- 172.21.15.162:0/263848765 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0708003910 con 0x7f0718101f30 2024-03-19T01:04:07.138 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.136+0000 7f071d4a2700 1 -- 172.21.15.162:0/263848765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0718197890 con 0x7f0718101f30 2024-03-19T01:04:07.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.137+0000 7f071d4a2700 1 -- 172.21.15.162:0/263848765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0718103410 con 0x7f0718101f30 2024-03-19T01:04:07.139 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.137+0000 7f06fffff700 1 -- 172.21.15.162:0/263848765 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f0708029070 con 0x7f0718101f30 2024-03-19T01:04:07.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.137+0000 7f06fffff700 1 --2- 172.21.15.162:0/263848765 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0700038fc0 0x7f070003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:07.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.137+0000 7f06fffff700 1 -- 172.21.15.162:0/263848765 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f070801e070 con 0x7f0718101f30 2024-03-19T01:04:07.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.138+0000 7f07167fc700 1 --2- 172.21.15.162:0/263848765 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0700038fc0 0x7f070003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:07.140 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.138+0000 7f07167fc700 1 --2- 172.21.15.162:0/263848765 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0700038fc0 0x7f070003b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f070c006fd0 tx=0x7f070c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:07.143 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.143+0000 7f06fffff700 1 -- 172.21.15.162:0/263848765 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0708013070 con 0x7f0718101f30 2024-03-19T01:04:07.349 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.348+0000 7f071d4a2700 1 -- 172.21.15.162:0/263848765 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi156:172.21.15.156=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}) -- 0x7f0718198cb0 con 0x7f0700038fc0 2024-03-19T01:04:07.356 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.355+0000 7f06fffff700 1 -- 172.21.15.162:0/263848765 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0718198cb0 con 0x7f0700038fc0 2024-03-19T01:04:07.357 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mon update... 2024-03-19T01:04:07.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.357+0000 7f071d4a2700 1 -- 172.21.15.162:0/263848765 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0700038fc0 msgr2=0x7f070003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:07.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.357+0000 7f071d4a2700 1 --2- 172.21.15.162:0/263848765 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0700038fc0 0x7f070003b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f070c006fd0 tx=0x7f070c006e40 comp rx=0 tx=0).stop 2024-03-19T01:04:07.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.357+0000 7f071d4a2700 1 -- 172.21.15.162:0/263848765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718101f30 msgr2=0x7f071819d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:07.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.357+0000 7f071d4a2700 1 --2- 172.21.15.162:0/263848765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718101f30 0x7f071819d5f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0708006010 tx=0x7f0708005030 comp rx=0 tx=0).stop 2024-03-19T01:04:07.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.358+0000 7f071d4a2700 1 -- 172.21.15.162:0/263848765 shutdown_connections 2024-03-19T01:04:07.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.358+0000 7f071d4a2700 1 --2- 172.21.15.162:0/263848765 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0700038fc0 0x7f070003b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:07.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.358+0000 7f071d4a2700 1 --2- 172.21.15.162:0/263848765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0718101f30 0x7f071819d5f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:07.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.358+0000 7f071d4a2700 1 -- 172.21.15.162:0/263848765 >> 172.21.15.162:0/263848765 conn(0x7f07180fd7e0 msgr2=0x7f0718104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:07.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.358+0000 7f071d4a2700 1 -- 172.21.15.162:0/263848765 shutdown_connections 2024-03-19T01:04:07.359 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:07.358+0000 7f071d4a2700 1 -- 172.21.15.162:0/263848765 wait complete. 2024-03-19T01:04:07.947 DEBUG:teuthology.orchestra.run.smithi156:mon.b> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.b.service 2024-03-19T01:04:07.950 DEBUG:teuthology.orchestra.run.smithi162:mon.c> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.c.service 2024-03-19T01:04:07.952 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-19T01:04:07.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T01:04:08.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:08 smithi116 ceph-mon[62701]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:08.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:08.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:08.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:08.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:08.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:08.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:08.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:08.593 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:09.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:09 smithi116 ceph-mon[62701]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi156:172.21.15.156=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:04:09.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:09 smithi116 ceph-mon[62701]: Saving service mon spec with placement smithi116:172.21.15.116=a;smithi156:172.21.15.156=b;smithi162:172.21.15.162=c;count:3 2024-03-19T01:04:09.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:09 smithi116 ceph-mon[62701]: Deploying daemon mon.c on smithi162 2024-03-19T01:04:10.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:10 smithi116 ceph-mon[62701]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:11.329 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.328+0000 7f46dc6da700 1 -- 172.21.15.162:0/4225307924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d4100380 msgr2=0x7f46d4100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.328+0000 7f46dc6da700 1 --2- 172.21.15.162:0/4225307924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d4100380 0x7f46d4100760 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009910 tx=0x7f46c4009c20 comp rx=0 tx=0).stop 2024-03-19T01:04:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.328+0000 7f46dc6da700 1 -- 172.21.15.162:0/4225307924 shutdown_connections 2024-03-19T01:04:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.328+0000 7f46dc6da700 1 --2- 172.21.15.162:0/4225307924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d4100380 0x7f46d4100760 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.328+0000 7f46dc6da700 1 -- 172.21.15.162:0/4225307924 >> 172.21.15.162:0/4225307924 conn(0x7f46d40fbe80 msgr2=0x7f46d40fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.328+0000 7f46dc6da700 1 -- 172.21.15.162:0/4225307924 shutdown_connections 2024-03-19T01:04:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.328+0000 7f46dc6da700 1 -- 172.21.15.162:0/4225307924 wait complete. 2024-03-19T01:04:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.328+0000 7f46dc6da700 1 Processor -- start 2024-03-19T01:04:11.330 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.328+0000 7f46dc6da700 1 -- start start 2024-03-19T01:04:11.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.328+0000 7f46dc6da700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d4100380 0x7f46d419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:11.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.328+0000 7f46dc6da700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46d410c370 con 0x7f46d4100380 2024-03-19T01:04:11.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.329+0000 7f46da476700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d4100380 0x7f46d419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:11.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.329+0000 7f46da476700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d4100380 0x7f46d419e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.162:42702/0 (socket says 172.21.15.162:42702) 2024-03-19T01:04:11.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.329+0000 7f46da476700 1 -- 172.21.15.162:0/1338075516 learned_addr learned my addr 172.21.15.162:0/1338075516 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T01:04:11.331 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.329+0000 7f46da476700 1 -- 172.21.15.162:0/1338075516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d419eb30 con 0x7f46d4100380 2024-03-19T01:04:11.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.329+0000 7f46da476700 1 --2- 172.21.15.162:0/1338075516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d4100380 0x7f46d419e5f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f46c4006010 tx=0x7f46c4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:11.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.329+0000 7f46cb7fe700 1 -- 172.21.15.162:0/1338075516 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46c4026070 con 0x7f46d4100380 2024-03-19T01:04:11.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.330+0000 7f46dc6da700 1 -- 172.21.15.162:0/1338075516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46d419ee20 con 0x7f46d4100380 2024-03-19T01:04:11.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.330+0000 7f46cb7fe700 1 -- 172.21.15.162:0/1338075516 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46c400bbd0 con 0x7f46d4100380 2024-03-19T01:04:11.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.330+0000 7f46dc6da700 1 -- 172.21.15.162:0/1338075516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46d41a26c0 con 0x7f46d4100380 2024-03-19T01:04:11.332 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.330+0000 7f46cb7fe700 1 -- 172.21.15.162:0/1338075516 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46c4003910 con 0x7f46d4100380 2024-03-19T01:04:11.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.330+0000 7f46dc6da700 1 -- 172.21.15.162:0/1338075516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46d4103440 con 0x7f46d4100380 2024-03-19T01:04:11.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.331+0000 7f46cb7fe700 1 -- 172.21.15.162:0/1338075516 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f46c4029070 con 0x7f46d4100380 2024-03-19T01:04:11.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.331+0000 7f46cb7fe700 1 --2- 172.21.15.162:0/1338075516 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f46c0039020 0x7f46c003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:11.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.331+0000 7f46cb7fe700 1 -- 172.21.15.162:0/1338075516 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f46c401e070 con 0x7f46d4100380 2024-03-19T01:04:11.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.331+0000 7f46d9c75700 1 --2- 172.21.15.162:0/1338075516 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f46c0039020 0x7f46c003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:11.334 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.332+0000 7f46d9c75700 1 --2- 172.21.15.162:0/1338075516 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f46c0039020 0x7f46c003b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f46d0006fd0 tx=0x7f46d0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:11.336 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.336+0000 7f46cb7fe700 1 -- 172.21.15.162:0/1338075516 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f46c4013070 con 0x7f46d4100380 2024-03-19T01:04:11.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.641+0000 7f46dc6da700 1 -- 172.21.15.162:0/1338075516 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f46d4063940 con 0x7f46d4100380 2024-03-19T01:04:11.641 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.641+0000 7f46cb7fe700 1 -- 172.21.15.162:0/1338075516 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f46c4030090 con 0x7f46d4100380 2024-03-19T01:04:11.642 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T01:04:11.642 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":1,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","modified":"2024-03-19T01:02:27.623031Z","created":"2024-03-19T01:02:27.623031Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-19T01:04:11.644 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.644+0000 7f46dc6da700 1 -- 172.21.15.162:0/1338075516 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f46c0039020 msgr2=0x7f46c003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:11.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.644+0000 7f46dc6da700 1 --2- 172.21.15.162:0/1338075516 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f46c0039020 0x7f46c003b4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f46d0006fd0 tx=0x7f46d0006e40 comp rx=0 tx=0).stop 2024-03-19T01:04:11.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.645+0000 7f46dc6da700 1 -- 172.21.15.162:0/1338075516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d4100380 msgr2=0x7f46d419e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:11.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.645+0000 7f46dc6da700 1 --2- 172.21.15.162:0/1338075516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d4100380 0x7f46d419e5f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f46c4006010 tx=0x7f46c4005030 comp rx=0 tx=0).stop 2024-03-19T01:04:11.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.645+0000 7f46dc6da700 1 -- 172.21.15.162:0/1338075516 shutdown_connections 2024-03-19T01:04:11.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.645+0000 7f46dc6da700 1 --2- 172.21.15.162:0/1338075516 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f46c0039020 0x7f46c003b4e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:11.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.645+0000 7f46dc6da700 1 --2- 172.21.15.162:0/1338075516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46d4100380 0x7f46d419e5f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:11.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.645+0000 7f46dc6da700 1 -- 172.21.15.162:0/1338075516 >> 172.21.15.162:0/1338075516 conn(0x7f46d40fbe80 msgr2=0x7f46d40fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:11.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.645+0000 7f46dc6da700 1 -- 172.21.15.162:0/1338075516 shutdown_connections 2024-03-19T01:04:11.645 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:11.645+0000 7f46dc6da700 1 -- 172.21.15.162:0/1338075516 wait complete. 2024-03-19T01:04:11.645 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 1 2024-03-19T01:04:11.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:11 smithi162 systemd[1]: Starting Ceph mon.c for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:04:12.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:12 smithi116 ceph-mon[62701]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:12.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:12 smithi116 ceph-mon[62701]: from='client.? 172.21.15.162:0/1338075516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T01:04:12.544 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 podman[67159]: 2024-03-19 01:04:12.298862514 +0000 UTC m=+0.017514455 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:04:12.544 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 podman[67159]: 2024-03-19 01:04:12.427399669 +0000 UTC m=+0.146051605 container create 8c9c409520b397746bd6310fadbae6ea8be3a5532c1ca84b396b0296047b898a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-03-19T01:04:12.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 podman[67159]: 2024-03-19 01:04:12.637837767 +0000 UTC m=+0.356489719 container init 8c9c409520b397746bd6310fadbae6ea8be3a5532c1ca84b396b0296047b898a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-19T01:04:12.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 podman[67159]: 2024-03-19 01:04:12.640910407 +0000 UTC m=+0.359562344 container start 8c9c409520b397746bd6310fadbae6ea8be3a5532c1ca84b396b0296047b898a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-c, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:04:12.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: set uid:gid to 167:167 (ceph:ceph) 2024-03-19T01:04:12.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: ceph version 19.0.0-2206-gb6ae2491 (b6ae24918b03b1b8a19c9263a50655bba80bb810) squid (dev), process ceph-mon, pid 2 2024-03-19T01:04:12.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: pidfile_write: ignore empty --pid-file 2024-03-19T01:04:12.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: load: jerasure load: lrc 2024-03-19T01:04:12.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: RocksDB version: 7.9.2 2024-03-19T01:04:12.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Git sha 0 2024-03-19T01:04:12.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Compile date 2024-03-18 18:57:12 2024-03-19T01:04:12.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: DB SUMMARY 2024-03-19T01:04:12.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: DB Session ID: RG6NUMYWGBEWOTZJJLMJ 2024-03-19T01:04:12.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: CURRENT file: CURRENT 2024-03-19T01:04:12.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: IDENTITY file: IDENTITY 2024-03-19T01:04:12.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-19T01:04:12.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-19T01:04:12.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-19T01:04:12.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.error_if_exists: 0 2024-03-19T01:04:12.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.create_if_missing: 0 2024-03-19T01:04:12.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.paranoid_checks: 1 2024-03-19T01:04:12.903 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-19T01:04:12.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-19T01:04:12.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-19T01:04:12.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.env: 0x5645bc9b2c60 2024-03-19T01:04:12.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.fs: PosixFileSystem 2024-03-19T01:04:12.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.info_log: 0x5645be7edae0 2024-03-19T01:04:12.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_file_opening_threads: 16 2024-03-19T01:04:12.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.statistics: (nil) 2024-03-19T01:04:12.904 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.use_fsync: 0 2024-03-19T01:04:12.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_log_file_size: 0 2024-03-19T01:04:12.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-19T01:04:12.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-19T01:04:12.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.keep_log_file_num: 1000 2024-03-19T01:04:12.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.recycle_log_file_num: 0 2024-03-19T01:04:12.905 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.allow_fallocate: 1 2024-03-19T01:04:12.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.allow_mmap_reads: 0 2024-03-19T01:04:12.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.allow_mmap_writes: 0 2024-03-19T01:04:12.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.use_direct_reads: 0 2024-03-19T01:04:12.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-19T01:04:12.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.create_missing_column_families: 0 2024-03-19T01:04:12.906 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.db_log_dir: 2024-03-19T01:04:12.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.wal_dir: 2024-03-19T01:04:12.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-19T01:04:12.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-19T01:04:12.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-19T01:04:12.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-19T01:04:12.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-19T01:04:12.907 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-19T01:04:12.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.advise_random_on_open: 1 2024-03-19T01:04:12.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.db_write_buffer_size: 0 2024-03-19T01:04:12.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.write_buffer_manager: 0x5645be7c01e0 2024-03-19T01:04:12.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-19T01:04:12.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-19T01:04:12.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-19T01:04:12.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.rate_limiter: (nil) 2024-03-19T01:04:12.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-19T01:04:12.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.wal_recovery_mode: 2 2024-03-19T01:04:12.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.enable_thread_tracking: 0 2024-03-19T01:04:12.908 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.enable_pipelined_write: 0 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.unordered_write: 0 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.row_cache: None 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.wal_filter: None 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.allow_ingest_behind: 0 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.two_write_queues: 0 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.manual_wal_flush: 0 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.wal_compression: 0 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.atomic_flush: 0 2024-03-19T01:04:12.909 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-19T01:04:12.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-19T01:04:12.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-19T01:04:12.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.log_readahead_size: 0 2024-03-19T01:04:12.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-19T01:04:12.910 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.best_efforts_recovery: 0 2024-03-19T01:04:12.911 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.allow_data_in_errors: 0 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.db_host_id: __hostname__ 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_background_jobs: 2 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_background_compactions: -1 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_subcompactions: 1 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_total_wal_size: 0 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-19T01:04:12.912 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_open_files: -1 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.bytes_per_sync: 0 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compaction_readahead_size: 0 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_background_flushes: -1 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Compression algorithms supported: 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: kZSTD supported: 0 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: kXpressCompression supported: 0 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: kLZ4HCCompression supported: 1 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: kZlibCompression supported: 1 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: kSnappyCompression supported: 1 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: kLZ4Compression supported: 1 2024-03-19T01:04:12.913 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: kBZip2Compression supported: 0 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.merge_operator: 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compaction_filter: None 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compaction_filter_factory: None 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.sst_partitioner_factory: None 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5645be7edbe0) 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout: cache_index_and_filter_blocks: 1 2024-03-19T01:04:12.914 INFO:journalctl@ceph.mon.c.smithi162.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: pin_top_level_index_and_filter: 1 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_type: 0 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: data_block_index_type: 0 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_shortening: 1 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: checksum: 4 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: no_block_cache: 0 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache: 0x5645be7f3090 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_name: BinnedLRUCache 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_options: 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: capacity : 536870912 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: num_shard_bits : 4 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: strict_capacity_limit : 0 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: high_pri_pool_ratio: 0.000 2024-03-19T01:04:12.915 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_cache_compressed: (nil) 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: persistent_cache: (nil) 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_size: 4096 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_size_deviation: 10 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_restart_interval: 16 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: index_block_restart_interval: 1 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: metadata_block_size: 4096 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: partition_filters: 0 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: use_delta_encoding: 1 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: filter_policy: bloomfilter 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: whole_key_filtering: 1 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: verify_compression: 0 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: read_amp_bytes_per_bit: 0 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: format_version: 5 2024-03-19T01:04:12.916 INFO:journalctl@ceph.mon.c.smithi162.stdout: enable_index_compression: 1 2024-03-19T01:04:12.917 INFO:journalctl@ceph.mon.c.smithi162.stdout: block_align: 0 2024-03-19T01:04:12.917 INFO:journalctl@ceph.mon.c.smithi162.stdout: max_auto_readahead_size: 262144 2024-03-19T01:04:12.917 INFO:journalctl@ceph.mon.c.smithi162.stdout: prepopulate_block_cache: 0 2024-03-19T01:04:12.917 INFO:journalctl@ceph.mon.c.smithi162.stdout: initial_auto_readahead_size: 8192 2024-03-19T01:04:12.918 INFO:journalctl@ceph.mon.c.smithi162.stdout: num_file_reads_for_auto_readahead: 2 2024-03-19T01:04:12.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.write_buffer_size: 33554432 2024-03-19T01:04:12.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_write_buffer_number: 2 2024-03-19T01:04:12.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compression: NoCompression 2024-03-19T01:04:12.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.bottommost_compression: Disabled 2024-03-19T01:04:12.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.prefix_extractor: nullptr 2024-03-19T01:04:12.918 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-19T01:04:12.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.num_levels: 7 2024-03-19T01:04:12.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-19T01:04:12.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-19T01:04:12.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-19T01:04:12.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-19T01:04:12.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-19T01:04:12.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-19T01:04:12.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-19T01:04:12.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-19T01:04:12.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-19T01:04:12.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-19T01:04:12.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-19T01:04:12.919 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compression_opts.level: 32767 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compression_opts.strategy: 0 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compression_opts.enabled: false 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.target_file_size_base: 67108864 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-19T01:04:12.920 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.arena_block_size: 1048576 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-19T01:04:12.921 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.disable_auto_compactions: 0 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.table_properties_collectors: 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.inplace_update_support: 0 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-19T01:04:12.922 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.bloom_locality: 0 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.max_successive_merges: 0 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.paranoid_file_checks: 0 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.force_consistency_checks: 1 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.report_bg_io_stats: 0 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.ttl: 2592000 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.enable_blob_files: false 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.min_blob_size: 0 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.blob_file_size: 268435456 2024-03-19T01:04:12.923 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.blob_file_starting_level: 0 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d6941385-f7e2-4ddf-bab1-e7a137037f8a 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710810252670514, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710810252670936, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710810252, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d6941385-f7e2-4ddf-bab1-e7a137037f8a", "db_session_id": "RG6NUMYWGBEWOTZJJLMJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710810252671015, "job": 1, "event": "recovery_finished"} 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-19T01:04:12.924 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-19T01:04:12.925 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5645be8e6000 2024-03-19T01:04:12.925 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: DB pointer 0x5645be8d0000 2024-03-19T01:04:12.925 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-19T01:04:12.925 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-19T01:04:12.925 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** DB Stats ** 2024-03-19T01:04:12.925 INFO:journalctl@ceph.mon.c.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T01:04:12.925 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-19T01:04:12.925 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T01:04:12.925 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T01:04:12.925 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-19T01:04:12.925 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-19T01:04:12.925 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-19T01:04:12.925 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-19T01:04:12.926 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** Compaction Stats [default] ** 2024-03-19T01:04:12.926 INFO:journalctl@ceph.mon.c.smithi162.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T01:04:12.926 INFO:journalctl@ceph.mon.c.smithi162.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-19T01:04:12.926 INFO:journalctl@ceph.mon.c.smithi162.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T01:04:12.926 INFO:journalctl@ceph.mon.c.smithi162.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T01:04:12.926 INFO:journalctl@ceph.mon.c.smithi162.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T01:04:12.926 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-19T01:04:12.926 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** Compaction Stats [default] ** 2024-03-19T01:04:12.926 INFO:journalctl@ceph.mon.c.smithi162.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-19T01:04:12.926 INFO:journalctl@ceph.mon.c.smithi162.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-19T01:04:12.926 INFO:journalctl@ceph.mon.c.smithi162.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-19T01:04:12.926 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-19T01:04:12.926 INFO:journalctl@ceph.mon.c.smithi162.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-19T01:04:12.927 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-19T01:04:12.927 INFO:journalctl@ceph.mon.c.smithi162.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-19T01:04:12.927 INFO:journalctl@ceph.mon.c.smithi162.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-19T01:04:12.927 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-19T01:04:12.927 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-19T01:04:12.927 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-19T01:04:12.927 INFO:journalctl@ceph.mon.c.smithi162.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-19T01:04:12.927 INFO:journalctl@ceph.mon.c.smithi162.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T01:04:12.927 INFO:journalctl@ceph.mon.c.smithi162.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-19T01:04:12.927 INFO:journalctl@ceph.mon.c.smithi162.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-19T01:04:12.927 INFO:journalctl@ceph.mon.c.smithi162.stdout: Block cache BinnedLRUCache@0x5645be7f3090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-19T01:04:12.927 INFO:journalctl@ceph.mon.c.smithi162.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-19T01:04:12.927 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: using public_addr v2:172.21.15.162:0/0 -> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: starting mon.c rank -1 at public addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] at bind addrs [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(???) e0 preinit fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(synchronizing).mds e1 new map 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout: e1 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout: legacy client fscid: -1 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout: 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout: No filesystems configured 2024-03-19T01:04:12.928 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-19T01:04:12.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-19T01:04:12.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-19T01:04:12.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-19T01:04:12.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-19T01:04:12.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-19T01:04:12.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-19T01:04:12.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T01:04:12.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T01:04:12.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-19T01:04:12.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:04:12.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:12.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:04:12.929 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:12.930 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.930 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.930 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.930 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.930 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:04:12.930 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:12.930 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:12.930 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: Updating smithi156:/etc/ceph/ceph.conf 2024-03-19T01:04:12.930 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: Deploying cephadm binary to smithi162 2024-03-19T01:04:12.930 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: Updating smithi156:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:12.930 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-03-19T01:04:12.930 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:12.930 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: Updating smithi156:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: Added host smithi162 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:12.931 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3112555393' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: Updating smithi162:/etc/ceph/ceph.conf 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: Updating smithi162:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: Updating smithi162:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3112555393' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T01:04:12.932 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi156:172.21.15.156=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: Saving service mon spec with placement smithi116:172.21.15.116=a;smithi156:172.21.15.156=b;smithi162:172.21.15.162=c;count:3 2024-03-19T01:04:12.933 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: Deploying daemon mon.c on smithi162 2024-03-19T01:04:12.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:12.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:12.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: from='client.? 172.21.15.162:0/1338075516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T01:04:12.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 ceph-mon[67178]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-19T01:04:12.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 bash[67159]: 8c9c409520b397746bd6310fadbae6ea8be3a5532c1ca84b396b0296047b898a 2024-03-19T01:04:12.934 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:12 smithi162 systemd[1]: Started Ceph mon.c for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:04:13.256 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-19T01:04:13.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T01:04:13.489 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:04:18.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.005+0000 7fc833699700 1 -- 172.21.15.162:0/2904704632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8180035c0 msgr2=0x7fc818005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:18.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.005+0000 7fc833699700 1 --2- 172.21.15.162:0/2904704632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8180035c0 0x7fc818005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc82c06b720 tx=0x7fc81c00feb0 comp rx=0 tx=0).stop 2024-03-19T01:04:18.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.005+0000 7fc833699700 1 -- 172.21.15.162:0/2904704632 shutdown_connections 2024-03-19T01:04:18.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.005+0000 7fc833699700 1 --2- 172.21.15.162:0/2904704632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8180035c0 0x7fc818005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:18.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.005+0000 7fc833699700 1 -- 172.21.15.162:0/2904704632 >> 172.21.15.162:0/2904704632 conn(0x7fc82c077300 msgr2=0x7fc82c077710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:18.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.005+0000 7fc833699700 1 -- 172.21.15.162:0/2904704632 shutdown_connections 2024-03-19T01:04:18.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.006+0000 7fc833699700 1 -- 172.21.15.162:0/2904704632 wait complete. 2024-03-19T01:04:18.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.006+0000 7fc833699700 1 Processor -- start 2024-03-19T01:04:18.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.006+0000 7fc833699700 1 -- start start 2024-03-19T01:04:18.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc833699700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8180035c0 0x7fc82c19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:18.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc833699700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc82c19db90 0x7fc82c197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:18.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc833699700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c108da0 con 0x7fc8180035c0 2024-03-19T01:04:18.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc833699700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc82c108c20 con 0x7fc82c19db90 2024-03-19T01:04:18.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc830c34700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc82c19db90 0x7fc82c197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:18.009 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc830c34700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc82c19db90 0x7fc82c197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:44428/0 (socket says 172.21.15.162:44428) 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc830c34700 1 -- 172.21.15.162:0/4071123865 learned_addr learned my addr 172.21.15.162:0/4071123865 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc831435700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8180035c0 0x7fc82c19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc830c34700 1 -- 172.21.15.162:0/4071123865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc82c19db90 msgr2=0x7fc82c197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc830c34700 1 -- 172.21.15.162:0/4071123865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc82c19db90 msgr2=0x7fc82c197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc830c34700 1 --2- 172.21.15.162:0/4071123865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc82c19db90 0x7fc82c197720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc830c34700 1 --2- 172.21.15.162:0/4071123865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc82c19db90 0x7fc82c197720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc831435700 1 -- 172.21.15.162:0/4071123865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc82c19db90 msgr2=0x7fc82c197720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc831435700 1 --2- 172.21.15.162:0/4071123865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc82c19db90 0x7fc82c197720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.007+0000 7fc831435700 1 -- 172.21.15.162:0/4071123865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc82c19e110 con 0x7fc8180035c0 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.008+0000 7fc831435700 1 --2- 172.21.15.162:0/4071123865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8180035c0 0x7fc82c19d650 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc81c010010 tx=0x7fc81c0055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.008+0000 7fc8227fc700 1 -- 172.21.15.162:0/4071123865 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc81c004b20 con 0x7fc8180035c0 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.008+0000 7fc8227fc700 1 -- 172.21.15.162:0/4071123865 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc81c013e40 con 0x7fc8180035c0 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.008+0000 7fc833699700 1 -- 172.21.15.162:0/4071123865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc82c197cc0 con 0x7fc8180035c0 2024-03-19T01:04:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.008+0000 7fc8227fc700 1 -- 172.21.15.162:0/4071123865 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fc81c011b60 con 0x7fc8180035c0 2024-03-19T01:04:18.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.008+0000 7fc833699700 1 -- 172.21.15.162:0/4071123865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc82c1982d0 con 0x7fc8180035c0 2024-03-19T01:04:18.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.009+0000 7fc8227fc700 1 -- 172.21.15.162:0/4071123865 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7fc81c023070 con 0x7fc8180035c0 2024-03-19T01:04:18.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.009+0000 7fc8227fc700 1 --2- 172.21.15.162:0/4071123865 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc810038db0 0x7fc81003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:18.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.009+0000 7fc833699700 1 -- 172.21.15.162:0/4071123865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc82c1985e0 con 0x7fc8180035c0 2024-03-19T01:04:18.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.009+0000 7fc8227fc700 1 -- 172.21.15.162:0/4071123865 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc81c020050 con 0x7fc8180035c0 2024-03-19T01:04:18.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.009+0000 7fc830c34700 1 --2- 172.21.15.162:0/4071123865 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc810038db0 0x7fc81003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:18.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.010+0000 7fc830c34700 1 --2- 172.21.15.162:0/4071123865 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc810038db0 0x7fc81003b270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc828009180 tx=0x7fc828008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:18.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.015+0000 7fc8227fc700 1 -- 172.21.15.162:0/4071123865 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc81c028020 con 0x7fc8180035c0 2024-03-19T01:04:18.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:04:18.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: mon.a calling monitor election 2024-03-19T01:04:18.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:18.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:18.119 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: mon.c calling monitor election 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: monmap epoch 2 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: last_changed 2024-03-19T01:04:12.694462+0000 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: created 2024-03-19T01:02:27.623031+0000 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: min_mon_release 19 (squid) 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: election_strategy: 1 2024-03-19T01:04:18.120 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T01:04:18.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T01:04:18.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: fsmap 2024-03-19T01:04:18.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T01:04:18.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: mgrmap e13: x(active, since 50s) 2024-03-19T01:04:18.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: overall HEALTH_OK 2024-03-19T01:04:18.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:18.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:18.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:18.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:18.121 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:18.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:04:18.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: mon.a calling monitor election 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: mon.c calling monitor election 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: monmap epoch 2 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: last_changed 2024-03-19T01:04:12.694462+0000 2024-03-19T01:04:18.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: created 2024-03-19T01:02:27.623031+0000 2024-03-19T01:04:18.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: min_mon_release 19 (squid) 2024-03-19T01:04:18.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: election_strategy: 1 2024-03-19T01:04:18.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T01:04:18.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T01:04:18.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: fsmap 2024-03-19T01:04:18.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T01:04:18.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: mgrmap e13: x(active, since 50s) 2024-03-19T01:04:18.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: overall HEALTH_OK 2024-03-19T01:04:18.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:18.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:18.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:18.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:18.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:18.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.294+0000 7fc833699700 1 -- 172.21.15.162:0/4071123865 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fc82c198c50 con 0x7fc8180035c0 2024-03-19T01:04:18.296 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.295+0000 7fc8227fc700 1 -- 172.21.15.162:0/4071123865 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7fc81c01d0c0 con 0x7fc8180035c0 2024-03-19T01:04:18.296 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T01:04:18.296 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":2,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","modified":"2024-03-19T01:04:12.694462Z","created":"2024-03-19T01:02:27.623031Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-19T01:04:18.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.297+0000 7fc833699700 1 -- 172.21.15.162:0/4071123865 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc810038db0 msgr2=0x7fc81003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:18.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.297+0000 7fc833699700 1 --2- 172.21.15.162:0/4071123865 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc810038db0 0x7fc81003b270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc828009180 tx=0x7fc828008040 comp rx=0 tx=0).stop 2024-03-19T01:04:18.298 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.297+0000 7fc833699700 1 -- 172.21.15.162:0/4071123865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8180035c0 msgr2=0x7fc82c19d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:18.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.297+0000 7fc833699700 1 --2- 172.21.15.162:0/4071123865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8180035c0 0x7fc82c19d650 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc81c010010 tx=0x7fc81c0055a0 comp rx=0 tx=0).stop 2024-03-19T01:04:18.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.297+0000 7fc833699700 1 -- 172.21.15.162:0/4071123865 shutdown_connections 2024-03-19T01:04:18.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.297+0000 7fc833699700 1 --2- 172.21.15.162:0/4071123865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8180035c0 0x7fc82c19d650 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:18.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.297+0000 7fc833699700 1 --2- 172.21.15.162:0/4071123865 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc810038db0 0x7fc81003b270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:18.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.297+0000 7fc833699700 1 --2- 172.21.15.162:0/4071123865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc82c19db90 0x7fc82c197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:18.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.297+0000 7fc833699700 1 -- 172.21.15.162:0/4071123865 >> 172.21.15.162:0/4071123865 conn(0x7fc82c077300 msgr2=0x7fc82c100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:18.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.297+0000 7fc833699700 1 -- 172.21.15.162:0/4071123865 shutdown_connections 2024-03-19T01:04:18.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:18.297+0000 7fc833699700 1 -- 172.21.15.162:0/4071123865 wait complete. 2024-03-19T01:04:18.299 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 2 2024-03-19T01:04:19.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:18 smithi162 ceph-mon[67178]: Deploying daemon mon.b on smithi156 2024-03-19T01:04:19.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:18 smithi162 ceph-mon[67178]: from='client.? 172.21.15.162:0/4071123865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T01:04:19.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:18 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:19.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:18 smithi116 ceph-mon[62701]: Deploying daemon mon.b on smithi156 2024-03-19T01:04:19.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:18 smithi116 ceph-mon[62701]: from='client.? 172.21.15.162:0/4071123865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T01:04:19.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:18 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:19.826 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-19T01:04:19.827 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-19T01:04:20.071 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:04:20.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:19 smithi162 ceph-mon[67178]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:20.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:19 smithi116 ceph-mon[62701]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:21.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:21.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:21.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:21.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:21.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:21.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:21.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi116:172.21.15.116=a;smithi156:172.21.15.156=b;smithi162:172.21.15.162=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:04:21.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: Saving service mon spec with placement smithi116:172.21.15.116=a;smithi156:172.21.15.156=b;smithi162:172.21.15.162=c;count:3 2024-03-19T01:04:21.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: Deploying daemon mon.c on smithi162 2024-03-19T01:04:21.457 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:21.457 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:21.457 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='client.? 172.21.15.162:0/1338075516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T01:04:21.457 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:04:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: mon.a calling monitor election 2024-03-19T01:04:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:21.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:21.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:21.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: mon.c calling monitor election 2024-03-19T01:04:21.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:21.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:21.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:21.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:21.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:21.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T01:04:21.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: monmap epoch 2 2024-03-19T01:04:21.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: last_changed 2024-03-19T01:04:12.694462+0000 2024-03-19T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: created 2024-03-19T01:02:27.623031+0000 2024-03-19T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: min_mon_release 19 (squid) 2024-03-19T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: election_strategy: 1 2024-03-19T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: fsmap 2024-03-19T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: mgrmap e13: x(active, since 50s) 2024-03-19T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: overall HEALTH_OK 2024-03-19T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:21.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:21.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:21.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:21.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: Deploying daemon mon.b on smithi156 2024-03-19T01:04:21.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='client.? 172.21.15.162:0/4071123865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-19T01:04:21.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:21.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:21.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 ceph-mon[66799]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-19T01:04:21.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 bash[66780]: bc093f4fc87f0f42ca4e5e1c5b753cd3a1ad4a6d7bb669acc01d00edc87be279 2024-03-19T01:04:21.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:21 smithi156 systemd[1]: Started Ceph mon.b for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:04:26.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: mon.c calling monitor election 2024-03-19T01:04:26.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:04:26.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: mon.a calling monitor election 2024-03-19T01:04:26.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:26.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:26.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:26.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:26.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:26.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:26.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:26.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:26.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:26.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T01:04:26.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: monmap epoch 3 2024-03-19T01:04:26.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:04:26.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: last_changed 2024-03-19T01:04:21.459833+0000 2024-03-19T01:04:26.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: created 2024-03-19T01:02:27.623031+0000 2024-03-19T01:04:26.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: min_mon_release 19 (squid) 2024-03-19T01:04:26.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: election_strategy: 1 2024-03-19T01:04:26.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T01:04:26.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T01:04:26.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-03-19T01:04:26.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: fsmap 2024-03-19T01:04:26.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T01:04:26.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: mgrmap e13: x(active, since 59s) 2024-03-19T01:04:26.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T01:04:26.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-19T01:04:26.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-19T01:04:26.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: mon.b (rank 2) addr [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] is down (out of quorum) 2024-03-19T01:04:26.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:26.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:26.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:26.902 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:26 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:26.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: mon.c calling monitor election 2024-03-19T01:04:26.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:04:26.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: mon.a calling monitor election 2024-03-19T01:04:26.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:26.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:26.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:26.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:26.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:26.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:26.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: monmap epoch 3 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: last_changed 2024-03-19T01:04:21.459833+0000 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: created 2024-03-19T01:02:27.623031+0000 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: min_mon_release 19 (squid) 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: election_strategy: 1 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: fsmap 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: mgrmap e13: x(active, since 59s) 2024-03-19T01:04:26.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T01:04:26.908 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-19T01:04:26.908 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-19T01:04:26.908 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: mon.b (rank 2) addr [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] is down (out of quorum) 2024-03-19T01:04:26.908 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:26.908 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:26.908 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:26.908 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:26 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:27.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.575+0000 7f1d107fe700 1 -- 172.21.15.162:0/1104381114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cf4006230 msgr2=0x7f1cf40086a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:27.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.575+0000 7f1d107fe700 1 --2- 172.21.15.162:0/1104381114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cf4006230 0x7f1cf40086a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1d0806a750 tx=0x7f1d0400fe90 comp rx=0 tx=0).stop 2024-03-19T01:04:27.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.575+0000 7f1d107fe700 1 -- 172.21.15.162:0/1104381114 shutdown_connections 2024-03-19T01:04:27.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.575+0000 7f1d107fe700 1 --2- 172.21.15.162:0/1104381114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cf4006230 0x7f1cf40086a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:27.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.575+0000 7f1d107fe700 1 -- 172.21.15.162:0/1104381114 >> 172.21.15.162:0/1104381114 conn(0x7f1d080fc820 msgr2=0x7f1d080fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:27.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.575+0000 7f1d107fe700 1 -- 172.21.15.162:0/1104381114 shutdown_connections 2024-03-19T01:04:27.576 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.575+0000 7f1d107fe700 1 -- 172.21.15.162:0/1104381114 wait complete. 2024-03-19T01:04:27.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d107fe700 1 Processor -- start 2024-03-19T01:04:27.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d107fe700 1 -- start start 2024-03-19T01:04:27.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d107fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf4006230 0x7f1d0819d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:27.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d107fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0819dd20 0x7f1d081979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:27.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d107fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d08197ee0 0x7f1d08198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:27.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d107fe700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1d08109e90 con 0x7f1d0819dd20 2024-03-19T01:04:27.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d107fe700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1d08109d10 con 0x7f1d08197ee0 2024-03-19T01:04:27.579 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d107fe700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0810a010 con 0x7f1cf4006230 2024-03-19T01:04:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d0e59a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf4006230 0x7f1d0819d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d0e59a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf4006230 0x7f1d0819d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:51246/0 (socket says 172.21.15.162:51246) 2024-03-19T01:04:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d0e59a700 1 -- 172.21.15.162:0/340206891 learned_addr learned my addr 172.21.15.162:0/340206891 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T01:04:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d0ed9b700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d08197ee0 0x7f1d08198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d0e59a700 1 -- 172.21.15.162:0/340206891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf4006230 msgr2=0x7f1d0819d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-19T01:04:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d0dd99700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0819dd20 0x7f1d081979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d0e59a700 1 -- 172.21.15.162:0/340206891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf4006230 msgr2=0x7f1d0819d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-19T01:04:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d0e59a700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf4006230 0x7f1d0819d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:04:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.576+0000 7f1d0e59a700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf4006230 0x7f1d0819d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:04:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1d0ed9b700 1 -- 172.21.15.162:0/340206891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d08197ee0 msgr2=0x7f1d08198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-19T01:04:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1d0ed9b700 1 -- 172.21.15.162:0/340206891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d08197ee0 msgr2=0x7f1d08198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-19T01:04:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1d0ed9b700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d08197ee0 0x7f1d08198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-19T01:04:27.580 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1d0ed9b700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d08197ee0 0x7f1d08198390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-19T01:04:27.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1d0dd99700 1 -- 172.21.15.162:0/340206891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf4006230 msgr2=0x7f1d0819d7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:04:27.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1d0dd99700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf4006230 0x7f1d0819d7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:27.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1d0dd99700 1 -- 172.21.15.162:0/340206891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d08197ee0 msgr2=0x7f1d08198390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:04:27.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1d0dd99700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d08197ee0 0x7f1d08198390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:27.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1d0dd99700 1 -- 172.21.15.162:0/340206891 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d081a46b0 con 0x7f1d0819dd20 2024-03-19T01:04:27.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1d0dd99700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0819dd20 0x7f1d081979a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800b550 tx=0x7f1cf800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:27.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1cff7fe700 1 -- 172.21.15.162:0/340206891 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8004230 con 0x7f1d0819dd20 2024-03-19T01:04:27.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1cff7fe700 1 -- 172.21.15.162:0/340206891 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cf80043d0 con 0x7f1d0819dd20 2024-03-19T01:04:27.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1d107fe700 1 -- 172.21.15.162:0/340206891 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d081a48e0 con 0x7f1d0819dd20 2024-03-19T01:04:27.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1cff7fe700 1 -- 172.21.15.162:0/340206891 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8011c20 con 0x7f1d0819dd20 2024-03-19T01:04:27.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.577+0000 7f1d107fe700 1 -- 172.21.15.162:0/340206891 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d081a4e70 con 0x7f1d0819dd20 2024-03-19T01:04:27.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.578+0000 7f1cff7fe700 1 -- 172.21.15.162:0/340206891 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f1cf801a400 con 0x7f1d0819dd20 2024-03-19T01:04:27.581 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.578+0000 7f1d107fe700 1 -- 172.21.15.162:0/340206891 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d08063940 con 0x7f1d0819dd20 2024-03-19T01:04:27.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.578+0000 7f1cff7fe700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1cec039450 0x7f1cec03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:27.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.578+0000 7f1cff7fe700 1 -- 172.21.15.162:0/340206891 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1cf804bb50 con 0x7f1d0819dd20 2024-03-19T01:04:27.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.579+0000 7f1d0e59a700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1cec039450 0x7f1cec03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:27.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.579+0000 7f1d0e59a700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1cec039450 0x7f1cec03b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400f550 tx=0x7f1d0400f4e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:27.584 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.584+0000 7f1cff7fe700 1 -- 172.21.15.162:0/340206891 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1cf801a840 con 0x7f1d0819dd20 2024-03-19T01:04:27.875 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.874+0000 7f1d107fe700 1 -- 172.21.15.162:0/340206891 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f1d0802dbb0 con 0x7f1d0819dd20 2024-03-19T01:04:27.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:27.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:27 smithi162 ceph-mon[67178]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:27.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:27.875 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:27.876 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.875+0000 7f1cff7fe700 1 -- 172.21.15.162:0/340206891 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7f1cf8003c80 con 0x7f1d0819dd20 2024-03-19T01:04:27.876 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-19T01:04:27.876 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":3,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","modified":"2024-03-19T01:04:21.459833Z","created":"2024-03-19T01:02:27.623031Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:3300","nonce":0},{"type":"v1","addr":"172.21.15.116:6789","nonce":0}]},"addr":"172.21.15.116:6789/0","public_addr":"172.21.15.116:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-19T01:04:27.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.877+0000 7f1d107fe700 1 -- 172.21.15.162:0/340206891 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1cec039450 msgr2=0x7f1cec03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:27.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.877+0000 7f1d107fe700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1cec039450 0x7f1cec03b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400f550 tx=0x7f1d0400f4e0 comp rx=0 tx=0).stop 2024-03-19T01:04:27.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.877+0000 7f1d107fe700 1 -- 172.21.15.162:0/340206891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0819dd20 msgr2=0x7f1d081979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:27.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.877+0000 7f1d107fe700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0819dd20 0x7f1d081979a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800b550 tx=0x7f1cf800b910 comp rx=0 tx=0).stop 2024-03-19T01:04:27.878 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.877+0000 7f1d107fe700 1 -- 172.21.15.162:0/340206891 shutdown_connections 2024-03-19T01:04:27.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.877+0000 7f1d107fe700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d08197ee0 0x7f1d08198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:27.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.877+0000 7f1d107fe700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1cec039450 0x7f1cec03b910 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:27.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.877+0000 7f1d107fe700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1cf4006230 0x7f1d0819d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:27.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.877+0000 7f1d107fe700 1 --2- 172.21.15.162:0/340206891 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d0819dd20 0x7f1d081979a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:27.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.877+0000 7f1d107fe700 1 -- 172.21.15.162:0/340206891 >> 172.21.15.162:0/340206891 conn(0x7f1d080fc820 msgr2=0x7f1d08103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:27.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.877+0000 7f1d107fe700 1 -- 172.21.15.162:0/340206891 shutdown_connections 2024-03-19T01:04:27.879 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:27.877+0000 7f1d107fe700 1 -- 172.21.15.162:0/340206891 wait complete. 2024-03-19T01:04:27.879 INFO:teuthology.orchestra.run.smithi162.stderr:dumped monmap epoch 3 2024-03-19T01:04:27.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:27.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:27 smithi116 ceph-mon[62701]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:27.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:27.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:28.633 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-19T01:04:28.633 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-19T01:04:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:04:28.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: mon.c calling monitor election 2024-03-19T01:04:28.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:04:28.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: mon.a calling monitor election 2024-03-19T01:04:28.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:28.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:04:28.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:28.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: monmap epoch 3 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: last_changed 2024-03-19T01:04:21.459833+0000 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: created 2024-03-19T01:02:27.623031+0000 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: min_mon_release 19 (squid) 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: election_strategy: 1 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-03-19T01:04:28.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: fsmap 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: mgrmap e13: x(active, since 59s) 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: mon.b (rank 2) addr [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] is down (out of quorum) 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:28.976 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: mon.b calling monitor election 2024-03-19T01:04:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: Updating smithi116:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: Updating smithi156:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: Updating smithi162:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: mon.b calling monitor election 2024-03-19T01:04:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: mon.c calling monitor election 2024-03-19T01:04:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: mon.a calling monitor election 2024-03-19T01:04:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-19T01:04:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: monmap epoch 3 2024-03-19T01:04:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:04:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: last_changed 2024-03-19T01:04:21.459833+0000 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: created 2024-03-19T01:02:27.623031+0000 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: min_mon_release 19 (squid) 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: election_strategy: 1 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: fsmap 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: mgrmap e13: x(active, since 61s) 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: Cluster is now healthy 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: overall HEALTH_OK 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:29.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:29.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:29.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: Reconfiguring mon.a (unknown last config time)... 2024-03-19T01:04:29.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:29.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T01:04:29.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:29.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: Reconfiguring daemon mon.a on smithi116 2024-03-19T01:04:29.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:29 smithi116 ceph-mon[62701]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: mon.b calling monitor election 2024-03-19T01:04:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: Updating smithi116:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: Updating smithi156:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: Updating smithi162:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: mon.b calling monitor election 2024-03-19T01:04:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: mon.c calling monitor election 2024-03-19T01:04:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: mon.a calling monitor election 2024-03-19T01:04:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-19T01:04:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: monmap epoch 3 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: last_changed 2024-03-19T01:04:21.459833+0000 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: created 2024-03-19T01:02:27.623031+0000 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: min_mon_release 19 (squid) 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: election_strategy: 1 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: fsmap 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: mgrmap e13: x(active, since 61s) 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: Cluster is now healthy 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: overall HEALTH_OK 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:29.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:29.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:29.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: Reconfiguring mon.a (unknown last config time)... 2024-03-19T01:04:29.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:29.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T01:04:29.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:29.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: Reconfiguring daemon mon.a on smithi116 2024-03-19T01:04:29.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:29 smithi156 ceph-mon[66799]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:29.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: mon.b calling monitor election 2024-03-19T01:04:29.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: Updating smithi116:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:29.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: Updating smithi156:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: Updating smithi162:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: mon.b calling monitor election 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: mon.c calling monitor election 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: mon.a calling monitor election 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: monmap epoch 3 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: last_changed 2024-03-19T01:04:21.459833+0000 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: created 2024-03-19T01:02:27.623031+0000 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: min_mon_release 19 (squid) 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: election_strategy: 1 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: 0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: 1: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.c 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: 2: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-03-19T01:04:29.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: fsmap 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: osdmap e4: 0 total, 0 up, 0 in 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: mgrmap e13: x(active, since 61s) 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: Cluster is now healthy 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: overall HEALTH_OK 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: Reconfiguring mon.a (unknown last config time)... 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: Reconfiguring daemon mon.a on smithi116 2024-03-19T01:04:29.901 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:29 smithi162 ceph-mon[67178]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:30.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:30.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:30 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:30.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:30 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:04:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.016+0000 7f17c9d68700 1 -- 172.21.15.116:0/862286887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c4101f50 msgr2=0x7f17c4102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.016+0000 7f17c9d68700 1 --2- 172.21.15.116:0/862286887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c4101f50 0x7f17c4102330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009910 tx=0x7f17bc009c20 comp rx=0 tx=0).stop 2024-03-19T01:04:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.016+0000 7f17c9d68700 1 -- 172.21.15.116:0/862286887 shutdown_connections 2024-03-19T01:04:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.016+0000 7f17c9d68700 1 --2- 172.21.15.116:0/862286887 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c4101f50 0x7f17c4102330 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.016+0000 7f17c9d68700 1 -- 172.21.15.116:0/862286887 >> 172.21.15.116:0/862286887 conn(0x7f17c40fd7e0 msgr2=0x7f17c40ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.016+0000 7f17c9d68700 1 -- 172.21.15.116:0/862286887 shutdown_connections 2024-03-19T01:04:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.016+0000 7f17c9d68700 1 -- 172.21.15.116:0/862286887 wait complete. 2024-03-19T01:04:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.016+0000 7f17c9d68700 1 Processor -- start 2024-03-19T01:04:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.016+0000 7f17c9d68700 1 -- start start 2024-03-19T01:04:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c9d68700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17c4101f50 0x7f17c419d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c9d68700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c419dc60 0x7f17c41978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c9d68700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17c4197e20 0x7f17c41982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c9d68700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17c410c2e0 con 0x7f17c419dc60 2024-03-19T01:04:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c9d68700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f17c410c160 con 0x7f17c4101f50 2024-03-19T01:04:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c9d68700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f17c410c460 con 0x7f17c4197e20 2024-03-19T01:04:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c419dc60 0x7f17c41978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c2ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c419dc60 0x7f17c41978e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43000/0 (socket says 172.21.15.116:43000) 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c2ffd700 1 -- 172.21.15.116:0/3741147115 learned_addr learned my addr 172.21.15.116:0/3741147115 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c37fe700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17c4101f50 0x7f17c419d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c3fff700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17c4197e20 0x7f17c41982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c2ffd700 1 -- 172.21.15.116:0/3741147115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17c4197e20 msgr2=0x7f17c41982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c2ffd700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17c4197e20 0x7f17c41982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c2ffd700 1 -- 172.21.15.116:0/3741147115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17c4101f50 msgr2=0x7f17c419d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c2ffd700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17c4101f50 0x7f17c419d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c2ffd700 1 -- 172.21.15.116:0/3741147115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c41a45f0 con 0x7f17c419dc60 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c3fff700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17c4197e20 0x7f17c41982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c37fe700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17c4101f50 0x7f17c419d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c2ffd700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c419dc60 0x7f17c41978e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f17b800b780 tx=0x7f17b800bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.017+0000 7f17c0ff9700 1 -- 172.21.15.116:0/3741147115 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8004230 con 0x7f17c419dc60 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.018+0000 7f17c0ff9700 1 -- 172.21.15.116:0/3741147115 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b80043d0 con 0x7f17c419dc60 2024-03-19T01:04:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.018+0000 7f17c9d68700 1 -- 172.21.15.116:0/3741147115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c41a4820 con 0x7f17c419dc60 2024-03-19T01:04:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.018+0000 7f17c0ff9700 1 -- 172.21.15.116:0/3741147115 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8011c60 con 0x7f17c419dc60 2024-03-19T01:04:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.018+0000 7f17c9d68700 1 -- 172.21.15.116:0/3741147115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c41a4db0 con 0x7f17c419dc60 2024-03-19T01:04:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.019+0000 7f17c0ff9700 1 -- 172.21.15.116:0/3741147115 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f17b801a400 con 0x7f17c419dc60 2024-03-19T01:04:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.019+0000 7f17c9d68700 1 -- 172.21.15.116:0/3741147115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17c4063940 con 0x7f17c419dc60 2024-03-19T01:04:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.019+0000 7f17c0ff9700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f17ac039450 0x7f17ac03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.019+0000 7f17c0ff9700 1 -- 172.21.15.116:0/3741147115 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f17b804bd20 con 0x7f17c419dc60 2024-03-19T01:04:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.019+0000 7f17c37fe700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f17ac039450 0x7f17ac03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.020+0000 7f17c37fe700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f17ac039450 0x7f17ac03b910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009750 tx=0x7f17bc00b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.025+0000 7f17c0ff9700 1 -- 172.21.15.116:0/3741147115 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f17b8020500 con 0x7f17c419dc60 2024-03-19T01:04:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.223+0000 7f17c9d68700 1 -- 172.21.15.116:0/3741147115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f17c41a50a0 con 0x7f17c419dc60 2024-03-19T01:04:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.223+0000 7f17c0ff9700 1 -- 172.21.15.116:0/3741147115 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7f17b80202a0 con 0x7f17c419dc60 2024-03-19T01:04:31.224 INFO:teuthology.orchestra.run.smithi116.stdout:# minimal ceph.conf for 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:04:31.224 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-03-19T01:04:31.224 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:04:31.225 INFO:teuthology.orchestra.run.smithi116.stdout: mon_host = [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] 2024-03-19T01:04:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.225+0000 7f17c9d68700 1 -- 172.21.15.116:0/3741147115 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f17ac039450 msgr2=0x7f17ac03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.225+0000 7f17c9d68700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f17ac039450 0x7f17ac03b910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009750 tx=0x7f17bc00b540 comp rx=0 tx=0).stop 2024-03-19T01:04:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.225+0000 7f17c9d68700 1 -- 172.21.15.116:0/3741147115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c419dc60 msgr2=0x7f17c41978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.225+0000 7f17c9d68700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c419dc60 0x7f17c41978e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f17b800b780 tx=0x7f17b800bb40 comp rx=0 tx=0).stop 2024-03-19T01:04:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.225+0000 7f17c9d68700 1 -- 172.21.15.116:0/3741147115 shutdown_connections 2024-03-19T01:04:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.225+0000 7f17c9d68700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f17ac039450 0x7f17ac03b910 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.225+0000 7f17c9d68700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17c4197e20 0x7f17c41982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.225+0000 7f17c9d68700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17c4101f50 0x7f17c419d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.225+0000 7f17c9d68700 1 --2- 172.21.15.116:0/3741147115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17c419dc60 0x7f17c41978e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.225+0000 7f17c9d68700 1 -- 172.21.15.116:0/3741147115 >> 172.21.15.116:0/3741147115 conn(0x7f17c40fd7e0 msgr2=0x7f17c4103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.225+0000 7f17c9d68700 1 -- 172.21.15.116:0/3741147115 shutdown_connections 2024-03-19T01:04:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:31.225+0000 7f17c9d68700 1 -- 172.21.15.116:0/3741147115 wait complete. 2024-03-19T01:04:31.690 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:31 smithi116 ceph-mon[62701]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:31.691 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:31 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3741147115' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:31.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:31 smithi156 ceph-mon[66799]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:31.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:31 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3741147115' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:31.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:31 smithi162 ceph-mon[67178]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:31.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:31 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3741147115' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:32.102 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-19T01:04:32.102 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:04:32.102 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T01:04:32.131 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:04:32.131 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T01:04:32.199 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-19T01:04:32.199 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T01:04:32.229 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-19T01:04:32.229 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T01:04:32.297 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T01:04:32.297 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.conf 2024-03-19T01:04:32.332 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T01:04:32.332 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-19T01:04:32.401 INFO:tasks.cephadm:Adding mgr.x on smithi116 2024-03-19T01:04:32.402 INFO:tasks.cephadm:Adding mgr.y on smithi156 2024-03-19T01:04:32.402 INFO:tasks.cephadm:Adding mgr.z on smithi162 2024-03-19T01:04:32.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi116=x;smithi156=y;smithi162=z' 2024-03-19T01:04:32.552 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:32.552 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:32.552 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:32 smithi156 ceph-mon[66799]: Reconfiguring mon.b (monmap changed)... 2024-03-19T01:04:32.552 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:32.552 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T01:04:32.553 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:32.553 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:32 smithi156 ceph-mon[66799]: Reconfiguring daemon mon.b on smithi156 2024-03-19T01:04:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:32 smithi162 ceph-mon[67178]: Reconfiguring mon.b (monmap changed)... 2024-03-19T01:04:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T01:04:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:32 smithi162 ceph-mon[67178]: Reconfiguring daemon mon.b on smithi156 2024-03-19T01:04:32.676 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:04:32.886 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:32.886 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:32.886 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:32 smithi116 ceph-mon[62701]: Reconfiguring mon.b (monmap changed)... 2024-03-19T01:04:32.886 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:32.886 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T01:04:32.886 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:32.886 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:32 smithi116 ceph-mon[62701]: Reconfiguring daemon mon.b on smithi156 2024-03-19T01:04:33.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:33 smithi162 ceph-mon[67178]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:33.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:33 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:33.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:33 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:33.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:33 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:33.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:33 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T01:04:33.797 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:33 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:33.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:33 smithi116 ceph-mon[62701]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:33.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:33 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:33.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:33 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:33.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:33 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:33.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:33 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T01:04:33.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:33 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:33.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:33 smithi156 ceph-mon[66799]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:33.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:33 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:33.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:33 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:33.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:33 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-19T01:04:33.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:33 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-19T01:04:33.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:33 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:34.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:34 smithi162 ceph-mon[67178]: Reconfiguring mon.c (monmap changed)... 2024-03-19T01:04:34.799 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:34 smithi162 ceph-mon[67178]: Reconfiguring daemon mon.c on smithi162 2024-03-19T01:04:34.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:34 smithi116 ceph-mon[62701]: Reconfiguring mon.c (monmap changed)... 2024-03-19T01:04:34.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:34 smithi116 ceph-mon[62701]: Reconfiguring daemon mon.c on smithi162 2024-03-19T01:04:34.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:34 smithi156 ceph-mon[66799]: Reconfiguring mon.c (monmap changed)... 2024-03-19T01:04:34.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:34 smithi156 ceph-mon[66799]: Reconfiguring daemon mon.c on smithi162 2024-03-19T01:04:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.006+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3940661515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6c4102440 msgr2=0x7ff6c41028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.006+0000 7ff6ca35d700 1 --2- 172.21.15.162:0/3940661515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6c4102440 0x7ff6c41028a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4009990 tx=0x7ff6b4009ca0 comp rx=0 tx=0).stop 2024-03-19T01:04:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.006+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3940661515 shutdown_connections 2024-03-19T01:04:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.006+0000 7ff6ca35d700 1 --2- 172.21.15.162:0/3940661515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c4102de0 0x7ff6c4109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.006+0000 7ff6ca35d700 1 --2- 172.21.15.162:0/3940661515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6c4101240 0x7ff6c4101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:35.007 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.006+0000 7ff6ca35d700 1 --2- 172.21.15.162:0/3940661515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6c4102440 0x7ff6c41028a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:35.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.006+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3940661515 >> 172.21.15.162:0/3940661515 conn(0x7ff6c40fc800 msgr2=0x7ff6c40fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:35.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.006+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3940661515 shutdown_connections 2024-03-19T01:04:35.008 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.006+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3940661515 wait complete. 2024-03-19T01:04:35.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.007+0000 7ff6ca35d700 1 Processor -- start 2024-03-19T01:04:35.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.007+0000 7ff6ca35d700 1 -- start start 2024-03-19T01:04:35.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.007+0000 7ff6ca35d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6c4101240 0x7ff6c41051d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:35.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.007+0000 7ff6ca35d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c4102440 0x7ff6c4103820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:35.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.007+0000 7ff6ca35d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6c4102de0 0x7ff6c4103d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:35.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.007+0000 7ff6ca35d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c410ce90 con 0x7ff6c4102440 2024-03-19T01:04:35.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.007+0000 7ff6ca35d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c410cd10 con 0x7ff6c4101240 2024-03-19T01:04:35.010 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.007+0000 7ff6ca35d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c410d010 con 0x7ff6c4102de0 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.007+0000 7ff6c88fa700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6c4102de0 0x7ff6c4103d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.007+0000 7ff6c88fa700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6c4102de0 0x7ff6c4103d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:47194/0 (socket says 172.21.15.162:47194) 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.007+0000 7ff6c88fa700 1 -- 172.21.15.162:0/3707415809 learned_addr learned my addr 172.21.15.162:0/3707415809 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.007+0000 7ff6c37fe700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c4102440 0x7ff6c4103820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.007+0000 7ff6c3fff700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6c4101240 0x7ff6c41051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.008+0000 7ff6c88fa700 1 -- 172.21.15.162:0/3707415809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6c4101240 msgr2=0x7ff6c41051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.008+0000 7ff6c88fa700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6c4101240 0x7ff6c41051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.008+0000 7ff6c88fa700 1 -- 172.21.15.162:0/3707415809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c4102440 msgr2=0x7ff6c4103820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.008+0000 7ff6c88fa700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c4102440 0x7ff6c4103820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.008+0000 7ff6c88fa700 1 -- 172.21.15.162:0/3707415809 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c4104610 con 0x7ff6c4102de0 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.008+0000 7ff6c37fe700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c4102440 0x7ff6c4103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.008+0000 7ff6c88fa700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6c4102de0 0x7ff6c4103d60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc00ef30 tx=0x7ff6bc00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.008+0000 7ff6c17fa700 1 -- 172.21.15.162:0/3707415809 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc0092e0 con 0x7ff6c4102de0 2024-03-19T01:04:35.011 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.008+0000 7ff6c17fa700 1 -- 172.21.15.162:0/3707415809 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc004bb0 con 0x7ff6c4102de0 2024-03-19T01:04:35.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.008+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3707415809 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6c41a26c0 con 0x7ff6c4102de0 2024-03-19T01:04:35.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.008+0000 7ff6c17fa700 1 -- 172.21.15.162:0/3707415809 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc004280 con 0x7ff6c4102de0 2024-03-19T01:04:35.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.008+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3707415809 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6c41a2c00 con 0x7ff6c4102de0 2024-03-19T01:04:35.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.009+0000 7ff6c17fa700 1 -- 172.21.15.162:0/3707415809 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7ff6bc01b030 con 0x7ff6c4102de0 2024-03-19T01:04:35.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.009+0000 7ff6c17fa700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff6ac039200 0x7ff6ac03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:35.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.009+0000 7ff6c17fa700 1 -- 172.21.15.162:0/3707415809 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff6bc04bfe0 con 0x7ff6c4102de0 2024-03-19T01:04:35.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.009+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3707415809 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6c41a2e60 con 0x7ff6c4102de0 2024-03-19T01:04:35.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.009+0000 7ff6c3fff700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff6ac039200 0x7ff6ac03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:35.012 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.010+0000 7ff6c3fff700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff6ac039200 0x7ff6ac03b6c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8005950 tx=0x7ff6b800a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:35.015 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.014+0000 7ff6c17fa700 1 -- 172.21.15.162:0/3707415809 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff6bc022020 con 0x7ff6c4102de0 2024-03-19T01:04:35.228 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.228+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3707415809 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi156=y;smithi162=z", "target": ["mon-mgr", ""]}) -- 0x7ff6c40fea20 con 0x7ff6ac039200 2024-03-19T01:04:35.234 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.233+0000 7ff6c17fa700 1 -- 172.21.15.162:0/3707415809 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff6c40fea20 con 0x7ff6ac039200 2024-03-19T01:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mgr update... 2024-03-19T01:04:35.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.235+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3707415809 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff6ac039200 msgr2=0x7ff6ac03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:35.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.235+0000 7ff6ca35d700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff6ac039200 0x7ff6ac03b6c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8005950 tx=0x7ff6b800a300 comp rx=0 tx=0).stop 2024-03-19T01:04:35.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.235+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3707415809 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6c4102de0 msgr2=0x7ff6c4103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:35.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.235+0000 7ff6ca35d700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6c4102de0 0x7ff6c4103d60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc00ef30 tx=0x7ff6bc00c3a0 comp rx=0 tx=0).stop 2024-03-19T01:04:35.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.235+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3707415809 shutdown_connections 2024-03-19T01:04:35.236 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.235+0000 7ff6ca35d700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff6ac039200 0x7ff6ac03b6c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:35.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.235+0000 7ff6ca35d700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff6c4102de0 0x7ff6c4103d60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:35.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.235+0000 7ff6ca35d700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6c4101240 0x7ff6c41051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:35.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.235+0000 7ff6ca35d700 1 --2- 172.21.15.162:0/3707415809 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6c4102440 0x7ff6c4103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:35.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.235+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3707415809 >> 172.21.15.162:0/3707415809 conn(0x7ff6c40fc800 msgr2=0x7ff6c40fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:35.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.235+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3707415809 shutdown_connections 2024-03-19T01:04:35.237 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:04:35.235+0000 7ff6ca35d700 1 -- 172.21.15.162:0/3707415809 wait complete. 2024-03-19T01:04:35.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:35.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:35.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:35.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:35.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:35.759 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:35.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:35.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T01:04:35.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T01:04:35.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T01:04:35.760 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:35 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:35.790 DEBUG:teuthology.orchestra.run.smithi156:mgr.y> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.y.service 2024-03-19T01:04:35.793 DEBUG:teuthology.orchestra.run.smithi162:mgr.z> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.z.service 2024-03-19T01:04:35.795 INFO:tasks.cephadm:Deploying OSDs... 2024-03-19T01:04:35.795 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:04:35.795 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T01:04:35.813 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T01:04:35.813 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-03-19T01:04:35.819 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T01:04:35.820 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T01:04:35.821 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:35 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:35.870 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T01:04:35.870 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:04:35.870 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-19T01:04:35.870 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:04:35.870 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T01:04:35.870 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-19 01:03:39.724407827 +0000 2024-03-19T01:04:35.870 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-19 00:39:14.236413051 +0000 2024-03-19T01:04:35.870 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-19 00:39:14.236413051 +0000 2024-03-19T01:04:35.870 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-19 00:39:14.236413051 +0000 2024-03-19T01:04:35.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T01:04:35.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T01:04:35.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:35 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-19T01:04:35.935 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000230007 s, 2.2 MB/s 2024-03-19T01:04:35.936 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T01:04:35.993 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-03-19T01:04:36.049 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T01:04:36.049 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:04:36.049 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-19T01:04:36.049 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:04:36.049 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T01:04:36.049 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-19 01:03:39.757407277 +0000 2024-03-19T01:04:36.049 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-19 00:39:14.563407666 +0000 2024-03-19T01:04:36.049 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-19 00:39:14.563407666 +0000 2024-03-19T01:04:36.049 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-19 00:39:14.563407666 +0000 2024-03-19T01:04:36.050 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T01:04:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-19T01:04:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-19T01:04:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000174528 s, 2.9 MB/s 2024-03-19T01:04:36.117 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T01:04:36.173 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-03-19T01:04:36.228 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T01:04:36.229 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:04:36.229 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-19T01:04:36.229 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:04:36.229 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T01:04:36.229 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-19 01:03:39.791406710 +0000 2024-03-19T01:04:36.229 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-19 00:39:14.905402033 +0000 2024-03-19T01:04:36.229 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-19 00:39:14.905402033 +0000 2024-03-19T01:04:36.229 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-19 00:39:14.905402033 +0000 2024-03-19T01:04:36.229 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T01:04:36.294 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-19T01:04:36.294 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-19T01:04:36.294 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000208631 s, 2.5 MB/s 2024-03-19T01:04:36.295 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T01:04:36.351 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-03-19T01:04:36.406 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T01:04:36.406 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:04:36.406 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-19T01:04:36.406 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:04:36.406 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T01:04:36.406 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-03-19 01:03:39.824406159 +0000 2024-03-19T01:04:36.407 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-03-19 00:39:15.234396614 +0000 2024-03-19T01:04:36.407 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-03-19 00:39:15.234396614 +0000 2024-03-19T01:04:36.407 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-03-19 00:39:15.234396614 +0000 2024-03-19T01:04:36.407 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T01:04:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-03-19T01:04:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-03-19T01:04:36.471 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000176447 s, 2.9 MB/s 2024-03-19T01:04:36.472 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T01:04:36.528 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-03-19T01:04:36.528 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T01:04:36.546 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T01:04:36.547 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2024-03-19T01:04:36.604 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T01:04:36.604 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:04:36.604 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-19T01:04:36.604 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:04:36.604 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T01:04:36.604 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-19 01:04:20.916857103 +0000 2024-03-19T01:04:36.604 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-19 00:33:04.360726826 +0000 2024-03-19T01:04:36.604 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-19 00:33:04.360726826 +0000 2024-03-19T01:04:36.604 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-19 00:33:04.360726826 +0000 2024-03-19T01:04:36.604 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T01:04:36.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:36 smithi116 ceph-mon[62701]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi156=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:04:36.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:36 smithi116 ceph-mon[62701]: Saving service mgr spec with placement smithi116=x;smithi156=y;smithi162=z;count:3 2024-03-19T01:04:36.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:36 smithi116 ceph-mon[62701]: Deploying daemon mgr.z on smithi162 2024-03-19T01:04:36.673 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-19T01:04:36.673 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-19T01:04:36.673 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000131325 s, 3.9 MB/s 2024-03-19T01:04:36.675 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T01:04:36.732 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2024-03-19T01:04:36.789 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T01:04:36.789 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:04:36.789 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-19T01:04:36.789 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:04:36.789 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T01:04:36.789 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-19 01:04:20.916857103 +0000 2024-03-19T01:04:36.789 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-19 00:33:04.704721904 +0000 2024-03-19T01:04:36.789 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-19 00:33:04.704721904 +0000 2024-03-19T01:04:36.790 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-19 00:33:04.704721904 +0000 2024-03-19T01:04:36.790 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T01:04:36.830 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:36 smithi162 ceph-mon[67178]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi156=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:04:36.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:36 smithi162 ceph-mon[67178]: Saving service mgr spec with placement smithi116=x;smithi156=y;smithi162=z;count:3 2024-03-19T01:04:36.831 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:36 smithi162 ceph-mon[67178]: Deploying daemon mgr.z on smithi162 2024-03-19T01:04:36.855 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:36 smithi156 ceph-mon[66799]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi116=x;smithi156=y;smithi162=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:04:36.855 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:36 smithi156 ceph-mon[66799]: Saving service mgr spec with placement smithi116=x;smithi156=y;smithi162=z;count:3 2024-03-19T01:04:36.855 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:36 smithi156 ceph-mon[66799]: Deploying daemon mgr.z on smithi162 2024-03-19T01:04:36.857 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-19T01:04:36.857 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-19T01:04:36.857 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000116951 s, 4.4 MB/s 2024-03-19T01:04:36.858 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T01:04:36.915 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2024-03-19T01:04:36.971 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T01:04:36.972 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:04:36.972 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-19T01:04:36.972 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:04:36.972 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T01:04:36.972 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-19 01:04:20.917857088 +0000 2024-03-19T01:04:36.972 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-19 00:33:05.018717411 +0000 2024-03-19T01:04:36.972 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-19 00:33:05.018717411 +0000 2024-03-19T01:04:36.972 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-19 00:33:05.018717411 +0000 2024-03-19T01:04:36.972 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T01:04:37.038 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-19T01:04:37.039 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-19T01:04:37.039 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000116293 s, 4.4 MB/s 2024-03-19T01:04:37.040 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T01:04:37.096 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2024-03-19T01:04:37.152 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T01:04:37.152 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:04:37.152 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-19T01:04:37.152 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:04:37.152 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T01:04:37.152 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-03-19 01:04:20.917857088 +0000 2024-03-19T01:04:37.152 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-03-19 00:33:05.336712861 +0000 2024-03-19T01:04:37.152 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-03-19 00:33:05.336712861 +0000 2024-03-19T01:04:37.152 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-03-19 00:33:05.336712861 +0000 2024-03-19T01:04:37.153 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T01:04:37.219 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-03-19T01:04:37.220 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-03-19T01:04:37.220 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000123509 s, 4.1 MB/s 2024-03-19T01:04:37.221 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T01:04:37.276 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-19T01:04:37.276 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-03-19T01:04:37.294 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-19T01:04:37.294 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-03-19T01:04:37.348 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-19T01:04:37.348 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:04:37.348 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-19T01:04:37.348 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:04:37.349 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T01:04:37.349 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-19 01:04:37.127543194 +0000 2024-03-19T01:04:37.349 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-19 00:50:30.961218624 +0000 2024-03-19T01:04:37.349 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-19 00:50:30.961218624 +0000 2024-03-19T01:04:37.349 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-19 00:50:30.961218624 +0000 2024-03-19T01:04:37.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-19T01:04:37.416 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-19T01:04:37.416 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-19T01:04:37.416 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000251117 s, 2.0 MB/s 2024-03-19T01:04:37.417 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-19T01:04:37.473 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-03-19T01:04:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-19T01:04:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:04:37.529 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-19T01:04:37.529 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:04:37.529 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T01:04:37.529 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-19 01:04:37.128543174 +0000 2024-03-19T01:04:37.529 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-19 00:50:31.306211594 +0000 2024-03-19T01:04:37.529 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-19 00:50:31.306211594 +0000 2024-03-19T01:04:37.529 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-19 00:50:31.306211594 +0000 2024-03-19T01:04:37.529 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-19T01:04:37.598 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-19T01:04:37.598 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-19T01:04:37.598 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000198206 s, 2.6 MB/s 2024-03-19T01:04:37.600 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-19T01:04:37.656 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-03-19T01:04:37.711 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:37 smithi162 ceph-mon[67178]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:37.720 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-19T01:04:37.720 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:04:37.720 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-19T01:04:37.720 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:04:37.720 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T01:04:37.720 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-19 01:04:37.128543174 +0000 2024-03-19T01:04:37.720 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-19 00:50:31.657204442 +0000 2024-03-19T01:04:37.720 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-19 00:50:31.657204442 +0000 2024-03-19T01:04:37.720 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-19 00:50:31.657204442 +0000 2024-03-19T01:04:37.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-19T01:04:37.791 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-19T01:04:37.791 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-19T01:04:37.791 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000185008 s, 2.8 MB/s 2024-03-19T01:04:37.792 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-19T01:04:37.853 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-03-19T01:04:37.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:37 smithi116 ceph-mon[62701]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-19T01:04:37.908 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-19T01:04:37.908 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 717 Links: 1 2024-03-19T01:04:37.908 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-19T01:04:37.908 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-19T01:04:37.908 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-19 01:04:37.128543174 +0000 2024-03-19T01:04:37.909 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-19 00:50:31.998197493 +0000 2024-03-19T01:04:37.909 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-19 00:50:31.998197493 +0000 2024-03-19T01:04:37.909 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-19 00:50:31.998197493 +0000 2024-03-19T01:04:37.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-19T01:04:37.974 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:37 smithi162 systemd[1]: Started Ceph mgr.z for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:04:37.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:37 smithi156 ceph-mon[66799]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:37.976 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-19T01:04:37.977 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-19T01:04:37.977 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000171812 s, 3.0 MB/s 2024-03-19T01:04:37.978 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-19T01:04:38.036 INFO:tasks.cephadm:Deploying osd.0 on smithi116 with /dev/vg_nvme/lv_4... 2024-03-19T01:04:38.036 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T01:04:38.281 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:04:38.649 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:38 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:38.282+0000 7f81c598e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T01:04:38.649 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:38 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:38.377+0000 7f81c598e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T01:04:39.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:38 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:39.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:38 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:39.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:38 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:39.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:38 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T01:04:39.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:38 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T01:04:39.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:38 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T01:04:39.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:38 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:39.072 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:38 smithi116 ceph-mon[62701]: Deploying daemon mgr.y on smithi156 2024-03-19T01:04:39.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:38 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:39.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:38 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:39.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:38 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:39.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:38 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T01:04:39.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:38 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T01:04:39.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:38 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T01:04:39.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:38 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:39.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:38 smithi162 ceph-mon[67178]: Deploying daemon mgr.y on smithi156 2024-03-19T01:04:39.150 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:39 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:39.021+0000 7f81c598e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T01:04:39.150 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:39 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:39.115+0000 7f81c598e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T01:04:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:38 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:38 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:38 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:38 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T01:04:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:38 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-19T01:04:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:38 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T01:04:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:38 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:38 smithi156 ceph-mon[66799]: Deploying daemon mgr.y on smithi156 2024-03-19T01:04:39.649 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:39 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:39.256+0000 7f81c598e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T01:04:39.649 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:39 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:39.350+0000 7f81c598e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T01:04:39.649 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:39 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:39.431+0000 7f81c598e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T01:04:39.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:39 smithi156 ceph-mon[66799]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:39.984 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:39 smithi162 ceph-mon[67178]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:39.984 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:39 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:39.698+0000 7f81c598e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T01:04:39.985 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:39 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:39.806+0000 7f81c598e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T01:04:39.985 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:39 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:39.890+0000 7f81c598e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T01:04:39.985 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:39 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:39.983+0000 7f81c598e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T01:04:40.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:39 smithi116 ceph-mon[62701]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:40.167 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T01:04:40.167 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T01:04:40.167 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-19T01:04:40.167 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-19T01:04:40.167 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115667 s, 907 MB/s 2024-03-19T01:04:40.167 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-19T01:04:40.352 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:40 smithi156 ceph-mgr[68224]: -- 172.21.15.156:0/2945068474 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56203ac645a0 con 0x56203ac38800 2024-03-19T01:04:40.352 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:40 smithi156 bash[68204]: a73594c62eb8b699027bf61b2853a17a9444e6ac4264069c4a22fc486f80744f 2024-03-19T01:04:40.399 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:40 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:40.159+0000 7f81c598e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T01:04:40.622 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:40 smithi156 systemd[1]: Started Ceph mgr.y for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:04:40.778 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_4 2024-03-19T01:04:40.946 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:40 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:40.847+0000 7faa0566a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-19T01:04:40.946 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:40 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:40.945+0000 7faa0566a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-19T01:04:41.011 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:04:41.430 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:41.175+0000 7f81c598e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T01:04:41.430 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:41.312+0000 7f81c598e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T01:04:41.724 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:41.636+0000 7faa0566a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-19T01:04:41.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:41 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:41.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:41 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:41.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:41 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:41.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:41 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:41.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:41 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:41.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:41 smithi156 ceph-mon[66799]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:41.749 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:41 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:41.749 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:41 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:41.749 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:41 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:41.749 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:41 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:41.749 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:41 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:41.750 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:41 smithi116 ceph-mon[62701]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:41.755 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:41.656+0000 7f81c598e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T01:04:41.755 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:41 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:41.755 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:41 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:41.755 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:41 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:41.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:41 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:41.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:41 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:41.756 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:41 smithi162 ceph-mon[67178]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:42.062 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:41.729+0000 7faa0566a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-19T01:04:42.062 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:41.875+0000 7faa0566a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-19T01:04:42.062 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:41.975+0000 7faa0566a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-19T01:04:42.062 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:42.061+0000 7faa0566a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-19T01:04:42.137 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:41.754+0000 7f81c598e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T01:04:42.137 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:41.951+0000 7f81c598e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:04:42.138 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:42.136+0000 7f81c598e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T01:04:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.542+0000 7f11a5bd3700 1 -- 172.21.15.116:0/4267019945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11a0105730 msgr2=0x7f11a0107b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.542+0000 7f11a5bd3700 1 --2- 172.21.15.116:0/4267019945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11a0105730 0x7f11a0107b20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1194009990 tx=0x7f1194009ca0 comp rx=0 tx=0).stop 2024-03-19T01:04:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.542+0000 7f11a5bd3700 1 -- 172.21.15.116:0/4267019945 shutdown_connections 2024-03-19T01:04:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.542+0000 7f11a5bd3700 1 --2- 172.21.15.116:0/4267019945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11a0108060 0x7f11a010a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.542+0000 7f11a5bd3700 1 --2- 172.21.15.116:0/4267019945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11a0102e00 0x7f11a01051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.542+0000 7f11a5bd3700 1 --2- 172.21.15.116:0/4267019945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11a0105730 0x7f11a0107b20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:42.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.542+0000 7f11a5bd3700 1 -- 172.21.15.116:0/4267019945 >> 172.21.15.116:0/4267019945 conn(0x7f11a00fc830 msgr2=0x7f11a00fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:04:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.542+0000 7f11a5bd3700 1 -- 172.21.15.116:0/4267019945 shutdown_connections 2024-03-19T01:04:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.542+0000 7f11a5bd3700 1 -- 172.21.15.116:0/4267019945 wait complete. 2024-03-19T01:04:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.542+0000 7f11a5bd3700 1 Processor -- start 2024-03-19T01:04:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f11a5bd3700 1 -- start start 2024-03-19T01:04:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f11a5bd3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11a0102e00 0x7f11a0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f11a5bd3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11a0105730 0x7f11a0198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f11a5bd3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11a0108060 0x7f11a019fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f11a5bd3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11a010df80 con 0x7f11a0105730 2024-03-19T01:04:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f11a5bd3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f11a010de00 con 0x7f11a0108060 2024-03-19T01:04:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f11a5bd3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11a010e100 con 0x7f11a0102e00 2024-03-19T01:04:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f119effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11a0105730 0x7f11a0198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f119effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11a0105730 0x7f11a0198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45786/0 (socket says 172.21.15.116:45786) 2024-03-19T01:04:42.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f119effd700 1 -- 172.21.15.116:0/2812162886 learned_addr learned my addr 172.21.15.116:0/2812162886 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:04:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f119f7fe700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11a0102e00 0x7f11a0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f119ffff700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11a0108060 0x7f11a019fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f119effd700 1 -- 172.21.15.116:0/2812162886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11a0102e00 msgr2=0x7f11a0198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f119effd700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11a0102e00 0x7f11a0198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f119effd700 1 -- 172.21.15.116:0/2812162886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11a0108060 msgr2=0x7f11a019fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:04:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f119effd700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11a0108060 0x7f11a019fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:04:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f119effd700 1 -- 172.21.15.116:0/2812162886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a01a01b0 con 0x7f11a0105730 2024-03-19T01:04:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f119effd700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11a0105730 0x7f11a0198a90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1194005f50 tx=0x7f119400bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.543+0000 7f119cff9700 1 -- 172.21.15.116:0/2812162886 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1194026070 con 0x7f11a0105730 2024-03-19T01:04:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.544+0000 7f119cff9700 1 -- 172.21.15.116:0/2812162886 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f119400f460 con 0x7f11a0105730 2024-03-19T01:04:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.544+0000 7f119cff9700 1 -- 172.21.15.116:0/2812162886 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1194021650 con 0x7f11a0105730 2024-03-19T01:04:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.544+0000 7f11a5bd3700 1 -- 172.21.15.116:0/2812162886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a01a0440 con 0x7f11a0105730 2024-03-19T01:04:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.544+0000 7f11a5bd3700 1 -- 172.21.15.116:0/2812162886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a01a07e0 con 0x7f11a0105730 2024-03-19T01:04:42.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.545+0000 7f119cff9700 1 -- 172.21.15.116:0/2812162886 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 13) ==== 46163+0+0 (secure 0 0 0) 0x7f1194029070 con 0x7f11a0105730 2024-03-19T01:04:42.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.545+0000 7f11a5bd3700 1 -- 172.21.15.116:0/2812162886 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a0068fd0 con 0x7f11a0105730 2024-03-19T01:04:42.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.545+0000 7f119cff9700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1188039200 0x7f118803b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:04:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.545+0000 7f119cff9700 1 -- 172.21.15.116:0/2812162886 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f119404d0a0 con 0x7f11a0105730 2024-03-19T01:04:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.545+0000 7f119f7fe700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1188039200 0x7f118803b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:04:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.545+0000 7f119f7fe700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1188039200 0x7f118803b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1190009c60 tx=0x7f1190009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:04:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.550+0000 7f119cff9700 1 -- 172.21.15.116:0/2812162886 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1194013070 con 0x7f11a0105730 2024-03-19T01:04:42.623 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:42.334+0000 7faa0566a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-19T01:04:42.624 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:42.444+0000 7faa0566a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-19T01:04:42.624 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:42.528+0000 7faa0566a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-19T01:04:42.624 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:42.622+0000 7faa0566a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-19T01:04:42.677 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:42.416+0000 7f81c598e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:04:42.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:42.769+0000 7f11a5bd3700 1 -- 172.21.15.116:0/2812162886 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f11a0062750 con 0x7f1188039200 2024-03-19T01:04:42.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:42.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:42.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:42.952 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:42.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:42.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:42.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:42.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: Reconfiguring mgr.x (unknown last config time)... 2024-03-19T01:04:42.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T01:04:42.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T01:04:42.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:42.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: Reconfiguring daemon mgr.x on smithi116 2024-03-19T01:04:42.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:42.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:04:42.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:04:42.953 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:42.973 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:42.813+0000 7faa0566a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-19T01:04:42.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:42.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:42.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:42.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:42.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:42.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:42.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:42.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: Reconfiguring mgr.x (unknown last config time)... 2024-03-19T01:04:42.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T01:04:42.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T01:04:42.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:42.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: Reconfiguring daemon mgr.x on smithi116 2024-03-19T01:04:42.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:42.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:04:42.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:04:42.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:43.149 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:42.676+0000 7f81c598e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T01:04:43.149 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:42.908+0000 7f81c598e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T01:04:43.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:43.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:43.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:43.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:43.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:43.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:43.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:43.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: Reconfiguring mgr.x (unknown last config time)... 2024-03-19T01:04:43.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-19T01:04:43.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-19T01:04:43.151 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:43.151 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: Reconfiguring daemon mgr.x on smithi116 2024-03-19T01:04:43.151 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:43.151 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:04:43.151 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:04:43.151 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:43.916 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:43 smithi116 ceph-mon[62701]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:04:43.916 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:43 smithi116 ceph-mon[62701]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:43.916 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:43.916 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:43.916 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:43.917 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:43.917 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:43.917 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:44.149 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:43 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:43.704+0000 7f81c598e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T01:04:44.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:43 smithi162 ceph-mon[67178]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:04:44.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:43 smithi162 ceph-mon[67178]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:44.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:44.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:44.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:44.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:44.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:44.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:44.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:43 smithi156 ceph-mon[66799]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:04:44.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:43 smithi156 ceph-mon[66799]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:44.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:44.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:44.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:44.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:44.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:04:44.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:44.224 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:43 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:43.782+0000 7faa0566a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-19T01:04:44.225 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:43 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:43.916+0000 7faa0566a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-19T01:04:44.557 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:44 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:44.271+0000 7faa0566a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-19T01:04:44.557 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:44 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:44.361+0000 7faa0566a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-19T01:04:44.557 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:44 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:44.556+0000 7faa0566a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:04:44.589 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:44 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:44.279+0000 7f81c598e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T01:04:44.589 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:44 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:44.375+0000 7f81c598e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T01:04:44.590 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:04:44 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z[68935]: 2024-03-19T01:04:44.588+0000 7f81c598e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T01:04:44.842 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:44 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:44.740+0000 7faa0566a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-19T01:04:44.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:44.848+0000 7f119cff9700 1 -- 172.21.15.116:0/2812162886 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 14) ==== 91699+0+0 (secure 0 0 0) 0x7f1194009d10 con 0x7f11a0105730 2024-03-19T01:04:44.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:44 smithi116 ceph-mon[62701]: Standby manager daemon z started 2024-03-19T01:04:44.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:44 smithi116 ceph-mon[62701]: from='mgr.? 172.21.15.162:0/4152721099' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T01:04:44.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:44 smithi116 ceph-mon[62701]: from='mgr.? 172.21.15.162:0/4152721099' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T01:04:44.956 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:44 smithi116 ceph-mon[62701]: from='mgr.? 172.21.15.162:0/4152721099' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T01:04:44.957 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:44 smithi116 ceph-mon[62701]: from='mgr.? 172.21.15.162:0/4152721099' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T01:04:45.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:44 smithi162 ceph-mon[67178]: Standby manager daemon z started 2024-03-19T01:04:45.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:44 smithi162 ceph-mon[67178]: from='mgr.? 172.21.15.162:0/4152721099' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T01:04:45.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:44 smithi162 ceph-mon[67178]: from='mgr.? 172.21.15.162:0/4152721099' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T01:04:45.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:44 smithi162 ceph-mon[67178]: from='mgr.? 172.21.15.162:0/4152721099' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T01:04:45.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:44 smithi162 ceph-mon[67178]: from='mgr.? 172.21.15.162:0/4152721099' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T01:04:45.223 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:45 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:45.023+0000 7faa0566a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-19T01:04:45.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:44 smithi156 ceph-mon[66799]: Standby manager daemon z started 2024-03-19T01:04:45.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:44 smithi156 ceph-mon[66799]: from='mgr.? 172.21.15.162:0/4152721099' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-19T01:04:45.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:44 smithi156 ceph-mon[66799]: from='mgr.? 172.21.15.162:0/4152721099' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T01:04:45.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:44 smithi156 ceph-mon[66799]: from='mgr.? 172.21.15.162:0/4152721099' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-19T01:04:45.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:44 smithi156 ceph-mon[66799]: from='mgr.? 172.21.15.162:0/4152721099' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T01:04:45.723 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:45 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:45.294+0000 7faa0566a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-19T01:04:45.724 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:45 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:45.526+0000 7faa0566a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-19T01:04:46.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:45 smithi162 ceph-mon[67178]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-19T01:04:46.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T01:04:46.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:45 smithi162 ceph-mon[67178]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:46.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:45 smithi116 ceph-mon[62701]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-19T01:04:46.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T01:04:46.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:45 smithi116 ceph-mon[62701]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:46.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:45 smithi156 ceph-mon[66799]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-19T01:04:46.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-19T01:04:46.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:45 smithi156 ceph-mon[66799]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:46.723 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:46 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:46.324+0000 7faa0566a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-19T01:04:47.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:46 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2151713732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95728d5b-46c8-41c0-830a-7602d8b251fe"}]: dispatch 2024-03-19T01:04:47.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:46 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2151713732' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95728d5b-46c8-41c0-830a-7602d8b251fe"}]': finished 2024-03-19T01:04:47.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:46 smithi162 ceph-mon[67178]: osdmap e5: 1 total, 0 up, 1 in 2024-03-19T01:04:47.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:46 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:04:47.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:46 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/850124599' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:04:47.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:46 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2151713732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95728d5b-46c8-41c0-830a-7602d8b251fe"}]: dispatch 2024-03-19T01:04:47.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:46 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2151713732' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95728d5b-46c8-41c0-830a-7602d8b251fe"}]': finished 2024-03-19T01:04:47.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:46 smithi116 ceph-mon[62701]: osdmap e5: 1 total, 0 up, 1 in 2024-03-19T01:04:47.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:46 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:04:47.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:46 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/850124599' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:04:47.203 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:46 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2151713732' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95728d5b-46c8-41c0-830a-7602d8b251fe"}]: dispatch 2024-03-19T01:04:47.203 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:46 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2151713732' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95728d5b-46c8-41c0-830a-7602d8b251fe"}]': finished 2024-03-19T01:04:47.204 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:46 smithi156 ceph-mon[66799]: osdmap e5: 1 total, 0 up, 1 in 2024-03-19T01:04:47.204 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:46 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:04:47.204 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:46 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/850124599' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:04:47.204 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:46 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:46.894+0000 7faa0566a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-19T01:04:47.204 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:46 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:46.989+0000 7faa0566a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-19T01:04:47.204 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:04:47 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y[68219]: 2024-03-19T01:04:47.202+0000 7faa0566a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-19T01:04:47.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:04:47.864+0000 7f119cff9700 1 -- 172.21.15.116:0/2812162886 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7f119400f680 con 0x7f11a0105730 2024-03-19T01:04:48.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:47 smithi162 ceph-mon[67178]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:48.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:47 smithi162 ceph-mon[67178]: Standby manager daemon y started 2024-03-19T01:04:48.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:47 smithi162 ceph-mon[67178]: from='mgr.? 172.21.15.156:0/1298648493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T01:04:48.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:47 smithi162 ceph-mon[67178]: from='mgr.? 172.21.15.156:0/1298648493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T01:04:48.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:47 smithi162 ceph-mon[67178]: from='mgr.? 172.21.15.156:0/1298648493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T01:04:48.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:47 smithi162 ceph-mon[67178]: from='mgr.? 172.21.15.156:0/1298648493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T01:04:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:47 smithi116 ceph-mon[62701]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:47 smithi116 ceph-mon[62701]: Standby manager daemon y started 2024-03-19T01:04:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:47 smithi116 ceph-mon[62701]: from='mgr.? 172.21.15.156:0/1298648493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T01:04:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:47 smithi116 ceph-mon[62701]: from='mgr.? 172.21.15.156:0/1298648493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T01:04:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:47 smithi116 ceph-mon[62701]: from='mgr.? 172.21.15.156:0/1298648493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T01:04:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:47 smithi116 ceph-mon[62701]: from='mgr.? 172.21.15.156:0/1298648493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T01:04:48.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:47 smithi156 ceph-mon[66799]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:48.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:47 smithi156 ceph-mon[66799]: Standby manager daemon y started 2024-03-19T01:04:48.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:47 smithi156 ceph-mon[66799]: from='mgr.? 172.21.15.156:0/1298648493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-19T01:04:48.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:47 smithi156 ceph-mon[66799]: from='mgr.? 172.21.15.156:0/1298648493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-19T01:04:48.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:47 smithi156 ceph-mon[66799]: from='mgr.? 172.21.15.156:0/1298648493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-19T01:04:48.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:47 smithi156 ceph-mon[66799]: from='mgr.? 172.21.15.156:0/1298648493' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-19T01:04:49.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:48 smithi162 ceph-mon[67178]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-19T01:04:49.125 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:48 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T01:04:49.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:48 smithi116 ceph-mon[62701]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-19T01:04:49.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:48 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T01:04:49.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:48 smithi156 ceph-mon[66799]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-19T01:04:49.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:48 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-19T01:04:50.098 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:49 smithi116 ceph-mon[62701]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:50.148 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:49 smithi162 ceph-mon[67178]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:50.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:49 smithi156 ceph-mon[66799]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:52.340 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:52 smithi116 ceph-mon[62701]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:52.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:52 smithi162 ceph-mon[67178]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:52.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:52 smithi156 ceph-mon[66799]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:53.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T01:04:53.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:53.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T01:04:53.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:53.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-19T01:04:53.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:04:54.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:54 smithi116 ceph-mon[62701]: Deploying daemon osd.0 on smithi116 2024-03-19T01:04:54.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:54 smithi116 ceph-mon[62701]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:54 smithi162 ceph-mon[67178]: Deploying daemon osd.0 on smithi116 2024-03-19T01:04:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:54 smithi162 ceph-mon[67178]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:54 smithi156 ceph-mon[66799]: Deploying daemon osd.0 on smithi116 2024-03-19T01:04:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:54 smithi156 ceph-mon[66799]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:56.310 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:56 smithi116 ceph-mon[62701]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:56.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:56 smithi162 ceph-mon[67178]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:56.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:56 smithi156 ceph-mon[66799]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:58.257 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:58 smithi116 ceph-mon[62701]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:58.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:58 smithi162 ceph-mon[67178]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:58.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:58 smithi156 ceph-mon[66799]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:04:59 smithi162 ceph-mon[67178]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:04:59 smithi116 ceph-mon[62701]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:04:59.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:59.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:04:59.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:04:59.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:04:59 smithi156 ceph-mon[66799]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:05:02.078 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 0 on host 'smithi116' 2024-03-19T01:05:02.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:02.077+0000 7f119cff9700 1 -- 172.21.15.116:0/2812162886 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f11a0062750 con 0x7f1188039200 2024-03-19T01:05:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:02.079+0000 7f11a5bd3700 1 -- 172.21.15.116:0/2812162886 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1188039200 msgr2=0x7f118803b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:02.079+0000 7f11a5bd3700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1188039200 0x7f118803b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1190009c60 tx=0x7f1190009260 comp rx=0 tx=0).stop 2024-03-19T01:05:02.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:02.079+0000 7f11a5bd3700 1 -- 172.21.15.116:0/2812162886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11a0105730 msgr2=0x7f11a0198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:02.079+0000 7f11a5bd3700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11a0105730 0x7f11a0198a90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1194005f50 tx=0x7f119400bbb0 comp rx=0 tx=0).stop 2024-03-19T01:05:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:02.080+0000 7f11a5bd3700 1 -- 172.21.15.116:0/2812162886 shutdown_connections 2024-03-19T01:05:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:02.080+0000 7f11a5bd3700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11a0108060 0x7f11a019fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:02.080+0000 7f11a5bd3700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1188039200 0x7f118803b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:02.080+0000 7f11a5bd3700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11a0102e00 0x7f11a0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:02.080+0000 7f11a5bd3700 1 --2- 172.21.15.116:0/2812162886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11a0105730 0x7f11a0198a90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:02.080+0000 7f11a5bd3700 1 -- 172.21.15.116:0/2812162886 >> 172.21.15.116:0/2812162886 conn(0x7f11a00fc830 msgr2=0x7f11a0103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:05:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:02.080+0000 7f11a5bd3700 1 -- 172.21.15.116:0/2812162886 shutdown_connections 2024-03-19T01:05:02.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:02.080+0000 7f11a5bd3700 1 -- 172.21.15.116:0/2812162886 wait complete. 2024-03-19T01:05:02.377 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:02.377 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:01 smithi116 ceph-mon[62701]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:05:02.377 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:02.377 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:02.377 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:02.377 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:02.398 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:02.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:01 smithi162 ceph-mon[67178]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:05:02.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:02.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:02.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:02.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:02.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:02.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:01 smithi156 ceph-mon[66799]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:05:02.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:02.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:02.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:02.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:02.652 DEBUG:teuthology.orchestra.run.smithi116:osd.0> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.0.service 2024-03-19T01:05:02.654 INFO:tasks.cephadm:Deploying osd.1 on smithi156 with /dev/vg_nvme/lv_4... 2024-03-19T01:05:02.655 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T01:05:02.899 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T01:05:03.290 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:03 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:03.290 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:03 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:03.290 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:03 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:03.290 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:03 smithi156 ceph-mon[66799]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T01:05:03.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:03 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:03.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:03 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:03.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:03 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:03.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:03 smithi162 ceph-mon[67178]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T01:05:03.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:03 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:03.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:03 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:03.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:03 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:03.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:03 smithi116 ceph-mon[62701]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-19T01:05:04.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:04 smithi116 ceph-mon[62701]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:05:04.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:04 smithi116 ceph-mon[62701]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T01:05:04.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:04 smithi116 ceph-mon[62701]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T01:05:04.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:04 smithi116 ceph-mon[62701]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T01:05:04.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:04.180 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:04 smithi156 ceph-mon[66799]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:05:04.180 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:04 smithi156 ceph-mon[66799]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T01:05:04.180 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:04 smithi156 ceph-mon[66799]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T01:05:04.180 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:04 smithi156 ceph-mon[66799]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T01:05:04.180 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:04 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:04 smithi162 ceph-mon[67178]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:05:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:04 smithi162 ceph-mon[67178]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-19T01:05:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:04 smithi162 ceph-mon[67178]: osdmap e6: 1 total, 0 up, 1 in 2024-03-19T01:05:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:04 smithi162 ceph-mon[67178]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T01:05:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:04 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:04.728 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T01:05:04.728 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T01:05:04.728 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-03-19T01:05:04.728 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-03-19T01:05:04.728 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00851002 s, 1.2 GB/s 2024-03-19T01:05:04.728 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 2024-03-19T01:05:04.729 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-03-19T01:05:04.905 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:05:04 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0[77431]: 2024-03-19T01:05:04.488+0000 7f5035f57700 -1 osd.0 0 waiting for initial osdmap 2024-03-19T01:05:04.906 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:05:04 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0[77431]: 2024-03-19T01:05:04.492+0000 7f502d541700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:05:05.253 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi156:vg_nvme/lv_4 2024-03-19T01:05:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:05 smithi162 ceph-mon[67178]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T01:05:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:05 smithi162 ceph-mon[67178]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T01:05:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:05 smithi162 ceph-mon[67178]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' 2024-03-19T01:05:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:05 smithi116 ceph-mon[62701]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T01:05:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:05 smithi116 ceph-mon[62701]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T01:05:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:05 smithi116 ceph-mon[62701]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' 2024-03-19T01:05:05.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:05 smithi156 ceph-mon[66799]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T01:05:05.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:05 smithi156 ceph-mon[66799]: osdmap e7: 1 total, 0 up, 1 in 2024-03-19T01:05:05.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:05.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:05.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:05 smithi156 ceph-mon[66799]: from='osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811]' entity='osd.0' 2024-03-19T01:05:05.503 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T01:05:06.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: purged_snaps scrub starts 2024-03-19T01:05:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: purged_snaps scrub ok 2024-03-19T01:05:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:05:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811] boot 2024-03-19T01:05:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: osdmap e8: 1 total, 1 up, 1 in 2024-03-19T01:05:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:05:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:06 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:06.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: purged_snaps scrub starts 2024-03-19T01:05:06.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: purged_snaps scrub ok 2024-03-19T01:05:06.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:05:06.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:06.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811] boot 2024-03-19T01:05:06.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: osdmap e8: 1 total, 1 up, 1 in 2024-03-19T01:05:06.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:06.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:06.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:06.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:05:06.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:06.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:06.317 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:06.318 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:06 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:06.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: purged_snaps scrub starts 2024-03-19T01:05:06.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: purged_snaps scrub ok 2024-03-19T01:05:06.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-19T01:05:06.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:06.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: osd.0 [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811] boot 2024-03-19T01:05:06.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: osdmap e8: 1 total, 1 up, 1 in 2024-03-19T01:05:06.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-19T01:05:06.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:06.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:06.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:05:06.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:06.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:06.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:06.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:06 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:07.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.043+0000 7f3cefb1f700 1 -- 172.21.15.156:0/1904665910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce8102440 msgr2=0x7f3ce81028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:07.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.043+0000 7f3cefb1f700 1 --2- 172.21.15.156:0/1904665910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce8102440 0x7f3ce81028a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40099c0 tx=0x7f3ce4009cd0 comp rx=0 tx=0).stop 2024-03-19T01:05:07.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.043+0000 7f3cefb1f700 1 -- 172.21.15.156:0/1904665910 shutdown_connections 2024-03-19T01:05:07.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.043+0000 7f3cefb1f700 1 --2- 172.21.15.156:0/1904665910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ce8102de0 0x7f3ce8109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:07.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.043+0000 7f3cefb1f700 1 --2- 172.21.15.156:0/1904665910 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ce8101240 0x7f3ce8101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:07.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.043+0000 7f3cefb1f700 1 --2- 172.21.15.156:0/1904665910 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce8102440 0x7f3ce81028a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:07.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.043+0000 7f3cefb1f700 1 -- 172.21.15.156:0/1904665910 >> 172.21.15.156:0/1904665910 conn(0x7f3ce80fc7e0 msgr2=0x7f3ce80fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:05:07.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.043+0000 7f3cefb1f700 1 -- 172.21.15.156:0/1904665910 shutdown_connections 2024-03-19T01:05:07.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.043+0000 7f3cefb1f700 1 -- 172.21.15.156:0/1904665910 wait complete. 2024-03-19T01:05:07.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3cefb1f700 1 Processor -- start 2024-03-19T01:05:07.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3cefb1f700 1 -- start start 2024-03-19T01:05:07.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3cefb1f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce8101240 0x7f3ce81986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:05:07.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3cefb1f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ce8102440 0x7f3ce8198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:05:07.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3cefb1f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ce8102de0 0x7f3ce819fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:05:07.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3cefb1f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce810ce70 con 0x7f3ce8102440 2024-03-19T01:05:07.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3cefb1f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce810ccf0 con 0x7f3ce8101240 2024-03-19T01:05:07.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3cefb1f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce810cff0 con 0x7f3ce8102de0 2024-03-19T01:05:07.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3ced8bb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce8101240 0x7f3ce81986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:05:07.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3ced8bb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce8101240 0x7f3ce81986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60212/0 (socket says 172.21.15.156:60212) 2024-03-19T01:05:07.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3ced8bb700 1 -- 172.21.15.156:0/4291318428 learned_addr learned my addr 172.21.15.156:0/4291318428 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3cee0bc700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ce8102de0 0x7f3ce819fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3ced0ba700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ce8102440 0x7f3ce8198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3ced8bb700 1 -- 172.21.15.156:0/4291318428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ce8102de0 msgr2=0x7f3ce819fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3ced8bb700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ce8102de0 0x7f3ce819fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3ced8bb700 1 -- 172.21.15.156:0/4291318428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ce8102440 msgr2=0x7f3ce8198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3ced8bb700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ce8102440 0x7f3ce8198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3ced8bb700 1 -- 172.21.15.156:0/4291318428 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce81a0260 con 0x7f3ce8101240 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.044+0000 7f3ced0ba700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ce8102440 0x7f3ce8198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.045+0000 7f3ced8bb700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce8101240 0x7f3ce81986a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800e950 tx=0x7f3cd800ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.045+0000 7f3cdeffd700 1 -- 172.21.15.156:0/4291318428 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd800eea0 con 0x7f3ce8101240 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.045+0000 7f3cdeffd700 1 -- 172.21.15.156:0/4291318428 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3cd8004510 con 0x7f3ce8101240 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.045+0000 7f3cefb1f700 1 -- 172.21.15.156:0/4291318428 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce81a0550 con 0x7f3ce8101240 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.045+0000 7f3cdeffd700 1 -- 172.21.15.156:0/4291318428 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8010430 con 0x7f3ce8101240 2024-03-19T01:05:07.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.045+0000 7f3cefb1f700 1 -- 172.21.15.156:0/4291318428 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce81a0a40 con 0x7f3ce8101240 2024-03-19T01:05:07.049 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.046+0000 7f3cefb1f700 1 -- 172.21.15.156:0/4291318428 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce8068fd0 con 0x7f3ce8101240 2024-03-19T01:05:07.050 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.047+0000 7f3cdeffd700 1 -- 172.21.15.156:0/4291318428 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7f3cd8003eb0 con 0x7f3ce8101240 2024-03-19T01:05:07.050 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.047+0000 7f3cdeffd700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3cd40a2800 0x7f3cd40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:05:07.050 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.047+0000 7f3cdeffd700 1 -- 172.21.15.156:0/4291318428 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7f3cd80cd2f0 con 0x7f3ce8101240 2024-03-19T01:05:07.050 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.048+0000 7f3ced0ba700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3cd40a2800 0x7f3cd40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:05:07.050 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.048+0000 7f3ced0ba700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3cd40a2800 0x7f3cd40a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40038c0 tx=0x7f3ce4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:05:07.052 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.052+0000 7f3cdeffd700 1 -- 172.21.15.156:0/4291318428 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3cd809be00 con 0x7f3ce8101240 2024-03-19T01:05:07.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:07 smithi156 ceph-mon[66799]: Detected new or changed devices on smithi116 2024-03-19T01:05:07.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:07 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi116 to 17088M 2024-03-19T01:05:07.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:07.255+0000 7f3cefb1f700 1 -- 172.21.15.156:0/4291318428 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f3ce806a6d0 con 0x7f3cd40a2800 2024-03-19T01:05:07.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:07 smithi162 ceph-mon[67178]: Detected new or changed devices on smithi116 2024-03-19T01:05:07.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:07 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi116 to 17088M 2024-03-19T01:05:07.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:07 smithi116 ceph-mon[62701]: Detected new or changed devices on smithi116 2024-03-19T01:05:07.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:07 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi116 to 17088M 2024-03-19T01:05:08.118 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:08 smithi156 ceph-mon[66799]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:08.119 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:08 smithi156 ceph-mon[66799]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T01:05:08.119 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:08 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:05:08.119 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:08 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:05:08.119 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:08 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:08.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:08 smithi162 ceph-mon[67178]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:08.467 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:08 smithi162 ceph-mon[67178]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T01:05:08.467 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:08 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:05:08.468 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:08 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:05:08.468 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:08 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:08.468 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:08 smithi116 ceph-mon[62701]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:08.468 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:08 smithi116 ceph-mon[62701]: osdmap e9: 1 total, 1 up, 1 in 2024-03-19T01:05:08.468 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:05:08.468 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:05:08.468 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:09.107 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:09 smithi156 ceph-mon[66799]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:05:09.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:09 smithi162 ceph-mon[67178]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:05:09.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:09 smithi116 ceph-mon[62701]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:05:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:10 smithi162 ceph-mon[67178]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:10 smithi162 ceph-mon[67178]: from='client.? 172.21.15.156:0/797739710' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ab4837e-b52f-4696-9361-2395e835e740"}]: dispatch 2024-03-19T01:05:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:10 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ab4837e-b52f-4696-9361-2395e835e740"}]: dispatch 2024-03-19T01:05:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:10 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ab4837e-b52f-4696-9361-2395e835e740"}]': finished 2024-03-19T01:05:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:10 smithi162 ceph-mon[67178]: osdmap e10: 2 total, 1 up, 2 in 2024-03-19T01:05:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:10 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:10.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:10 smithi116 ceph-mon[62701]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:10.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:10 smithi116 ceph-mon[62701]: from='client.? 172.21.15.156:0/797739710' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ab4837e-b52f-4696-9361-2395e835e740"}]: dispatch 2024-03-19T01:05:10.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:10 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ab4837e-b52f-4696-9361-2395e835e740"}]: dispatch 2024-03-19T01:05:10.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:10 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ab4837e-b52f-4696-9361-2395e835e740"}]': finished 2024-03-19T01:05:10.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:10 smithi116 ceph-mon[62701]: osdmap e10: 2 total, 1 up, 2 in 2024-03-19T01:05:10.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:10 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:10 smithi156 ceph-mon[66799]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:10 smithi156 ceph-mon[66799]: from='client.? 172.21.15.156:0/797739710' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ab4837e-b52f-4696-9361-2395e835e740"}]: dispatch 2024-03-19T01:05:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:10 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6ab4837e-b52f-4696-9361-2395e835e740"}]: dispatch 2024-03-19T01:05:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:10 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6ab4837e-b52f-4696-9361-2395e835e740"}]': finished 2024-03-19T01:05:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:10 smithi156 ceph-mon[66799]: osdmap e10: 2 total, 1 up, 2 in 2024-03-19T01:05:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:10 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:11.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:11 smithi156 ceph-mon[66799]: from='client.? 172.21.15.156:0/653711430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:05:11.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:11 smithi162 ceph-mon[67178]: from='client.? 172.21.15.156:0/653711430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:05:11.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:11 smithi116 ceph-mon[62701]: from='client.? 172.21.15.156:0/653711430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:05:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:12 smithi162 ceph-mon[67178]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:12.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:12 smithi116 ceph-mon[62701]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:12.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:12 smithi156 ceph-mon[66799]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:14 smithi162 ceph-mon[67178]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:14 smithi116 ceph-mon[62701]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:14 smithi156 ceph-mon[66799]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:16.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:16 smithi156 ceph-mon[66799]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:16 smithi162 ceph-mon[67178]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:16 smithi116 ceph-mon[62701]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T01:05:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T01:05:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:17.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:17 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-19T01:05:17.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:17 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:18.213 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:18 smithi156 ceph-mon[66799]: Deploying daemon osd.1 on smithi156 2024-03-19T01:05:18.213 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:18 smithi156 ceph-mon[66799]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:18.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:18 smithi162 ceph-mon[67178]: Deploying daemon osd.1 on smithi156 2024-03-19T01:05:18.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:18 smithi162 ceph-mon[67178]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:18 smithi116 ceph-mon[62701]: Deploying daemon osd.1 on smithi156 2024-03-19T01:05:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:18 smithi116 ceph-mon[62701]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:19 smithi116 ceph-mon[62701]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:19.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:19 smithi156 ceph-mon[66799]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:19.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:19 smithi162 ceph-mon[67178]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:22.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:22 smithi116 ceph-mon[62701]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:22.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:22 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:22.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:22 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:22.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:22 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:22.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:22 smithi162 ceph-mon[67178]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:22.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:22 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:22.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:22 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:22.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:22 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:22.439 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:22 smithi156 ceph-mon[66799]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:22.439 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:22 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:22.439 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:22 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:22.439 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:22 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:24.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:24 smithi162 ceph-mon[67178]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:24.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:24 smithi116 ceph-mon[62701]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:24.469 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:24 smithi156 ceph-mon[66799]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:25.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:25.413+0000 7f3cdeffd700 1 -- 172.21.15.156:0/4291318428 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3ce806a6d0 con 0x7f3cd40a2800 2024-03-19T01:05:25.414 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 1 on host 'smithi156' 2024-03-19T01:05:25.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:25.415+0000 7f3cefb1f700 1 -- 172.21.15.156:0/4291318428 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3cd40a2800 msgr2=0x7f3cd40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:25.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:25.415+0000 7f3cefb1f700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3cd40a2800 0x7f3cd40a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40038c0 tx=0x7f3ce4000bc0 comp rx=0 tx=0).stop 2024-03-19T01:05:25.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:25.415+0000 7f3cefb1f700 1 -- 172.21.15.156:0/4291318428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce8101240 msgr2=0x7f3ce81986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:25.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:25.415+0000 7f3cefb1f700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce8101240 0x7f3ce81986a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800e950 tx=0x7f3cd800ec60 comp rx=0 tx=0).stop 2024-03-19T01:05:25.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:25.415+0000 7f3cefb1f700 1 -- 172.21.15.156:0/4291318428 shutdown_connections 2024-03-19T01:05:25.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:25.415+0000 7f3cefb1f700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3cd40a2800 0x7f3cd40a4cc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:25.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:25.415+0000 7f3cefb1f700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ce8102de0 0x7f3ce819fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:25.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:25.415+0000 7f3cefb1f700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce8101240 0x7f3ce81986a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:25.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:25.415+0000 7f3cefb1f700 1 --2- 172.21.15.156:0/4291318428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ce8102440 0x7f3ce8198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:25.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:25.415+0000 7f3cefb1f700 1 -- 172.21.15.156:0/4291318428 >> 172.21.15.156:0/4291318428 conn(0x7f3ce80fc7e0 msgr2=0x7f3ce8069fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:05:25.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:25.415+0000 7f3cefb1f700 1 -- 172.21.15.156:0/4291318428 shutdown_connections 2024-03-19T01:05:25.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:05:25.415+0000 7f3cefb1f700 1 -- 172.21.15.156:0/4291318428 wait complete. 2024-03-19T01:05:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:25 smithi156 ceph-mon[66799]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:25.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:25 smithi156 ceph-mon[66799]: from='osd.1 [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T01:05:25.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:25 smithi156 ceph-mon[66799]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T01:05:26.124 DEBUG:teuthology.orchestra.run.smithi156:osd.1> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.1.service 2024-03-19T01:05:26.127 INFO:tasks.cephadm:Deploying osd.2 on smithi162 with /dev/vg_nvme/lv_4... 2024-03-19T01:05:26.127 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-19T01:05:26.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:26.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:26.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:26.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:26.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:26.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:25 smithi116 ceph-mon[62701]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:26.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:26.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:26.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:26.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:25 smithi116 ceph-mon[62701]: from='osd.1 [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T01:05:26.157 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:25 smithi116 ceph-mon[62701]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T01:05:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:26.157 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:25 smithi162 ceph-mon[67178]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:26.158 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:26.158 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:26.158 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:26.158 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:25 smithi162 ceph-mon[67178]: from='osd.1 [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T01:05:26.158 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:25 smithi162 ceph-mon[67178]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-19T01:05:26.372 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:05:27.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:27 smithi156 ceph-mon[66799]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T01:05:27.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:27 smithi156 ceph-mon[66799]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T01:05:27.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:27 smithi156 ceph-mon[66799]: from='osd.1 [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:05:27.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:27.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:27 smithi156 ceph-mon[66799]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:05:27.608 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:27 smithi156 ceph-mon[66799]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:27.609 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:05:27 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1[71462]: 2024-03-19T01:05:27.377+0000 7fa763085700 -1 osd.1 0 waiting for initial osdmap 2024-03-19T01:05:27.609 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:05:27 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1[71462]: 2024-03-19T01:05:27.382+0000 7fa75ee78700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:05:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:27 smithi162 ceph-mon[67178]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T01:05:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:27 smithi162 ceph-mon[67178]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T01:05:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:27 smithi162 ceph-mon[67178]: from='osd.1 [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:05:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:27 smithi162 ceph-mon[67178]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:05:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:27 smithi162 ceph-mon[67178]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:27.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:27 smithi116 ceph-mon[62701]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-19T01:05:27.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:27 smithi116 ceph-mon[62701]: osdmap e11: 2 total, 1 up, 2 in 2024-03-19T01:05:27.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:27 smithi116 ceph-mon[62701]: from='osd.1 [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:05:27.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:27.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:27 smithi116 ceph-mon[62701]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:05:27.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:27 smithi116 ceph-mon[62701]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-19T01:05:28.038 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-19T01:05:28.038 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-19T01:05:28.038 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-03-19T01:05:28.038 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-03-19T01:05:28.038 INFO:teuthology.orchestra.run.smithi162.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113444 s, 924 MB/s 2024-03-19T01:05:28.038 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-03-19T01:05:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:28 smithi162 ceph-mon[67178]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-19T01:05:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:28 smithi162 ceph-mon[67178]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T01:05:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:28.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:28 smithi116 ceph-mon[62701]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-19T01:05:28.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:28 smithi116 ceph-mon[62701]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T01:05:28.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:28.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:28.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_4 2024-03-19T01:05:28.718 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:28 smithi156 ceph-mon[66799]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-19T01:05:28.718 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:28 smithi156 ceph-mon[66799]: osdmap e12: 2 total, 1 up, 2 in 2024-03-19T01:05:28.718 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:28.718 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:28.976 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:05:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: purged_snaps scrub starts 2024-03-19T01:05:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: purged_snaps scrub ok 2024-03-19T01:05:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: osd.1 [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908] boot 2024-03-19T01:05:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: osdmap e13: 2 total, 2 up, 2 in 2024-03-19T01:05:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: Detected new or changed devices on smithi156 2024-03-19T01:05:29.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:29.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:29.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:05:29.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi156 to 17080M 2024-03-19T01:05:29.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:29.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:29.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:29.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:29.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: purged_snaps scrub starts 2024-03-19T01:05:29.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: purged_snaps scrub ok 2024-03-19T01:05:29.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: osd.1 [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908] boot 2024-03-19T01:05:29.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: osdmap e13: 2 total, 2 up, 2 in 2024-03-19T01:05:29.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:29.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:29.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: Detected new or changed devices on smithi156 2024-03-19T01:05:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:05:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi156 to 17080M 2024-03-19T01:05:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:29.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: purged_snaps scrub starts 2024-03-19T01:05:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: purged_snaps scrub ok 2024-03-19T01:05:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: osd.1 [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908] boot 2024-03-19T01:05:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: osdmap e13: 2 total, 2 up, 2 in 2024-03-19T01:05:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-19T01:05:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: Detected new or changed devices on smithi156 2024-03-19T01:05:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:05:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi156 to 17080M 2024-03-19T01:05:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:29.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:30.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.523+0000 7f0c8133a700 1 -- 172.21.15.162:0/2743668012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c7c103ee0 msgr2=0x7f0c7c10a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:30.527 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.523+0000 7f0c8133a700 1 --2- 172.21.15.162:0/2743668012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c7c103ee0 0x7f0c7c10a790 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400b0a0 tx=0x7f0c7400b3b0 comp rx=0 tx=0).stop 2024-03-19T01:05:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.523+0000 7f0c8133a700 1 -- 172.21.15.162:0/2743668012 shutdown_connections 2024-03-19T01:05:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.523+0000 7f0c8133a700 1 --2- 172.21.15.162:0/2743668012 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c7c103ee0 0x7f0c7c10a790 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.523+0000 7f0c8133a700 1 --2- 172.21.15.162:0/2743668012 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c7c102340 0x7f0c7c102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.523+0000 7f0c8133a700 1 --2- 172.21.15.162:0/2743668012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c7c103540 0x7f0c7c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.523+0000 7f0c8133a700 1 -- 172.21.15.162:0/2743668012 >> 172.21.15.162:0/2743668012 conn(0x7f0c7c0fd920 msgr2=0x7f0c7c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:05:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.524+0000 7f0c8133a700 1 -- 172.21.15.162:0/2743668012 shutdown_connections 2024-03-19T01:05:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.524+0000 7f0c8133a700 1 -- 172.21.15.162:0/2743668012 wait complete. 2024-03-19T01:05:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.524+0000 7f0c8133a700 1 Processor -- start 2024-03-19T01:05:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.524+0000 7f0c8133a700 1 -- start start 2024-03-19T01:05:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c8133a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c7c102340 0x7f0c7c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:05:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c8133a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c7c103540 0x7f0c7c198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:05:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c8133a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c7c103ee0 0x7f0c7c19fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:05:30.528 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c8133a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7c10dfd0 con 0x7f0c7c103ee0 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c8133a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7c10de50 con 0x7f0c7c102340 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c8133a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7c10e150 con 0x7f0c7c103540 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c7a7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c7c103540 0x7f0c7c198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c7a7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c7c103540 0x7f0c7c198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:38690/0 (socket says 172.21.15.162:38690) 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c7a7fc700 1 -- 172.21.15.162:0/1438458343 learned_addr learned my addr 172.21.15.162:0/1438458343 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c7affd700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c7c102340 0x7f0c7c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c7b7fe700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c7c103ee0 0x7f0c7c19fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c7a7fc700 1 -- 172.21.15.162:0/1438458343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c7c102340 msgr2=0x7f0c7c198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c7a7fc700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c7c102340 0x7f0c7c198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c7a7fc700 1 -- 172.21.15.162:0/1438458343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c7c103ee0 msgr2=0x7f0c7c19fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c7a7fc700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c7c103ee0 0x7f0c7c19fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c7a7fc700 1 -- 172.21.15.162:0/1438458343 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7c1a01f0 con 0x7f0c7c103540 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c7affd700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c7c102340 0x7f0c7c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:05:30.529 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c7a7fc700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c7c103540 0x7f0c7c198ad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0c7000d9b0 tx=0x7f0c7000dd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:05:30.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c63fff700 1 -- 172.21.15.162:0/1438458343 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c70017070 con 0x7f0c7c103540 2024-03-19T01:05:30.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c8133a700 1 -- 172.21.15.162:0/1438458343 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c7c1a04e0 con 0x7f0c7c103540 2024-03-19T01:05:30.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.525+0000 7f0c63fff700 1 -- 172.21.15.162:0/1438458343 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c70004d10 con 0x7f0c7c103540 2024-03-19T01:05:30.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.526+0000 7f0c63fff700 1 -- 172.21.15.162:0/1438458343 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c70005500 con 0x7f0c7c103540 2024-03-19T01:05:30.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.526+0000 7f0c8133a700 1 -- 172.21.15.162:0/1438458343 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c7c1a0a50 con 0x7f0c7c103540 2024-03-19T01:05:30.530 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.526+0000 7f0c8133a700 1 -- 172.21.15.162:0/1438458343 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c7c068fd0 con 0x7f0c7c103540 2024-03-19T01:05:30.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.528+0000 7f0c63fff700 1 -- 172.21.15.162:0/1438458343 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7f0c70004750 con 0x7f0c7c103540 2024-03-19T01:05:30.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.528+0000 7f0c63fff700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0c640a2800 0x7f0c640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:05:30.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.528+0000 7f0c63fff700 1 -- 172.21.15.162:0/1438458343 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f0c700cd520 con 0x7f0c7c103540 2024-03-19T01:05:30.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.528+0000 7f0c7affd700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0c640a2800 0x7f0c640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:05:30.531 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.529+0000 7f0c7affd700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0c640a2800 0x7f0c640a4cc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c1033a0 tx=0x7f0c6c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:05:30.532 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.532+0000 7f0c63fff700 1 -- 172.21.15.162:0/1438458343 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0c70020020 con 0x7f0c7c103540 2024-03-19T01:05:30.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:30 smithi162 ceph-mon[67178]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T01:05:30.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:30 smithi116 ceph-mon[62701]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T01:05:30.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:30 smithi156 ceph-mon[66799]: osdmap e14: 2 total, 2 up, 2 in 2024-03-19T01:05:30.736 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:30.735+0000 7f0c8133a700 1 -- 172.21.15.162:0/1438458343 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f0c7c062750 con 0x7f0c640a2800 2024-03-19T01:05:31.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:31 smithi162 ceph-mon[67178]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:05:31.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:31 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:05:31.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:31 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:05:31.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:31 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:31.632 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:31 smithi162 ceph-mon[67178]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:31.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:31 smithi116 ceph-mon[62701]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:05:31.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:05:31.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:05:31.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:31.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:31 smithi116 ceph-mon[62701]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:31.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:31 smithi156 ceph-mon[66799]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:05:31.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:31 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:05:31.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:31 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:05:31.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:31 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:31.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:31 smithi156 ceph-mon[66799]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:34.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:34 smithi162 ceph-mon[67178]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:34.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:34 smithi162 ceph-mon[67178]: from='client.? 172.21.15.162:0/55520625' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef1c978e-270f-4b74-844d-2fe4c09f2ef5"}]: dispatch 2024-03-19T01:05:34.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:34 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef1c978e-270f-4b74-844d-2fe4c09f2ef5"}]: dispatch 2024-03-19T01:05:34.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:34 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef1c978e-270f-4b74-844d-2fe4c09f2ef5"}]': finished 2024-03-19T01:05:34.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:34 smithi162 ceph-mon[67178]: osdmap e15: 3 total, 2 up, 3 in 2024-03-19T01:05:34.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:34 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:34.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:34 smithi162 ceph-mon[67178]: from='client.? 172.21.15.162:0/3595359395' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:05:34.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:34 smithi116 ceph-mon[62701]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:34.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:34 smithi116 ceph-mon[62701]: from='client.? 172.21.15.162:0/55520625' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef1c978e-270f-4b74-844d-2fe4c09f2ef5"}]: dispatch 2024-03-19T01:05:34.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:34 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef1c978e-270f-4b74-844d-2fe4c09f2ef5"}]: dispatch 2024-03-19T01:05:34.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:34 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef1c978e-270f-4b74-844d-2fe4c09f2ef5"}]': finished 2024-03-19T01:05:34.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:34 smithi116 ceph-mon[62701]: osdmap e15: 3 total, 2 up, 3 in 2024-03-19T01:05:34.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:34 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:34.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:34 smithi116 ceph-mon[62701]: from='client.? 172.21.15.162:0/3595359395' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:05:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:34 smithi156 ceph-mon[66799]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:34 smithi156 ceph-mon[66799]: from='client.? 172.21.15.162:0/55520625' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef1c978e-270f-4b74-844d-2fe4c09f2ef5"}]: dispatch 2024-03-19T01:05:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:34 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef1c978e-270f-4b74-844d-2fe4c09f2ef5"}]: dispatch 2024-03-19T01:05:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:34 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef1c978e-270f-4b74-844d-2fe4c09f2ef5"}]': finished 2024-03-19T01:05:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:34 smithi156 ceph-mon[66799]: osdmap e15: 3 total, 2 up, 3 in 2024-03-19T01:05:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:34 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:34 smithi156 ceph-mon[66799]: from='client.? 172.21.15.162:0/3595359395' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:05:36.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:36 smithi162 ceph-mon[67178]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:36 smithi116 ceph-mon[62701]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:36.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:36 smithi156 ceph-mon[66799]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:38.286 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:38 smithi162 ceph-mon[67178]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:38.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:38 smithi116 ceph-mon[62701]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:38.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:38 smithi156 ceph-mon[66799]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:40 smithi162 ceph-mon[67178]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:40 smithi116 ceph-mon[62701]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:40.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:40 smithi156 ceph-mon[66799]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:41.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:41 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T01:05:41.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:41 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:41.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:41 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T01:05:41.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:41 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:41.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:41 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-19T01:05:41.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:41 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:42.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:42 smithi162 ceph-mon[67178]: Deploying daemon osd.2 on smithi162 2024-03-19T01:05:42.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:42 smithi162 ceph-mon[67178]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:42.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:42 smithi116 ceph-mon[62701]: Deploying daemon osd.2 on smithi162 2024-03-19T01:05:42.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:42 smithi116 ceph-mon[62701]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:42.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:42 smithi156 ceph-mon[66799]: Deploying daemon osd.2 on smithi162 2024-03-19T01:05:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:42 smithi156 ceph-mon[66799]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:44.238 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:44 smithi162 ceph-mon[67178]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:44.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:44 smithi116 ceph-mon[62701]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:44.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:44 smithi156 ceph-mon[66799]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:46.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:46 smithi162 ceph-mon[67178]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:46.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:46 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:46.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:46 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:46.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:46 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:46.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:46 smithi116 ceph-mon[62701]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:46.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:46 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:46.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:46 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:46.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:46 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:46.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:46 smithi156 ceph-mon[66799]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:46.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:46 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:46.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:46 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:46.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:46 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:48.373 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:48 smithi162 ceph-mon[67178]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:48.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:48 smithi116 ceph-mon[62701]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:48.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:48 smithi156 ceph-mon[66799]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:49.307 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:49.307+0000 7f0c63fff700 1 -- 172.21.15.162:0/1438458343 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0c7c062750 con 0x7f0c640a2800 2024-03-19T01:05:49.307 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 2 on host 'smithi162' 2024-03-19T01:05:49.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:49.308+0000 7f0c8133a700 1 -- 172.21.15.162:0/1438458343 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0c640a2800 msgr2=0x7f0c640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:49.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:49.308+0000 7f0c8133a700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0c640a2800 0x7f0c640a4cc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c1033a0 tx=0x7f0c6c008040 comp rx=0 tx=0).stop 2024-03-19T01:05:49.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:49.308+0000 7f0c8133a700 1 -- 172.21.15.162:0/1438458343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c7c103540 msgr2=0x7f0c7c198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:49.309 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:49.308+0000 7f0c8133a700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c7c103540 0x7f0c7c198ad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0c7000d9b0 tx=0x7f0c7000dd70 comp rx=0 tx=0).stop 2024-03-19T01:05:49.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:49.308+0000 7f0c8133a700 1 -- 172.21.15.162:0/1438458343 shutdown_connections 2024-03-19T01:05:49.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:49.308+0000 7f0c8133a700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c7c103ee0 0x7f0c7c19fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:49.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:49.308+0000 7f0c8133a700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c7c102340 0x7f0c7c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:49.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:49.308+0000 7f0c8133a700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0c640a2800 0x7f0c640a4cc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:49.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:49.308+0000 7f0c8133a700 1 --2- 172.21.15.162:0/1438458343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0c7c103540 0x7f0c7c198ad0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:49.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:49.308+0000 7f0c8133a700 1 -- 172.21.15.162:0/1438458343 >> 172.21.15.162:0/1438458343 conn(0x7f0c7c0fd920 msgr2=0x7f0c7c1067e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:05:49.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:49.309+0000 7f0c8133a700 1 -- 172.21.15.162:0/1438458343 shutdown_connections 2024-03-19T01:05:49.310 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:05:49.309+0000 7f0c8133a700 1 -- 172.21.15.162:0/1438458343 wait complete. 2024-03-19T01:05:49.613 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:49 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.613 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:49 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:49 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:49.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:49 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:49.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:49 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:49 smithi162 ceph-mon[67178]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:49.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:49 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:49 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:49 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:49.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:49 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:49 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:49 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:49.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:49 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:49.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:49 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:49 smithi156 ceph-mon[66799]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:49.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:49 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:49 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:49 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:49.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:49.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:49.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:49 smithi116 ceph-mon[62701]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:49.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:49.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:05:49.923 DEBUG:teuthology.orchestra.run.smithi162:osd.2> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.2.service 2024-03-19T01:05:49.925 INFO:tasks.cephadm:Deploying osd.3 on smithi116 with /dev/vg_nvme/lv_3... 2024-03-19T01:05:49.925 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T01:05:50.172 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:05:50.574 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:50 smithi116 ceph-mon[62701]: from='osd.2 [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T01:05:50.574 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:50 smithi116 ceph-mon[62701]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T01:05:50.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:50 smithi156 ceph-mon[66799]: from='osd.2 [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T01:05:50.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:50 smithi156 ceph-mon[66799]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T01:05:50.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:50 smithi162 ceph-mon[67178]: from='osd.2 [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T01:05:50.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:50 smithi162 ceph-mon[67178]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-19T01:05:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:51 smithi116 ceph-mon[62701]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T01:05:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:51 smithi116 ceph-mon[62701]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T01:05:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:51 smithi116 ceph-mon[62701]: from='osd.2 [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:05:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:51 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:51 smithi116 ceph-mon[62701]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:05:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:51 smithi116 ceph-mon[62701]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:51.508 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:51 smithi116 ceph-mon[62701]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T01:05:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:51 smithi162 ceph-mon[67178]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T01:05:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:51 smithi162 ceph-mon[67178]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T01:05:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:51 smithi162 ceph-mon[67178]: from='osd.2 [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:05:51.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:51 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:51.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:51 smithi162 ceph-mon[67178]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:05:51.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:51 smithi162 ceph-mon[67178]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:51.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:51 smithi162 ceph-mon[67178]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T01:05:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:51 smithi156 ceph-mon[66799]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-19T01:05:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:51 smithi156 ceph-mon[66799]: osdmap e16: 3 total, 2 up, 3 in 2024-03-19T01:05:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:51 smithi156 ceph-mon[66799]: from='osd.2 [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:05:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:51 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:51 smithi156 ceph-mon[66799]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:05:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:51 smithi156 ceph-mon[66799]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:51 smithi156 ceph-mon[66799]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T01:05:52.036 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T01:05:52.036 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T01:05:52.036 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-19T01:05:52.036 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-19T01:05:52.036 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011449 s, 916 MB/s 2024-03-19T01:05:52.036 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-19T01:05:52.246 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:05:51 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2[72346]: 2024-03-19T01:05:51.933+0000 7fa7d03e1700 -1 osd.2 0 waiting for initial osdmap 2024-03-19T01:05:52.247 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:05:51 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2[72346]: 2024-03-19T01:05:51.938+0000 7fa7c81cc700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:05:52.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:52 smithi162 ceph-mon[67178]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T01:05:52.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:52 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:52.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:52 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:52.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:52 smithi162 ceph-mon[67178]: from='osd.2 ' entity='osd.2' 2024-03-19T01:05:52.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:52 smithi116 ceph-mon[62701]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T01:05:52.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:52 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:52.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:52 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:52.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:52 smithi116 ceph-mon[62701]: from='osd.2 ' entity='osd.2' 2024-03-19T01:05:52.704 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_3 2024-03-19T01:05:52.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:52 smithi156 ceph-mon[66799]: osdmap e17: 3 total, 2 up, 3 in 2024-03-19T01:05:52.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:52 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:52.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:52 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:52.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:52 smithi156 ceph-mon[66799]: from='osd.2 ' entity='osd.2' 2024-03-19T01:05:52.940 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:05:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: purged_snaps scrub starts 2024-03-19T01:05:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: purged_snaps scrub ok 2024-03-19T01:05:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: osd.2 [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758] boot 2024-03-19T01:05:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: osdmap e18: 3 total, 3 up, 3 in 2024-03-19T01:05:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-19T01:05:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: Detected new or changed devices on smithi162 2024-03-19T01:05:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:05:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:53.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:53.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:53.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: purged_snaps scrub starts 2024-03-19T01:05:53.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: purged_snaps scrub ok 2024-03-19T01:05:53.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:53.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: osd.2 [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758] boot 2024-03-19T01:05:53.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: osdmap e18: 3 total, 3 up, 3 in 2024-03-19T01:05:53.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:53.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:53.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-19T01:05:53.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: Detected new or changed devices on smithi162 2024-03-19T01:05:53.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:53.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:53.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:05:53.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:53.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:53.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:53.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:53.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: purged_snaps scrub starts 2024-03-19T01:05:53.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: purged_snaps scrub ok 2024-03-19T01:05:53.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:53.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: osd.2 [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758] boot 2024-03-19T01:05:53.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: osdmap e18: 3 total, 3 up, 3 in 2024-03-19T01:05:53.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-19T01:05:53.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-19T01:05:53.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-19T01:05:53.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: Detected new or changed devices on smithi162 2024-03-19T01:05:53.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:53.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:53.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:05:53.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:53.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:53.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:05:53.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:54.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.496+0000 7faf33817700 1 -- 172.21.15.116:0/868664515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf2c103ee0 msgr2=0x7faf2c10a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:54.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.496+0000 7faf33817700 1 --2- 172.21.15.116:0/868664515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf2c103ee0 0x7faf2c10a790 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7faf2400b0d0 tx=0x7faf2400b3e0 comp rx=0 tx=0).stop 2024-03-19T01:05:54.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.496+0000 7faf33817700 1 -- 172.21.15.116:0/868664515 shutdown_connections 2024-03-19T01:05:54.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.496+0000 7faf33817700 1 --2- 172.21.15.116:0/868664515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf2c103ee0 0x7faf2c10a790 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:54.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.496+0000 7faf33817700 1 --2- 172.21.15.116:0/868664515 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf2c102340 0x7faf2c102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:54.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.496+0000 7faf33817700 1 --2- 172.21.15.116:0/868664515 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf2c103540 0x7faf2c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:54.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.496+0000 7faf33817700 1 -- 172.21.15.116:0/868664515 >> 172.21.15.116:0/868664515 conn(0x7faf2c0fd920 msgr2=0x7faf2c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:05:54.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.497+0000 7faf33817700 1 -- 172.21.15.116:0/868664515 shutdown_connections 2024-03-19T01:05:54.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.497+0000 7faf33817700 1 -- 172.21.15.116:0/868664515 wait complete. 2024-03-19T01:05:54.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.497+0000 7faf33817700 1 Processor -- start 2024-03-19T01:05:54.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.497+0000 7faf33817700 1 -- start start 2024-03-19T01:05:54.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.497+0000 7faf33817700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf2c102340 0x7faf2c19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:05:54.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.497+0000 7faf33817700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf2c103540 0x7faf2c19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.497+0000 7faf33817700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf2c103ee0 0x7faf2c1a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.497+0000 7faf33817700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c10df80 con 0x7faf2c102340 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.497+0000 7faf33817700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c10de00 con 0x7faf2c103ee0 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.497+0000 7faf33817700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c10e100 con 0x7faf2c103540 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf315b3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf2c102340 0x7faf2c19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf315b3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf2c102340 0x7faf2c19a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44828/0 (socket says 172.21.15.116:44828) 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf315b3700 1 -- 172.21.15.116:0/434511849 learned_addr learned my addr 172.21.15.116:0/434511849 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf31db4700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf2c103ee0 0x7faf2c1a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf30db2700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf2c103540 0x7faf2c19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf315b3700 1 -- 172.21.15.116:0/434511849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf2c103540 msgr2=0x7faf2c19ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf315b3700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf2c103540 0x7faf2c19ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf315b3700 1 -- 172.21.15.116:0/434511849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf2c103ee0 msgr2=0x7faf2c1a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf315b3700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf2c103ee0 0x7faf2c1a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf315b3700 1 -- 172.21.15.116:0/434511849 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2c1a23a0 con 0x7faf2c102340 2024-03-19T01:05:54.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf315b3700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf2c102340 0x7faf2c19a740 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faf2800c9a0 tx=0x7faf2800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:05:54.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf227fc700 1 -- 172.21.15.116:0/434511849 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf28004220 con 0x7faf2c102340 2024-03-19T01:05:54.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf227fc700 1 -- 172.21.15.116:0/434511849 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf28007d40 con 0x7faf2c102340 2024-03-19T01:05:54.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf33817700 1 -- 172.21.15.116:0/434511849 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2c1a2690 con 0x7faf2c102340 2024-03-19T01:05:54.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf227fc700 1 -- 172.21.15.116:0/434511849 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf28003da0 con 0x7faf2c102340 2024-03-19T01:05:54.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.498+0000 7faf33817700 1 -- 172.21.15.116:0/434511849 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf2c1a2ca0 con 0x7faf2c102340 2024-03-19T01:05:54.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.499+0000 7faf33817700 1 -- 172.21.15.116:0/434511849 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf2c068fd0 con 0x7faf2c102340 2024-03-19T01:05:54.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.500+0000 7faf227fc700 1 -- 172.21.15.116:0/434511849 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 15) ==== 137235+0+0 (secure 0 0 0) 0x7faf28007860 con 0x7faf2c102340 2024-03-19T01:05:54.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.501+0000 7faf227fc700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7faf180a2880 0x7faf180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:05:54.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.501+0000 7faf227fc700 1 -- 172.21.15.116:0/434511849 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7faf280cd130 con 0x7faf2c102340 2024-03-19T01:05:54.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.501+0000 7faf30db2700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7faf180a2880 0x7faf180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:05:54.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.501+0000 7faf30db2700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7faf180a2880 0x7faf180a4d40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faf1c006fd0 tx=0x7faf1c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:05:54.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.506+0000 7faf227fc700 1 -- 172.21.15.116:0/434511849 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faf28014030 con 0x7faf2c102340 2024-03-19T01:05:54.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:54.709+0000 7faf33817700 1 -- 172.21.15.116:0/434511849 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7faf2c075c60 con 0x7faf180a2880 2024-03-19T01:05:55.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:54 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi162 to 17080M 2024-03-19T01:05:55.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-19T01:05:55.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:54 smithi116 ceph-mon[62701]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T01:05:55.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T01:05:55.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:05:55.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:05:55.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:55.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:54 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi162 to 17080M 2024-03-19T01:05:55.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-19T01:05:55.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:54 smithi156 ceph-mon[66799]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T01:05:55.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T01:05:55.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:05:55.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:05:55.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:55.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:54 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi162 to 17080M 2024-03-19T01:05:55.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-19T01:05:55.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:54 smithi162 ceph-mon[67178]: osdmap e19: 3 total, 3 up, 3 in 2024-03-19T01:05:55.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T01:05:55.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:05:55.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:05:55.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:05:55.545 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:05:55 smithi116 sudo[81798]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T01:05:55.546 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:05:55 smithi116 sudo[81798]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T01:05:55.546 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:05:55 smithi116 sudo[81798]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T01:05:55.546 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:05:55 smithi116 sudo[81798]: pam_unix(sudo:session): session closed for user root 2024-03-19T01:05:55.905 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:05:55 smithi116 sudo[81805]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T01:05:55.905 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:05:55 smithi116 sudo[81805]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T01:05:55.905 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:05:55 smithi116 sudo[81805]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T01:05:55.905 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:05:55 smithi116 sudo[81805]: pam_unix(sudo:session): session closed for user root 2024-03-19T01:05:56.079 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:05:55 smithi156 sudo[75585]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T01:05:56.079 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:05:55 smithi156 sudo[75585]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T01:05:56.079 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:05:55 smithi156 sudo[75585]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T01:05:56.079 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:05:55 smithi156 sudo[75585]: pam_unix(sudo:session): session closed for user root 2024-03-19T01:05:56.079 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:55 smithi156 ceph-mon[66799]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:05:56.080 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:55 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T01:05:56.080 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:55 smithi156 ceph-mon[66799]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T01:05:56.080 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:55 smithi156 ceph-mon[66799]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:05:56.080 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:55 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:56.316 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:55 smithi116 ceph-mon[62701]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:05:56.316 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:55 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T01:05:56.316 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:55 smithi116 ceph-mon[62701]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T01:05:56.316 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:55 smithi116 ceph-mon[62701]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:05:56.316 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:55 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:56.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:55 smithi162 ceph-mon[67178]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:05:56.344 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:55 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-19T01:05:56.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:55 smithi162 ceph-mon[67178]: osdmap e20: 3 total, 3 up, 3 in 2024-03-19T01:05:56.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:55 smithi162 ceph-mon[67178]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:05:56.345 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:55 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:56.474 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:05:56 smithi156 sudo[75588]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T01:05:56.474 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:05:56 smithi156 sudo[75588]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T01:05:56.474 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:05:56 smithi156 sudo[75588]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T01:05:56.474 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:05:56 smithi156 sudo[75588]: pam_unix(sudo:session): session closed for user root 2024-03-19T01:05:56.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:05:56.585+0000 7faf227fc700 1 -- 172.21.15.116:0/434511849 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7faf28097600 con 0x7faf2c102340 2024-03-19T01:05:56.597 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:05:56 smithi162 sudo[76153]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T01:05:56.597 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:05:56 smithi162 sudo[76153]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T01:05:56.597 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:05:56 smithi162 sudo[76153]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T01:05:56.598 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:05:56 smithi162 sudo[76153]: pam_unix(sudo:session): session closed for user root 2024-03-19T01:05:56.899 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:05:56 smithi162 sudo[76156]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T01:05:56.899 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:05:56 smithi162 sudo[76156]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T01:05:56.899 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:05:56 smithi162 sudo[76156]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T01:05:56.899 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:05:56 smithi162 sudo[76156]: pam_unix(sudo:session): session closed for user root 2024-03-19T01:05:57.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:56 smithi116 sudo[81853]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T01:05:57.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:56 smithi116 sudo[81853]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T01:05:57.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:56 smithi116 sudo[81853]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T01:05:57.144 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:56 smithi116 sudo[81853]: pam_unix(sudo:session): session closed for user root 2024-03-19T01:05:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:57 smithi162 ceph-mon[67178]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T01:05:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:57 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:57 smithi162 ceph-mon[67178]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-19T01:05:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:57 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:57 smithi162 ceph-mon[67178]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T01:05:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:57 smithi162 sudo[76159]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T01:05:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:57 smithi162 sudo[76159]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T01:05:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:57 smithi162 sudo[76159]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T01:05:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:57 smithi116 sudo[81875]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T01:05:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:57 smithi116 sudo[81875]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T01:05:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:57 smithi116 sudo[81875]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T01:05:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:57 smithi116 sudo[81875]: pam_unix(sudo:session): session closed for user root 2024-03-19T01:05:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:57 smithi116 ceph-mon[62701]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T01:05:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:57 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:57 smithi116 ceph-mon[62701]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-19T01:05:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:57 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:57 smithi116 ceph-mon[62701]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T01:05:57.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:57 smithi156 ceph-mon[66799]: osdmap e21: 3 total, 3 up, 3 in 2024-03-19T01:05:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:57 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:57 smithi156 ceph-mon[66799]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-19T01:05:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:57 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:05:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:57 smithi156 ceph-mon[66799]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T01:05:57.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:57 smithi162 sudo[76159]: pam_unix(sudo:session): session closed for user root 2024-03-19T01:05:57.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:57 smithi162 sudo[76162]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T01:05:57.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:57 smithi162 sudo[76162]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T01:05:57.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:57 smithi162 sudo[76162]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T01:05:57.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:57 smithi162 sudo[76162]: pam_unix(sudo:session): session closed for user root 2024-03-19T01:05:58.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:57 smithi156 sudo[75591]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-19T01:05:58.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:57 smithi156 sudo[75591]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T01:05:58.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:57 smithi156 sudo[75591]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T01:05:58.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:57 smithi156 sudo[75591]: pam_unix(sudo:session): session closed for user root 2024-03-19T01:05:58.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 sudo[75594]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-19T01:05:58.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 sudo[75594]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-19T01:05:58.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 sudo[75594]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-19T01:05:58.133 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 sudo[75594]: pam_unix(sudo:session): session closed for user root 2024-03-19T01:05:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:05:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T01:05:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:05:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:05:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:05:58.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T01:05:58.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:05:58.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:05:58.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:05:58.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2591848476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "944f6747-a95f-49ae-8b3b-d7c48dc15059"}]: dispatch 2024-03-19T01:05:58.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T01:05:58.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2591848476' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "944f6747-a95f-49ae-8b3b-d7c48dc15059"}]': finished 2024-03-19T01:05:58.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T01:05:58.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: osdmap e22: 4 total, 3 up, 4 in 2024-03-19T01:05:58.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:05:58.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:05:58.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:05:58.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:05:58.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:58 smithi116 ceph-mon[62701]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T01:05:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:05:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T01:05:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:05:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:05:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:05:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T01:05:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:05:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:05:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:05:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2591848476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "944f6747-a95f-49ae-8b3b-d7c48dc15059"}]: dispatch 2024-03-19T01:05:58.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T01:05:58.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2591848476' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "944f6747-a95f-49ae-8b3b-d7c48dc15059"}]': finished 2024-03-19T01:05:58.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T01:05:58.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: osdmap e22: 4 total, 3 up, 4 in 2024-03-19T01:05:58.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:05:58.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:05:58.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:05:58.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:05:58.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:58 smithi156 ceph-mon[66799]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T01:05:58.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: pgmap v86: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:05:58.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T01:05:58.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:05:58.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2591848476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "944f6747-a95f-49ae-8b3b-d7c48dc15059"}]: dispatch 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2591848476' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "944f6747-a95f-49ae-8b3b-d7c48dc15059"}]': finished 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: osdmap e22: 4 total, 3 up, 4 in 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-19T01:05:58.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-19T01:05:58.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:58 smithi162 ceph-mon[67178]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-19T01:05:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:59 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2037825583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:05:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:05:59 smithi156 ceph-mon[66799]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:05:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:59 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2037825583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:05:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:05:59 smithi162 ceph-mon[67178]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:05:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:59 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2037825583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:05:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:05:59 smithi116 ceph-mon[62701]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:02.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:02 smithi162 ceph-mon[67178]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:02.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:02 smithi116 ceph-mon[62701]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:02.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:02 smithi156 ceph-mon[66799]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:04.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:04 smithi116 ceph-mon[62701]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:04 smithi162 ceph-mon[67178]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:04.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:04 smithi156 ceph-mon[66799]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T01:06:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T01:06:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:05.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-19T01:06:05.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:06.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:06 smithi116 ceph-mon[62701]: Deploying daemon osd.3 on smithi116 2024-03-19T01:06:06.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:06 smithi116 ceph-mon[62701]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:06.316 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:06 smithi162 ceph-mon[67178]: Deploying daemon osd.3 on smithi116 2024-03-19T01:06:06.316 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:06 smithi162 ceph-mon[67178]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:06.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:06 smithi156 ceph-mon[66799]: Deploying daemon osd.3 on smithi116 2024-03-19T01:06:06.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:06 smithi156 ceph-mon[66799]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:08.366 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:08 smithi116 ceph-mon[62701]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:08.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:08 smithi162 ceph-mon[67178]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:08.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:08 smithi156 ceph-mon[66799]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:10.143 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:10 smithi116 ceph-mon[62701]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:10 smithi162 ceph-mon[67178]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:10.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:10 smithi156 ceph-mon[66799]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:11.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:11.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:11.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:06:11.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:11 smithi116 ceph-mon[62701]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:11.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:11.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:11.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:06:11.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:11 smithi162 ceph-mon[67178]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:11.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:11.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:11.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:06:11.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:11 smithi156 ceph-mon[66799]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:14.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:14 smithi116 ceph-mon[62701]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:14.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:14.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:14.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:14.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:06:14.394 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:14 smithi162 ceph-mon[67178]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:06:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:14.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:06:14.470+0000 7faf227fc700 1 -- 172.21.15.116:0/434511849 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7faf2c075c60 con 0x7faf180a2880 2024-03-19T01:06:14.471 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 3 on host 'smithi116' 2024-03-19T01:06:14.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:06:14.471+0000 7faf33817700 1 -- 172.21.15.116:0/434511849 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7faf180a2880 msgr2=0x7faf180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:06:14.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:06:14.471+0000 7faf33817700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7faf180a2880 0x7faf180a4d40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7faf1c006fd0 tx=0x7faf1c008040 comp rx=0 tx=0).stop 2024-03-19T01:06:14.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:06:14.471+0000 7faf33817700 1 -- 172.21.15.116:0/434511849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf2c102340 msgr2=0x7faf2c19a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:06:14.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:06:14.471+0000 7faf33817700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf2c102340 0x7faf2c19a740 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faf2800c9a0 tx=0x7faf2800ccb0 comp rx=0 tx=0).stop 2024-03-19T01:06:14.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:06:14.471+0000 7faf33817700 1 -- 172.21.15.116:0/434511849 shutdown_connections 2024-03-19T01:06:14.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:06:14.471+0000 7faf33817700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faf2c103ee0 0x7faf2c1a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:14.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:06:14.471+0000 7faf33817700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf2c102340 0x7faf2c19a740 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:14.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:06:14.471+0000 7faf33817700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7faf180a2880 0x7faf180a4d40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:14.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:06:14.472+0000 7faf33817700 1 --2- 172.21.15.116:0/434511849 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faf2c103540 0x7faf2c19ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:14.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:06:14.472+0000 7faf33817700 1 -- 172.21.15.116:0/434511849 >> 172.21.15.116:0/434511849 conn(0x7faf2c0fd920 msgr2=0x7faf2c074540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:06:14.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:06:14.472+0000 7faf33817700 1 -- 172.21.15.116:0/434511849 shutdown_connections 2024-03-19T01:06:14.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:06:14.472+0000 7faf33817700 1 -- 172.21.15.116:0/434511849 wait complete. 2024-03-19T01:06:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:14 smithi156 ceph-mon[66799]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:14.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:14.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:14.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:14.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:06:14.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:15.108 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:15 smithi116 ceph-mon[62701]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T01:06:15.108 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:15 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:15.108 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:15 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:15.108 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:15 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:06:15.136 DEBUG:teuthology.orchestra.run.smithi116:osd.3> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.3.service 2024-03-19T01:06:15.139 INFO:tasks.cephadm:Deploying osd.4 on smithi156 with /dev/vg_nvme/lv_3... 2024-03-19T01:06:15.139 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T01:06:15.383 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T01:06:15.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:15 smithi162 ceph-mon[67178]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T01:06:15.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:15 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:15.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:15 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:15.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:15 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:06:15.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:15 smithi156 ceph-mon[66799]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-19T01:06:15.491 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:15 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:15.491 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:15 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:15.491 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:15 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:06:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:16 smithi162 ceph-mon[67178]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:16 smithi162 ceph-mon[67178]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T01:06:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:16 smithi162 ceph-mon[67178]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T01:06:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:16 smithi162 ceph-mon[67178]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T01:06:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:16 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:16 smithi116 ceph-mon[62701]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:16 smithi116 ceph-mon[62701]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T01:06:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:16 smithi116 ceph-mon[62701]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T01:06:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:16 smithi116 ceph-mon[62701]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T01:06:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:16 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:16 smithi156 ceph-mon[66799]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:16 smithi156 ceph-mon[66799]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-19T01:06:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:16 smithi156 ceph-mon[66799]: osdmap e23: 4 total, 3 up, 4 in 2024-03-19T01:06:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:16 smithi156 ceph-mon[66799]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T01:06:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:16 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:16.751 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:06:16 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3[83944]: 2024-03-19T01:06:16.497+0000 7f6d4754a700 -1 osd.3 0 waiting for initial osdmap 2024-03-19T01:06:16.751 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:06:16 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3[83944]: 2024-03-19T01:06:16.502+0000 7f6d3f335700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:06:17.176 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:17 smithi156 ceph-mon[66799]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T01:06:17.176 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:17 smithi156 ceph-mon[66799]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T01:06:17.176 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:17 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:17.176 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:17 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:17.176 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:17 smithi156 ceph-mon[66799]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' 2024-03-19T01:06:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:17 smithi162 ceph-mon[67178]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T01:06:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:17 smithi162 ceph-mon[67178]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T01:06:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:17 smithi162 ceph-mon[67178]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' 2024-03-19T01:06:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:17 smithi116 ceph-mon[62701]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T01:06:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:17 smithi116 ceph-mon[62701]: osdmap e24: 4 total, 3 up, 4 in 2024-03-19T01:06:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:17 smithi116 ceph-mon[62701]: from='osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607]' entity='osd.3' 2024-03-19T01:06:17.507 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T01:06:17.508 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T01:06:17.508 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-03-19T01:06:17.508 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-03-19T01:06:17.508 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00908818 s, 1.2 GB/s 2024-03-19T01:06:17.508 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-03-19T01:06:18.126 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi156:vg_nvme/lv_3 2024-03-19T01:06:18.371 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T01:06:18.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:18 smithi162 ceph-mon[67178]: purged_snaps scrub starts 2024-03-19T01:06:18.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:18 smithi162 ceph-mon[67178]: purged_snaps scrub ok 2024-03-19T01:06:18.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:18 smithi162 ceph-mon[67178]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:18.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:18 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:18.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:18 smithi162 ceph-mon[67178]: osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607] boot 2024-03-19T01:06:18.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:18 smithi162 ceph-mon[67178]: osdmap e25: 4 total, 4 up, 4 in 2024-03-19T01:06:18.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:18 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:18 smithi116 ceph-mon[62701]: purged_snaps scrub starts 2024-03-19T01:06:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:18 smithi116 ceph-mon[62701]: purged_snaps scrub ok 2024-03-19T01:06:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:18 smithi116 ceph-mon[62701]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:18 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:18.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:18 smithi116 ceph-mon[62701]: osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607] boot 2024-03-19T01:06:18.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:18 smithi116 ceph-mon[62701]: osdmap e25: 4 total, 4 up, 4 in 2024-03-19T01:06:18.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:18 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:18 smithi156 ceph-mon[66799]: purged_snaps scrub starts 2024-03-19T01:06:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:18 smithi156 ceph-mon[66799]: purged_snaps scrub ok 2024-03-19T01:06:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:18 smithi156 ceph-mon[66799]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-19T01:06:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:18 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:18 smithi156 ceph-mon[66799]: osd.3 [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607] boot 2024-03-19T01:06:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:18 smithi156 ceph-mon[66799]: osdmap e25: 4 total, 4 up, 4 in 2024-03-19T01:06:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:18 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-19T01:06:19.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:19 smithi162 ceph-mon[67178]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T01:06:19.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:19.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:19.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:06:19.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:06:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:06:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:19 smithi116 ceph-mon[62701]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T01:06:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:06:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:06:19.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:19.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:19.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:06:19.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:19.455 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:19 smithi156 ceph-mon[66799]: osdmap e26: 4 total, 4 up, 4 in 2024-03-19T01:06:19.455 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:19.455 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:19.455 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:06:19.455 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:06:19.455 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:19.455 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:19.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:06:19.456 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:19.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.985+0000 7fc32642f700 1 -- 172.21.15.156:0/1415132225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc320103f00 msgr2=0x7fc32010a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:06:19.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.985+0000 7fc32642f700 1 --2- 172.21.15.156:0/1415132225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc320103f00 0x7fc32010a7b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc31800b0d0 tx=0x7fc31800b3e0 comp rx=0 tx=0).stop 2024-03-19T01:06:19.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.985+0000 7fc32642f700 1 -- 172.21.15.156:0/1415132225 shutdown_connections 2024-03-19T01:06:19.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.985+0000 7fc32642f700 1 --2- 172.21.15.156:0/1415132225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc320103f00 0x7fc32010a7b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:19.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.985+0000 7fc32642f700 1 --2- 172.21.15.156:0/1415132225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc320102360 0x7fc320102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:19.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.985+0000 7fc32642f700 1 --2- 172.21.15.156:0/1415132225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc320103560 0x7fc3201039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:19.988 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.985+0000 7fc32642f700 1 -- 172.21.15.156:0/1415132225 >> 172.21.15.156:0/1415132225 conn(0x7fc3200fd8c0 msgr2=0x7fc3200ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.985+0000 7fc32642f700 1 -- 172.21.15.156:0/1415132225 shutdown_connections 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.985+0000 7fc32642f700 1 -- 172.21.15.156:0/1415132225 wait complete. 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.985+0000 7fc32642f700 1 Processor -- start 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc32642f700 1 -- start start 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc32642f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc320102360 0x7fc320198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc32642f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc320103560 0x7fc320198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc32642f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc320103f00 0x7fc32019fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc32642f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc32010dfd0 con 0x7fc320102360 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc32642f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc32010de50 con 0x7fc320103560 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc32642f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc32010e150 con 0x7fc320103f00 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc31f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc320103560 0x7fc320198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc31f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc320103560 0x7fc320198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48918/0 (socket says 172.21.15.156:48918) 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc31f7fe700 1 -- 172.21.15.156:0/3286642118 learned_addr learned my addr 172.21.15.156:0/3286642118 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-19T01:06:19.989 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc31ffff700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc320102360 0x7fc320198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:06:19.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc3249cc700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc320103f00 0x7fc32019fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:06:19.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc31f7fe700 1 -- 172.21.15.156:0/3286642118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc320103f00 msgr2=0x7fc32019fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:06:19.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc31f7fe700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc320103f00 0x7fc32019fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:19.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc31f7fe700 1 -- 172.21.15.156:0/3286642118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc320102360 msgr2=0x7fc320198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:06:19.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc31f7fe700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc320102360 0x7fc320198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:19.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.986+0000 7fc31f7fe700 1 -- 172.21.15.156:0/3286642118 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3201a01b0 con 0x7fc320103560 2024-03-19T01:06:19.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.987+0000 7fc31f7fe700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc320103560 0x7fc320198a90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc31400c920 tx=0x7fc31400cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:06:19.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.987+0000 7fc31d7fa700 1 -- 172.21.15.156:0/3286642118 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc314007b00 con 0x7fc320103560 2024-03-19T01:06:19.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.987+0000 7fc32642f700 1 -- 172.21.15.156:0/3286642118 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3201a04a0 con 0x7fc320103560 2024-03-19T01:06:19.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.987+0000 7fc31d7fa700 1 -- 172.21.15.156:0/3286642118 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc314004d10 con 0x7fc320103560 2024-03-19T01:06:19.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.987+0000 7fc31d7fa700 1 -- 172.21.15.156:0/3286642118 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc314005680 con 0x7fc320103560 2024-03-19T01:06:19.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.987+0000 7fc32642f700 1 -- 172.21.15.156:0/3286642118 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3201a0ab0 con 0x7fc320103560 2024-03-19T01:06:19.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.988+0000 7fc32642f700 1 -- 172.21.15.156:0/3286642118 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc320068fd0 con 0x7fc320103560 2024-03-19T01:06:19.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.989+0000 7fc31d7fa700 1 -- 172.21.15.156:0/3286642118 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fc314004750 con 0x7fc320103560 2024-03-19T01:06:19.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.989+0000 7fc31d7fa700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc3080a2890 0x7fc3080a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:06:19.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.989+0000 7fc31d7fa700 1 -- 172.21.15.156:0/3286642118 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7fc3140cd390 con 0x7fc320103560 2024-03-19T01:06:19.992 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.990+0000 7fc31ffff700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc3080a2890 0x7fc3080a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:06:19.992 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.990+0000 7fc31ffff700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc3080a2890 0x7fc3080a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc3201033c0 tx=0x7fc310009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:06:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:19.994+0000 7fc31d7fa700 1 -- 172.21.15.156:0/3286642118 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc314014030 con 0x7fc320103560 2024-03-19T01:06:20.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:20.198+0000 7fc32642f700 1 -- 172.21.15.156:0/3286642118 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fc320062750 con 0x7fc3080a2890 2024-03-19T01:06:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:20 smithi162 ceph-mon[67178]: Detected new or changed devices on smithi116 2024-03-19T01:06:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:20 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi116 to 8544M 2024-03-19T01:06:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:20 smithi162 ceph-mon[67178]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:20 smithi162 ceph-mon[67178]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T01:06:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:20 smithi162 ceph-mon[67178]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T01:06:20.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:20 smithi116 ceph-mon[62701]: Detected new or changed devices on smithi116 2024-03-19T01:06:20.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:20 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi116 to 8544M 2024-03-19T01:06:20.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:20 smithi116 ceph-mon[62701]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:20.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:20 smithi116 ceph-mon[62701]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T01:06:20.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:20 smithi116 ceph-mon[62701]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T01:06:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:20 smithi156 ceph-mon[66799]: Detected new or changed devices on smithi116 2024-03-19T01:06:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:20 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi116 to 8544M 2024-03-19T01:06:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:20 smithi156 ceph-mon[66799]: pgmap v102: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:20 smithi156 ceph-mon[66799]: osdmap e27: 4 total, 4 up, 4 in 2024-03-19T01:06:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:20 smithi156 ceph-mon[66799]: osdmap e28: 4 total, 4 up, 4 in 2024-03-19T01:06:21.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:06:21.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:06:21.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:21.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:21 smithi162 ceph-mon[67178]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T01:06:21.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:06:21.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:06:21.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:21.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:21 smithi116 ceph-mon[62701]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T01:06:21.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:06:21.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:06:21.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:21.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:21 smithi156 ceph-mon[66799]: osdmap e29: 4 total, 4 up, 4 in 2024-03-19T01:06:22.318 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:22 smithi156 ceph-mon[66799]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:06:22.318 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:22 smithi156 ceph-mon[66799]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:22.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:22 smithi162 ceph-mon[67178]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:06:22.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:22 smithi162 ceph-mon[67178]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:22.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:22 smithi116 ceph-mon[62701]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:06:22.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:22 smithi116 ceph-mon[62701]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:23 smithi162 ceph-mon[67178]: from='client.? 172.21.15.156:0/1843382673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bd47f0a-a0d1-4402-8de0-940cd4f6a69b"}]: dispatch 2024-03-19T01:06:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:23 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bd47f0a-a0d1-4402-8de0-940cd4f6a69b"}]: dispatch 2024-03-19T01:06:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:23 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bd47f0a-a0d1-4402-8de0-940cd4f6a69b"}]': finished 2024-03-19T01:06:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:23 smithi162 ceph-mon[67178]: osdmap e30: 5 total, 4 up, 5 in 2024-03-19T01:06:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:23 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:23 smithi116 ceph-mon[62701]: from='client.? 172.21.15.156:0/1843382673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bd47f0a-a0d1-4402-8de0-940cd4f6a69b"}]: dispatch 2024-03-19T01:06:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:23 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bd47f0a-a0d1-4402-8de0-940cd4f6a69b"}]: dispatch 2024-03-19T01:06:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:23 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bd47f0a-a0d1-4402-8de0-940cd4f6a69b"}]': finished 2024-03-19T01:06:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:23 smithi116 ceph-mon[62701]: osdmap e30: 5 total, 4 up, 5 in 2024-03-19T01:06:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:23 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:23.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:23 smithi156 ceph-mon[66799]: from='client.? 172.21.15.156:0/1843382673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bd47f0a-a0d1-4402-8de0-940cd4f6a69b"}]: dispatch 2024-03-19T01:06:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:23 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1bd47f0a-a0d1-4402-8de0-940cd4f6a69b"}]: dispatch 2024-03-19T01:06:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:23 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1bd47f0a-a0d1-4402-8de0-940cd4f6a69b"}]': finished 2024-03-19T01:06:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:23 smithi156 ceph-mon[66799]: osdmap e30: 5 total, 4 up, 5 in 2024-03-19T01:06:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:23 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:24.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:24 smithi162 ceph-mon[67178]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:24.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:24 smithi162 ceph-mon[67178]: from='client.? 172.21.15.156:0/3043179065' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:06:24.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:24 smithi116 ceph-mon[62701]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:24.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:24 smithi116 ceph-mon[62701]: from='client.? 172.21.15.156:0/3043179065' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:06:24.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:24 smithi156 ceph-mon[66799]: pgmap v108: 1 pgs: 1 remapped+peering; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:24.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:24 smithi156 ceph-mon[66799]: from='client.? 172.21.15.156:0/3043179065' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:06:26.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:26 smithi162 ceph-mon[67178]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-19T01:06:26.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:26 smithi116 ceph-mon[62701]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-19T01:06:26.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:26 smithi156 ceph-mon[66799]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-19T01:06:28.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:28 smithi156 ceph-mon[66799]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-19T01:06:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:28 smithi162 ceph-mon[67178]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-19T01:06:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:28 smithi116 ceph-mon[62701]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-19T01:06:30.329 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:30 smithi156 ceph-mon[66799]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-19T01:06:30.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:30 smithi162 ceph-mon[67178]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-19T01:06:30.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:30 smithi116 ceph-mon[62701]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-19T01:06:31.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:31 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T01:06:31.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:31 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:31.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:31 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T01:06:31.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:31 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-19T01:06:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:32.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:32 smithi156 ceph-mon[66799]: Deploying daemon osd.4 on smithi156 2024-03-19T01:06:32.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:32 smithi156 ceph-mon[66799]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T01:06:32.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:32 smithi162 ceph-mon[67178]: Deploying daemon osd.4 on smithi156 2024-03-19T01:06:32.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:32 smithi162 ceph-mon[67178]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T01:06:32.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:32 smithi116 ceph-mon[62701]: Deploying daemon osd.4 on smithi156 2024-03-19T01:06:32.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:32 smithi116 ceph-mon[62701]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T01:06:34.222 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:34 smithi156 ceph-mon[66799]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-19T01:06:34.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:34 smithi162 ceph-mon[67178]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-19T01:06:34.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:34 smithi116 ceph-mon[62701]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-19T01:06:35.466 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:35 smithi156 ceph-mon[66799]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-19T01:06:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:35 smithi162 ceph-mon[67178]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-19T01:06:35.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:35 smithi116 ceph-mon[62701]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-03-19T01:06:37.068 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:36 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:37.068 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:36 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:37.068 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:36 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:06:37.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:36 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:37.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:36 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:37.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:36 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:06:37.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:36 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:37.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:36 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:37.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:36 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:06:37.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:37 smithi156 ceph-mon[66799]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:38.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:37 smithi162 ceph-mon[67178]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:38.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:37 smithi116 ceph-mon[62701]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:39.614 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 4 on host 'smithi156' 2024-03-19T01:06:39.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:39.613+0000 7fc31d7fa700 1 -- 172.21.15.156:0/3286642118 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc320062750 con 0x7fc3080a2890 2024-03-19T01:06:39.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:39.616+0000 7fc32642f700 1 -- 172.21.15.156:0/3286642118 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc3080a2890 msgr2=0x7fc3080a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:06:39.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:39.616+0000 7fc32642f700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc3080a2890 0x7fc3080a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc3201033c0 tx=0x7fc310009190 comp rx=0 tx=0).stop 2024-03-19T01:06:39.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:39.616+0000 7fc32642f700 1 -- 172.21.15.156:0/3286642118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc320103560 msgr2=0x7fc320198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:06:39.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:39.616+0000 7fc32642f700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc320103560 0x7fc320198a90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc31400c920 tx=0x7fc31400cc30 comp rx=0 tx=0).stop 2024-03-19T01:06:39.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:39.616+0000 7fc32642f700 1 -- 172.21.15.156:0/3286642118 shutdown_connections 2024-03-19T01:06:39.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:39.616+0000 7fc32642f700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc3080a2890 0x7fc3080a4d50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:39.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:39.616+0000 7fc32642f700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc320103f00 0x7fc32019fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:39.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:39.616+0000 7fc32642f700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc320102360 0x7fc320198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:39.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:39.616+0000 7fc32642f700 1 --2- 172.21.15.156:0/3286642118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc320103560 0x7fc320198a90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:39.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:39.616+0000 7fc32642f700 1 -- 172.21.15.156:0/3286642118 >> 172.21.15.156:0/3286642118 conn(0x7fc3200fd8c0 msgr2=0x7fc320106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:06:39.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:39.616+0000 7fc32642f700 1 -- 172.21.15.156:0/3286642118 shutdown_connections 2024-03-19T01:06:39.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:06:39.616+0000 7fc32642f700 1 -- 172.21.15.156:0/3286642118 wait complete. 2024-03-19T01:06:39.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:39.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:39.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:39.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:06:39.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:39.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:39 smithi116 ceph-mon[62701]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:39.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:39.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:39.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:06:39.916 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:39 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:39.916 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:39 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:39.916 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:39 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:39.917 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:39 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:06:39.917 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:39 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:39.917 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:39 smithi156 ceph-mon[66799]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:39.917 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:39 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:39.917 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:39 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:39.917 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:39 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:06:40.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:39 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:40.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:39 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:40.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:39 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:40.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:39 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:06:40.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:39 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:40.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:39 smithi162 ceph-mon[67178]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:40.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:39 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:40.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:39 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:40.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:39 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:06:40.192 DEBUG:teuthology.orchestra.run.smithi156:osd.4> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.4.service 2024-03-19T01:06:40.194 INFO:tasks.cephadm:Deploying osd.5 on smithi162 with /dev/vg_nvme/lv_3... 2024-03-19T01:06:40.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-19T01:06:40.439 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:06:40.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:40 smithi162 ceph-mon[67178]: from='osd.4 [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T01:06:40.849 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:40 smithi162 ceph-mon[67178]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T01:06:40.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:40 smithi156 ceph-mon[66799]: from='osd.4 [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T01:06:40.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:40 smithi156 ceph-mon[66799]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T01:06:41.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:40 smithi116 ceph-mon[62701]: from='osd.4 [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T01:06:41.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:40 smithi116 ceph-mon[62701]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-19T01:06:41.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:41 smithi162 ceph-mon[67178]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T01:06:41.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:41 smithi162 ceph-mon[67178]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T01:06:41.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:41 smithi162 ceph-mon[67178]: from='osd.4 [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:06:41.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:41 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:41.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:41 smithi162 ceph-mon[67178]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:06:41.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:41 smithi162 ceph-mon[67178]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:41.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:41 smithi156 ceph-mon[66799]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T01:06:41.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:41 smithi156 ceph-mon[66799]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T01:06:41.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:41 smithi156 ceph-mon[66799]: from='osd.4 [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:06:41.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:41 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:41.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:41 smithi156 ceph-mon[66799]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:06:41.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:41 smithi156 ceph-mon[66799]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:42.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:41 smithi116 ceph-mon[62701]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-19T01:06:42.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:41 smithi116 ceph-mon[62701]: osdmap e31: 5 total, 4 up, 5 in 2024-03-19T01:06:42.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:41 smithi116 ceph-mon[62701]: from='osd.4 [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:06:42.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:41 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:42.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:41 smithi116 ceph-mon[62701]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:06:42.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:41 smithi116 ceph-mon[62701]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:42.359 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-19T01:06:42.359 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-19T01:06:42.359 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-03-19T01:06:42.359 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-03-19T01:06:42.360 INFO:teuthology.orchestra.run.smithi162.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112935 s, 928 MB/s 2024-03-19T01:06:42.360 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-03-19T01:06:42.474 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:06:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:06:42.105+0000 7f0f0343e700 -1 osd.4 0 waiting for initial osdmap 2024-03-19T01:06:42.474 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:06:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:06:42.111+0000 7f0efb229700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:06:42.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:42 smithi162 ceph-mon[67178]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-19T01:06:42.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:42 smithi162 ceph-mon[67178]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T01:06:42.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:42.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:42.744 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:42 smithi162 ceph-mon[67178]: from='osd.4 ' entity='osd.4' 2024-03-19T01:06:42.830 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:42 smithi156 ceph-mon[66799]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-19T01:06:42.830 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:42 smithi156 ceph-mon[66799]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T01:06:42.830 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:42.830 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:42.830 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:42 smithi156 ceph-mon[66799]: from='osd.4 ' entity='osd.4' 2024-03-19T01:06:43.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_3 2024-03-19T01:06:43.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:42 smithi116 ceph-mon[62701]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-19T01:06:43.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:42 smithi116 ceph-mon[62701]: osdmap e32: 5 total, 4 up, 5 in 2024-03-19T01:06:43.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:43.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:43.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:42 smithi116 ceph-mon[62701]: from='osd.4 ' entity='osd.4' 2024-03-19T01:06:43.315 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:06:43.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: purged_snaps scrub starts 2024-03-19T01:06:43.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: purged_snaps scrub ok 2024-03-19T01:06:43.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:43.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:43.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: osd.4 [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382] boot 2024-03-19T01:06:43.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: osdmap e33: 5 total, 5 up, 5 in 2024-03-19T01:06:43.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:43.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:43.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:43.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:06:43.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:06:43.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:43.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:43.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:06:43.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:44.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: purged_snaps scrub starts 2024-03-19T01:06:44.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: purged_snaps scrub ok 2024-03-19T01:06:44.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:44.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:44.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: osd.4 [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382] boot 2024-03-19T01:06:44.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: osdmap e33: 5 total, 5 up, 5 in 2024-03-19T01:06:44.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:44.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:44.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:44.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:06:44.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:06:44.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:44.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:44.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:06:44.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:44.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: purged_snaps scrub starts 2024-03-19T01:06:44.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: purged_snaps scrub ok 2024-03-19T01:06:44.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:44.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-19T01:06:44.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: osd.4 [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382] boot 2024-03-19T01:06:44.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: osdmap e33: 5 total, 5 up, 5 in 2024-03-19T01:06:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-19T01:06:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:06:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:06:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:06:44.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:06:44.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:44 smithi162 ceph-mon[67178]: Detected new or changed devices on smithi156 2024-03-19T01:06:44.804 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:44 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi156 to 8540M 2024-03-19T01:06:44.805 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:44 smithi162 ceph-mon[67178]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T01:06:44.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:44 smithi156 ceph-mon[66799]: Detected new or changed devices on smithi156 2024-03-19T01:06:44.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:44 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi156 to 8540M 2024-03-19T01:06:44.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:44 smithi156 ceph-mon[66799]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T01:06:45.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.038+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1272219811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc102dd0 msgr2=0x7f17dc109680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:06:45.041 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.038+0000 7f17e2ed6700 1 --2- 172.21.15.162:0/1272219811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc102dd0 0x7f17dc109680 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f17d4009990 tx=0x7f17d4009ca0 comp rx=0 tx=0).stop 2024-03-19T01:06:45.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.038+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1272219811 shutdown_connections 2024-03-19T01:06:45.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.038+0000 7f17e2ed6700 1 --2- 172.21.15.162:0/1272219811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc102dd0 0x7f17dc109680 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:45.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.038+0000 7f17e2ed6700 1 --2- 172.21.15.162:0/1272219811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17dc101230 0x7f17dc101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:45.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.038+0000 7f17e2ed6700 1 --2- 172.21.15.162:0/1272219811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17dc102430 0x7f17dc102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:45.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.038+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1272219811 >> 172.21.15.162:0/1272219811 conn(0x7f17dc0fc7d0 msgr2=0x7f17dc0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:06:45.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.038+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1272219811 shutdown_connections 2024-03-19T01:06:45.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.038+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1272219811 wait complete. 2024-03-19T01:06:45.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e2ed6700 1 Processor -- start 2024-03-19T01:06:45.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e2ed6700 1 -- start start 2024-03-19T01:06:45.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e2ed6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17dc101230 0x7f17dc1984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:06:45.042 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e2ed6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17dc102430 0x7f17dc198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:06:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e2ed6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc102dd0 0x7f17dc19faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:06:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e2ed6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc10ce90 con 0x7f17dc101230 2024-03-19T01:06:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e2ed6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc10cd10 con 0x7f17dc102430 2024-03-19T01:06:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e2ed6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f17dc10d010 con 0x7f17dc102dd0 2024-03-19T01:06:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e1473700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc102dd0 0x7f17dc19faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:06:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e1473700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc102dd0 0x7f17dc19faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:39292/0 (socket says 172.21.15.162:39292) 2024-03-19T01:06:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e1473700 1 -- 172.21.15.162:0/1350253391 learned_addr learned my addr 172.21.15.162:0/1350253391 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T01:06:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17d3fff700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17dc102430 0x7f17dc198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:06:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e0c72700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17dc101230 0x7f17dc1984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:06:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e1473700 1 -- 172.21.15.162:0/1350253391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17dc102430 msgr2=0x7f17dc198a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:06:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e1473700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17dc102430 0x7f17dc198a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e1473700 1 -- 172.21.15.162:0/1350253391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17dc101230 msgr2=0x7f17dc1984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:06:45.043 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e1473700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17dc101230 0x7f17dc1984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:06:45.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e1473700 1 -- 172.21.15.162:0/1350253391 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17dc1a0140 con 0x7f17dc102dd0 2024-03-19T01:06:45.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17d3fff700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17dc102430 0x7f17dc198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:06:45.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e0c72700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17dc101230 0x7f17dc1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:06:45.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17e1473700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc102dd0 0x7f17dc19faa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17d4003890 tx=0x7f17d4004790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:06:45.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.039+0000 7f17d1ffb700 1 -- 172.21.15.162:0/1350253391 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d4026070 con 0x7f17dc102dd0 2024-03-19T01:06:45.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.040+0000 7f17d1ffb700 1 -- 172.21.15.162:0/1350253391 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17d4004800 con 0x7f17dc102dd0 2024-03-19T01:06:45.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.040+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1350253391 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17dc1a03d0 con 0x7f17dc102dd0 2024-03-19T01:06:45.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.040+0000 7f17d1ffb700 1 -- 172.21.15.162:0/1350253391 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d40054f0 con 0x7f17dc102dd0 2024-03-19T01:06:45.044 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.040+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1350253391 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17dc1a08a0 con 0x7f17dc102dd0 2024-03-19T01:06:45.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.041+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1350253391 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17dc068fd0 con 0x7f17dc102dd0 2024-03-19T01:06:45.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.042+0000 7f17d1ffb700 1 -- 172.21.15.162:0/1350253391 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f17d4029070 con 0x7f17dc102dd0 2024-03-19T01:06:45.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.042+0000 7f17d1ffb700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f17c40a2890 0x7f17c40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:06:45.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.042+0000 7f17d1ffb700 1 -- 172.21.15.162:0/1350253391 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f17d40ce670 con 0x7f17dc102dd0 2024-03-19T01:06:45.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.043+0000 7f17e0c72700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f17c40a2890 0x7f17c40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:06:45.045 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.043+0000 7f17e0c72700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f17c40a2890 0x7f17c40a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f17dc102290 tx=0x7f17d80073c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:06:45.047 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.047+0000 7f17d1ffb700 1 -- 172.21.15.162:0/1350253391 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f17d405d740 con 0x7f17dc102dd0 2024-03-19T01:06:45.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:44 smithi116 ceph-mon[62701]: Detected new or changed devices on smithi156 2024-03-19T01:06:45.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:44 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi156 to 8540M 2024-03-19T01:06:45.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:44 smithi116 ceph-mon[62701]: osdmap e34: 5 total, 5 up, 5 in 2024-03-19T01:06:45.254 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:06:45.253+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1350253391 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f17dc0fe840 con 0x7f17c40a2890 2024-03-19T01:06:45.954 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:45 smithi162 ceph-mon[67178]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:45.955 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:06:45.955 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:06:45.955 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:45.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:45 smithi156 ceph-mon[66799]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:45.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:06:45.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:06:45.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:46.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:45 smithi116 ceph-mon[62701]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:46.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:06:46.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:06:46.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:46.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:46 smithi156 ceph-mon[66799]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:06:46.993 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:46 smithi162 ceph-mon[67178]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:06:47.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:46 smithi116 ceph-mon[62701]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:06:47.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:47 smithi156 ceph-mon[66799]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:48.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:47 smithi162 ceph-mon[67178]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:47 smithi116 ceph-mon[62701]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:48.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:48 smithi156 ceph-mon[66799]: from='client.? 172.21.15.162:0/958291118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "457a29a6-6e9d-406a-95f6-d34e1881e6ca"}]: dispatch 2024-03-19T01:06:48.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:48 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "457a29a6-6e9d-406a-95f6-d34e1881e6ca"}]: dispatch 2024-03-19T01:06:48.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:48 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "457a29a6-6e9d-406a-95f6-d34e1881e6ca"}]': finished 2024-03-19T01:06:48.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:48 smithi156 ceph-mon[66799]: osdmap e35: 6 total, 5 up, 6 in 2024-03-19T01:06:48.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:48 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:06:48.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:48 smithi156 ceph-mon[66799]: from='client.? 172.21.15.162:0/1957611027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:06:49.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:48 smithi162 ceph-mon[67178]: from='client.? 172.21.15.162:0/958291118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "457a29a6-6e9d-406a-95f6-d34e1881e6ca"}]: dispatch 2024-03-19T01:06:49.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:48 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "457a29a6-6e9d-406a-95f6-d34e1881e6ca"}]: dispatch 2024-03-19T01:06:49.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:48 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "457a29a6-6e9d-406a-95f6-d34e1881e6ca"}]': finished 2024-03-19T01:06:49.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:48 smithi162 ceph-mon[67178]: osdmap e35: 6 total, 5 up, 6 in 2024-03-19T01:06:49.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:48 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:06:49.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:48 smithi162 ceph-mon[67178]: from='client.? 172.21.15.162:0/1957611027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:06:49.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:48 smithi116 ceph-mon[62701]: from='client.? 172.21.15.162:0/958291118' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "457a29a6-6e9d-406a-95f6-d34e1881e6ca"}]: dispatch 2024-03-19T01:06:49.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:48 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "457a29a6-6e9d-406a-95f6-d34e1881e6ca"}]: dispatch 2024-03-19T01:06:49.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:48 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "457a29a6-6e9d-406a-95f6-d34e1881e6ca"}]': finished 2024-03-19T01:06:49.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:48 smithi116 ceph-mon[62701]: osdmap e35: 6 total, 5 up, 6 in 2024-03-19T01:06:49.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:48 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:06:49.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:48 smithi116 ceph-mon[62701]: from='client.? 172.21.15.162:0/1957611027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:06:49.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:49 smithi156 ceph-mon[66799]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:50.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:49 smithi162 ceph-mon[67178]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:50.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:49 smithi116 ceph-mon[62701]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:52.360 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:52 smithi162 ceph-mon[67178]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:52.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:52 smithi116 ceph-mon[62701]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:52.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:52 smithi156 ceph-mon[66799]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:54.330 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:54 smithi162 ceph-mon[67178]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:54 smithi116 ceph-mon[62701]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:54.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:54 smithi156 ceph-mon[66799]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:56.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:56 smithi162 ceph-mon[67178]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:56.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:56 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T01:06:56.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:56 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:56.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:56 smithi116 ceph-mon[62701]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:56.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T01:06:56.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:56.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:56 smithi156 ceph-mon[66799]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:56.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:56 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-19T01:06:56.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:56 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:06:57.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:57 smithi162 ceph-mon[67178]: Deploying daemon osd.5 on smithi162 2024-03-19T01:06:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:57 smithi116 ceph-mon[62701]: Deploying daemon osd.5 on smithi162 2024-03-19T01:06:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:57 smithi156 ceph-mon[66799]: Deploying daemon osd.5 on smithi162 2024-03-19T01:06:58.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:06:58 smithi162 ceph-mon[67178]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:06:58 smithi116 ceph-mon[62701]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:06:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:06:58 smithi156 ceph-mon[66799]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:00.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:00 smithi162 ceph-mon[67178]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:00.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:00 smithi116 ceph-mon[62701]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:00.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:00 smithi156 ceph-mon[66799]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:02.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:02 smithi162 ceph-mon[67178]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:02.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:02 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:02.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:02 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:02.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:02 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:02.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:02 smithi116 ceph-mon[62701]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:02.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:02 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:02.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:02 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:02.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:02 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:02.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:02 smithi156 ceph-mon[66799]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:02.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:02 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:02.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:02 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:02.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:02 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:04 smithi162 ceph-mon[67178]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:04 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:04 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:04 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:04 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:07:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:04 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:04.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:04 smithi116 ceph-mon[62701]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:04.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:04.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:04.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:04.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:07:04.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:04.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:04 smithi156 ceph-mon[66799]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:04.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:04 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:04.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:04 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:04.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:04 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:04.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:04 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:07:04.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:04 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:04.969 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:07:04.968+0000 7f17d1ffb700 1 -- 172.21.15.162:0/1350253391 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f17dc0fe840 con 0x7f17c40a2890 2024-03-19T01:07:04.970 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 5 on host 'smithi162' 2024-03-19T01:07:04.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:07:04.971+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1350253391 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f17c40a2890 msgr2=0x7f17c40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:07:04.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:07:04.971+0000 7f17e2ed6700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f17c40a2890 0x7f17c40a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f17dc102290 tx=0x7f17d80073c0 comp rx=0 tx=0).stop 2024-03-19T01:07:04.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:07:04.971+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1350253391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc102dd0 msgr2=0x7f17dc19faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:07:04.972 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:07:04.971+0000 7f17e2ed6700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc102dd0 0x7f17dc19faa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17d4003890 tx=0x7f17d4004790 comp rx=0 tx=0).stop 2024-03-19T01:07:04.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:07:04.971+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1350253391 shutdown_connections 2024-03-19T01:07:04.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:07:04.971+0000 7f17e2ed6700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f17c40a2890 0x7f17c40a4d50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:04.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:07:04.971+0000 7f17e2ed6700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f17dc102dd0 0x7f17dc19faa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:04.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:07:04.971+0000 7f17e2ed6700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17dc101230 0x7f17dc1984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:04.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:07:04.971+0000 7f17e2ed6700 1 --2- 172.21.15.162:0/1350253391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17dc102430 0x7f17dc198a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:04.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:07:04.971+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1350253391 >> 172.21.15.162:0/1350253391 conn(0x7f17dc0fc7d0 msgr2=0x7f17dc0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:07:04.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:07:04.971+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1350253391 shutdown_connections 2024-03-19T01:07:04.973 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:07:04.971+0000 7f17e2ed6700 1 -- 172.21.15.162:0/1350253391 wait complete. 2024-03-19T01:07:05.744 DEBUG:teuthology.orchestra.run.smithi162:osd.5> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.5.service 2024-03-19T01:07:05.746 INFO:tasks.cephadm:Deploying osd.6 on smithi116 with /dev/vg_nvme/lv_2... 2024-03-19T01:07:05.747 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T01:07:05.993 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:07:05.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:05.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:05.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:05.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:05 smithi162 ceph-mon[67178]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:05.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:05 smithi162 ceph-mon[67178]: from='osd.5 [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T01:07:05.994 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:05 smithi162 ceph-mon[67178]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T01:07:06.114 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:06.114 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:06.115 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:06.115 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:05 smithi116 ceph-mon[62701]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:06.115 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:05 smithi116 ceph-mon[62701]: from='osd.5 [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T01:07:06.115 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:05 smithi116 ceph-mon[62701]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T01:07:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:05 smithi156 ceph-mon[66799]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:05 smithi156 ceph-mon[66799]: from='osd.5 [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T01:07:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:05 smithi156 ceph-mon[66799]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-19T01:07:07.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:06 smithi162 ceph-mon[67178]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T01:07:07.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:06 smithi162 ceph-mon[67178]: from='osd.5 [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:07:07.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:06 smithi162 ceph-mon[67178]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T01:07:07.212 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:06 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:07.213 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:06 smithi162 ceph-mon[67178]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:07:07.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:06 smithi156 ceph-mon[66799]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T01:07:07.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:06 smithi156 ceph-mon[66799]: from='osd.5 [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:07:07.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:06 smithi156 ceph-mon[66799]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T01:07:07.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:06 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:07.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:06 smithi156 ceph-mon[66799]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:07:07.380 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:06 smithi116 ceph-mon[62701]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-19T01:07:07.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:06 smithi116 ceph-mon[62701]: from='osd.5 [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:07:07.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:06 smithi116 ceph-mon[62701]: osdmap e36: 6 total, 5 up, 6 in 2024-03-19T01:07:07.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:06 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:07.381 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:06 smithi116 ceph-mon[62701]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:07:07.520 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:07:07 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:07:07.388+0000 7f20ff44d700 -1 osd.5 0 waiting for initial osdmap 2024-03-19T01:07:07.520 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:07:07 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:07:07.396+0000 7f20f6a37700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:07:08.096 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T01:07:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T01:07:08.097 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-19T01:07:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-19T01:07:08.097 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113677 s, 922 MB/s 2024-03-19T01:07:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-19T01:07:08.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:07 smithi156 ceph-mon[66799]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T01:07:08.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:07 smithi156 ceph-mon[66799]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T01:07:08.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:07 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:08.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:07 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:08.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:07 smithi156 ceph-mon[66799]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:08.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:07 smithi156 ceph-mon[66799]: from='osd.5 ' entity='osd.5' 2024-03-19T01:07:08.239 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:07 smithi162 ceph-mon[67178]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T01:07:08.239 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:07 smithi162 ceph-mon[67178]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T01:07:08.239 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:07 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:08.239 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:07 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:08.239 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:07 smithi162 ceph-mon[67178]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:08.239 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:07 smithi162 ceph-mon[67178]: from='osd.5 ' entity='osd.5' 2024-03-19T01:07:08.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:07 smithi116 ceph-mon[62701]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T01:07:08.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:07 smithi116 ceph-mon[62701]: osdmap e37: 6 total, 5 up, 6 in 2024-03-19T01:07:08.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:07 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:08.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:07 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:08.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:07 smithi116 ceph-mon[62701]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 150 MiB used, 447 GiB / 447 GiB avail 2024-03-19T01:07:08.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:07 smithi116 ceph-mon[62701]: from='osd.5 ' entity='osd.5' 2024-03-19T01:07:08.771 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_2 2024-03-19T01:07:09.013 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:07:09.136 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:08 smithi116 ceph-mon[62701]: purged_snaps scrub starts 2024-03-19T01:07:09.136 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:08 smithi116 ceph-mon[62701]: purged_snaps scrub ok 2024-03-19T01:07:09.136 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:09.136 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:08 smithi116 ceph-mon[62701]: osd.5 [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633] boot 2024-03-19T01:07:09.136 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:08 smithi116 ceph-mon[62701]: osdmap e38: 6 total, 6 up, 6 in 2024-03-19T01:07:09.136 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:09.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:08 smithi162 ceph-mon[67178]: purged_snaps scrub starts 2024-03-19T01:07:09.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:08 smithi162 ceph-mon[67178]: purged_snaps scrub ok 2024-03-19T01:07:09.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:08 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:09.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:08 smithi162 ceph-mon[67178]: osd.5 [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633] boot 2024-03-19T01:07:09.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:08 smithi162 ceph-mon[67178]: osdmap e38: 6 total, 6 up, 6 in 2024-03-19T01:07:09.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:08 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:09.332 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:08 smithi156 ceph-mon[66799]: purged_snaps scrub starts 2024-03-19T01:07:09.332 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:08 smithi156 ceph-mon[66799]: purged_snaps scrub ok 2024-03-19T01:07:09.332 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:08 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:09.332 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:08 smithi156 ceph-mon[66799]: osd.5 [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633] boot 2024-03-19T01:07:09.332 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:08 smithi156 ceph-mon[66799]: osdmap e38: 6 total, 6 up, 6 in 2024-03-19T01:07:09.332 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:08 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-19T01:07:10.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:09 smithi116 ceph-mon[62701]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T01:07:10.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:09 smithi116 ceph-mon[62701]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:10.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:09 smithi116 ceph-mon[62701]: Detected new or changed devices on smithi162 2024-03-19T01:07:10.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:09 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:10.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:09 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:10.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:09 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:10.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:09 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:10.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:09 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi162 to 8540M 2024-03-19T01:07:10.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:09 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:10.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:09 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:10.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:09 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:07:10.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:09 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:09 smithi162 ceph-mon[67178]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T01:07:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:09 smithi162 ceph-mon[67178]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:09 smithi162 ceph-mon[67178]: Detected new or changed devices on smithi162 2024-03-19T01:07:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:09 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:09 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:09 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:09 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:10.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:09 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi162 to 8540M 2024-03-19T01:07:10.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:09 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:10.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:09 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:10.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:09 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:07:10.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:09 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:09 smithi156 ceph-mon[66799]: osdmap e39: 6 total, 6 up, 6 in 2024-03-19T01:07:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:09 smithi156 ceph-mon[66799]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:09 smithi156 ceph-mon[66799]: Detected new or changed devices on smithi162 2024-03-19T01:07:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:09 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:09 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:09 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:09 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:10.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:09 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi162 to 8540M 2024-03-19T01:07:10.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:09 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:10.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:09 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:10.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:09 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:07:10.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:09 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:10.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.670+0000 7f95689bb700 1 -- 172.21.15.116:0/4024375535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9564101230 msgr2=0x7f9564101650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:07:10.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.670+0000 7f95689bb700 1 --2- 172.21.15.116:0/4024375535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9564101230 0x7f9564101650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f95540099c0 tx=0x7f9554009cd0 comp rx=0 tx=0).stop 2024-03-19T01:07:10.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.670+0000 7f95689bb700 1 -- 172.21.15.116:0/4024375535 shutdown_connections 2024-03-19T01:07:10.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.670+0000 7f95689bb700 1 --2- 172.21.15.116:0/4024375535 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9564102dd0 0x7f9564109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:10.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.670+0000 7f95689bb700 1 --2- 172.21.15.116:0/4024375535 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9564101230 0x7f9564101650 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:10.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.670+0000 7f95689bb700 1 --2- 172.21.15.116:0/4024375535 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9564102430 0x7f9564102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:10.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.670+0000 7f95689bb700 1 -- 172.21.15.116:0/4024375535 >> 172.21.15.116:0/4024375535 conn(0x7f95640fc7d0 msgr2=0x7f95640fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:07:10.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.670+0000 7f95689bb700 1 -- 172.21.15.116:0/4024375535 shutdown_connections 2024-03-19T01:07:10.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.670+0000 7f95689bb700 1 -- 172.21.15.116:0/4024375535 wait complete. 2024-03-19T01:07:10.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.670+0000 7f95689bb700 1 Processor -- start 2024-03-19T01:07:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f95689bb700 1 -- start start 2024-03-19T01:07:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f95689bb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9564101230 0x7f95641974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:07:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f95689bb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9564102430 0x7f95641979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:07:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f95689bb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9564102dd0 0x7f956419ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:07:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f95689bb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f956410ce90 con 0x7f9564101230 2024-03-19T01:07:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f95689bb700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f956410cd10 con 0x7f9564102430 2024-03-19T01:07:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f95689bb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f956410d010 con 0x7f9564102dd0 2024-03-19T01:07:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f956259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9564101230 0x7f95641974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:07:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f956259c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9564101230 0x7f95641974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46330/0 (socket says 172.21.15.116:46330) 2024-03-19T01:07:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f956259c700 1 -- 172.21.15.116:0/1578535627 learned_addr learned my addr 172.21.15.116:0/1578535627 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:07:10.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f9559d9b700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9564102430 0x7f95641979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f9562d9d700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9564102dd0 0x7f956419ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f956259c700 1 -- 172.21.15.116:0/1578535627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9564102dd0 msgr2=0x7f956419ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f956259c700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9564102dd0 0x7f956419ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f956259c700 1 -- 172.21.15.116:0/1578535627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9564102430 msgr2=0x7f95641979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f956259c700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9564102430 0x7f95641979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f956259c700 1 -- 172.21.15.116:0/1578535627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956406aa70 con 0x7f9564101230 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f9559d9b700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9564102430 0x7f95641979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.671+0000 7f956259c700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9564101230 0x7f95641974b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f95540057c0 tx=0x7f95540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.672+0000 7f955bfff700 1 -- 172.21.15.116:0/1578535627 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9554027070 con 0x7f9564101230 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.672+0000 7f95689bb700 1 -- 172.21.15.116:0/1578535627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956406ad00 con 0x7f9564101230 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.672+0000 7f955bfff700 1 -- 172.21.15.116:0/1578535627 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95540058e0 con 0x7f9564101230 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.672+0000 7f955bfff700 1 -- 172.21.15.116:0/1578535627 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955400f460 con 0x7f9564101230 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.672+0000 7f95689bb700 1 -- 172.21.15.116:0/1578535627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f956406b260 con 0x7f9564101230 2024-03-19T01:07:10.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.673+0000 7f95689bb700 1 -- 172.21.15.116:0/1578535627 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9564068fd0 con 0x7f9564101230 2024-03-19T01:07:10.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.674+0000 7f955bfff700 1 -- 172.21.15.116:0/1578535627 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f955402a070 con 0x7f9564101230 2024-03-19T01:07:10.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.674+0000 7f955bfff700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f95500a2890 0x7f95500a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:07:10.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.674+0000 7f9559d9b700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f95500a2890 0x7f95500a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:07:10.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.674+0000 7f955bfff700 1 -- 172.21.15.116:0/1578535627 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7f95540cf320 con 0x7f9564101230 2024-03-19T01:07:10.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.675+0000 7f9559d9b700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f95500a2890 0x7f95500a4d50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f954c0099d0 tx=0x7f954c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:07:10.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.678+0000 7f955bfff700 1 -- 172.21.15.116:0/1578535627 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9554031020 con 0x7f9564101230 2024-03-19T01:07:10.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:10.884+0000 7f95689bb700 1 -- 172.21.15.116:0/1578535627 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f956406b4c0 con 0x7f95500a2890 2024-03-19T01:07:11.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:10 smithi116 ceph-mon[62701]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T01:07:11.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:10 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:07:11.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:10 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:07:11.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:10 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:11.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:10 smithi162 ceph-mon[67178]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T01:07:11.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:10 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:07:11.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:10 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:07:11.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:10 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:11.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:10 smithi156 ceph-mon[66799]: osdmap e40: 6 total, 6 up, 6 in 2024-03-19T01:07:11.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:10 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:07:11.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:10 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:07:11.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:10 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:12.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:11 smithi116 ceph-mon[62701]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:07:12.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:11 smithi116 ceph-mon[62701]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T01:07:12.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:11 smithi116 ceph-mon[62701]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:11 smithi162 ceph-mon[67178]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:07:12.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:11 smithi162 ceph-mon[67178]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T01:07:12.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:11 smithi162 ceph-mon[67178]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:12.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:11 smithi156 ceph-mon[66799]: from='client.14334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:07:12.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:11 smithi156 ceph-mon[66799]: osdmap e41: 6 total, 6 up, 6 in 2024-03-19T01:07:12.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:11 smithi156 ceph-mon[66799]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:13.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:12 smithi116 ceph-mon[62701]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T01:07:13.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:12 smithi162 ceph-mon[67178]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T01:07:13.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:12 smithi156 ceph-mon[66799]: osdmap e42: 6 total, 6 up, 6 in 2024-03-19T01:07:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:13 smithi162 ceph-mon[67178]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:13 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3229303637' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7db79fde-7c61-4d3e-adb5-78b906eb3173"}]: dispatch 2024-03-19T01:07:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:13 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3229303637' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7db79fde-7c61-4d3e-adb5-78b906eb3173"}]': finished 2024-03-19T01:07:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:13 smithi162 ceph-mon[67178]: osdmap e43: 7 total, 6 up, 7 in 2024-03-19T01:07:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:13 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:13 smithi116 ceph-mon[62701]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:13 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3229303637' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7db79fde-7c61-4d3e-adb5-78b906eb3173"}]: dispatch 2024-03-19T01:07:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:13 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3229303637' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7db79fde-7c61-4d3e-adb5-78b906eb3173"}]': finished 2024-03-19T01:07:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:13 smithi116 ceph-mon[62701]: osdmap e43: 7 total, 6 up, 7 in 2024-03-19T01:07:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:13 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:13 smithi156 ceph-mon[66799]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:13 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3229303637' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7db79fde-7c61-4d3e-adb5-78b906eb3173"}]: dispatch 2024-03-19T01:07:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:13 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3229303637' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7db79fde-7c61-4d3e-adb5-78b906eb3173"}]': finished 2024-03-19T01:07:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:13 smithi156 ceph-mon[66799]: osdmap e43: 7 total, 6 up, 7 in 2024-03-19T01:07:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:13 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:15.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:14 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1598502862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:07:15.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:14 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1598502862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:07:15.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:14 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1598502862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:07:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:16 smithi162 ceph-mon[67178]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-19T01:07:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:16 smithi116 ceph-mon[62701]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-19T01:07:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:16 smithi156 ceph-mon[66799]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 114 KiB/s, 0 objects/s recovering 2024-03-19T01:07:18.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:18 smithi162 ceph-mon[67178]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T01:07:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:18 smithi116 ceph-mon[62701]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T01:07:18.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:18 smithi156 ceph-mon[66799]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-19T01:07:20.345 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:20 smithi116 ceph-mon[62701]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T01:07:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:20 smithi162 ceph-mon[67178]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T01:07:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:20 smithi156 ceph-mon[66799]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T01:07:21.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-19T01:07:21.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:21.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-19T01:07:21.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:21.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-19T01:07:21.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:22.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:22 smithi162 ceph-mon[67178]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-19T01:07:22.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:22 smithi162 ceph-mon[67178]: Deploying daemon osd.6 on smithi116 2024-03-19T01:07:22.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:22 smithi116 ceph-mon[62701]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-19T01:07:22.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:22 smithi116 ceph-mon[62701]: Deploying daemon osd.6 on smithi116 2024-03-19T01:07:22.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:22 smithi156 ceph-mon[66799]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-03-19T01:07:22.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:22 smithi156 ceph-mon[66799]: Deploying daemon osd.6 on smithi116 2024-03-19T01:07:24.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:24 smithi162 ceph-mon[67178]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T01:07:24.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:24 smithi116 ceph-mon[62701]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T01:07:24.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:24 smithi156 ceph-mon[66799]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T01:07:26.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:26 smithi162 ceph-mon[67178]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:26.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:26 smithi116 ceph-mon[62701]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:26.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:26 smithi156 ceph-mon[66799]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:28 smithi162 ceph-mon[67178]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:28 smithi116 ceph-mon[62701]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:28 smithi156 ceph-mon[66799]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:30.392 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:30 smithi116 ceph-mon[62701]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:30.398 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:30 smithi162 ceph-mon[67178]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:30.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:30 smithi156 ceph-mon[66799]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:31.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:31.242+0000 7f955bfff700 1 -- 172.21.15.116:0/1578535627 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f956406b4c0 con 0x7f95500a2890 2024-03-19T01:07:31.244 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 6 on host 'smithi116' 2024-03-19T01:07:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:31.245+0000 7f95689bb700 1 -- 172.21.15.116:0/1578535627 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f95500a2890 msgr2=0x7f95500a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:07:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:31.245+0000 7f95689bb700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f95500a2890 0x7f95500a4d50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f954c0099d0 tx=0x7f954c008040 comp rx=0 tx=0).stop 2024-03-19T01:07:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:31.245+0000 7f95689bb700 1 -- 172.21.15.116:0/1578535627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9564101230 msgr2=0x7f95641974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:07:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:31.245+0000 7f95689bb700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9564101230 0x7f95641974b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f95540057c0 tx=0x7f95540057f0 comp rx=0 tx=0).stop 2024-03-19T01:07:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:31.245+0000 7f95689bb700 1 -- 172.21.15.116:0/1578535627 shutdown_connections 2024-03-19T01:07:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:31.245+0000 7f95689bb700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f95500a2890 0x7f95500a4d50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:31.245+0000 7f95689bb700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9564102dd0 0x7f956419ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:31.245+0000 7f95689bb700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9564101230 0x7f95641974b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:31.245+0000 7f95689bb700 1 --2- 172.21.15.116:0/1578535627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9564102430 0x7f95641979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:31.245+0000 7f95689bb700 1 -- 172.21.15.116:0/1578535627 >> 172.21.15.116:0/1578535627 conn(0x7f95640fc7d0 msgr2=0x7f9564105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:07:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:31.245+0000 7f95689bb700 1 -- 172.21.15.116:0/1578535627 shutdown_connections 2024-03-19T01:07:31.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:07:31.245+0000 7f95689bb700 1 -- 172.21.15.116:0/1578535627 wait complete. 2024-03-19T01:07:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:31 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:31 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:31 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:31 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:07:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:31 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:31 smithi156 ceph-mon[66799]: from='osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T01:07:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:31 smithi156 ceph-mon[66799]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:31.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:31.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:31.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:31.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:07:31.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:31.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:31 smithi116 ceph-mon[62701]: from='osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T01:07:31.624 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:31 smithi116 ceph-mon[62701]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:31 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:31 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:31 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:31 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:07:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:31 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:31 smithi162 ceph-mon[67178]: from='osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-19T01:07:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:31 smithi162 ceph-mon[67178]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:32.058 DEBUG:teuthology.orchestra.run.smithi116:osd.6> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.6.service 2024-03-19T01:07:32.060 INFO:tasks.cephadm:Deploying osd.7 on smithi156 with /dev/vg_nvme/lv_2... 2024-03-19T01:07:32.060 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T01:07:32.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:32 smithi116 ceph-mon[62701]: from='osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-19T01:07:32.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:32 smithi116 ceph-mon[62701]: osdmap e44: 7 total, 6 up, 7 in 2024-03-19T01:07:32.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:32 smithi116 ceph-mon[62701]: from='osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T01:07:32.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:32.298 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:32.298 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:32.298 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:32.298 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:07:32 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:07:32.211+0000 7fb3265c7700 -1 osd.6 0 waiting for initial osdmap 2024-03-19T01:07:32.298 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:07:32 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:07:32.218+0000 7fb3223ba700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:07:32.306 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T01:07:32.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:32 smithi156 ceph-mon[66799]: from='osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-19T01:07:32.428 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:32 smithi156 ceph-mon[66799]: osdmap e44: 7 total, 6 up, 7 in 2024-03-19T01:07:32.428 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:32 smithi156 ceph-mon[66799]: from='osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T01:07:32.428 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:32.428 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:32.428 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:32.428 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:32 smithi162 ceph-mon[67178]: from='osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-19T01:07:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:32 smithi162 ceph-mon[67178]: osdmap e44: 7 total, 6 up, 7 in 2024-03-19T01:07:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:32 smithi162 ceph-mon[67178]: from='osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T01:07:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:33.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:33 smithi156 ceph-mon[66799]: from='osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T01:07:33.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:33 smithi156 ceph-mon[66799]: osdmap e45: 7 total, 6 up, 7 in 2024-03-19T01:07:33.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:33 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:33.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:33 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:33.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:33 smithi156 ceph-mon[66799]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:33.599 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:33 smithi116 ceph-mon[62701]: from='osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T01:07:33.599 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:33 smithi116 ceph-mon[62701]: osdmap e45: 7 total, 6 up, 7 in 2024-03-19T01:07:33.600 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:33 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:33.600 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:33 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:33.600 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:33 smithi116 ceph-mon[62701]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:33 smithi162 ceph-mon[67178]: from='osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T01:07:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:33 smithi162 ceph-mon[67178]: osdmap e45: 7 total, 6 up, 7 in 2024-03-19T01:07:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:33 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:33 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:33 smithi162 ceph-mon[67178]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-19T01:07:34.439 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T01:07:34.440 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T01:07:34.440 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-03-19T01:07:34.440 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-03-19T01:07:34.440 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00863032 s, 1.2 GB/s 2024-03-19T01:07:34.440 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-03-19T01:07:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:34 smithi156 ceph-mon[66799]: purged_snaps scrub starts 2024-03-19T01:07:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:34 smithi156 ceph-mon[66799]: purged_snaps scrub ok 2024-03-19T01:07:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:34 smithi156 ceph-mon[66799]: osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241] boot 2024-03-19T01:07:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:34 smithi156 ceph-mon[66799]: osdmap e46: 7 total, 7 up, 7 in 2024-03-19T01:07:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:34 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:34.552 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:34 smithi116 ceph-mon[62701]: purged_snaps scrub starts 2024-03-19T01:07:34.552 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:34 smithi116 ceph-mon[62701]: purged_snaps scrub ok 2024-03-19T01:07:34.553 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:34 smithi116 ceph-mon[62701]: osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241] boot 2024-03-19T01:07:34.553 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:34 smithi116 ceph-mon[62701]: osdmap e46: 7 total, 7 up, 7 in 2024-03-19T01:07:34.553 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:34 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:34 smithi162 ceph-mon[67178]: purged_snaps scrub starts 2024-03-19T01:07:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:34 smithi162 ceph-mon[67178]: purged_snaps scrub ok 2024-03-19T01:07:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:34 smithi162 ceph-mon[67178]: osd.6 [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241] boot 2024-03-19T01:07:34.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:34 smithi162 ceph-mon[67178]: osdmap e46: 7 total, 7 up, 7 in 2024-03-19T01:07:34.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:34 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-19T01:07:35.120 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi156:vg_nvme/lv_2 2024-03-19T01:07:35.365 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T01:07:35.458 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:35 smithi156 ceph-mon[66799]: osdmap e47: 7 total, 7 up, 7 in 2024-03-19T01:07:35.459 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:35 smithi156 ceph-mon[66799]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:35 smithi162 ceph-mon[67178]: osdmap e47: 7 total, 7 up, 7 in 2024-03-19T01:07:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:35 smithi162 ceph-mon[67178]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:35.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:35 smithi116 ceph-mon[62701]: osdmap e47: 7 total, 7 up, 7 in 2024-03-19T01:07:35.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:35 smithi116 ceph-mon[62701]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:36.646 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:36 smithi156 ceph-mon[66799]: Detected new or changed devices on smithi116 2024-03-19T01:07:36.646 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:36 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:36.646 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:36 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:36.646 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:36 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:36.646 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:36 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:36.646 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:36 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:36.646 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:36 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi116 to 5696M 2024-03-19T01:07:36.646 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:36 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:36.646 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:36 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:36.647 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:36 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:07:36.647 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:36 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:36.867 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.864+0000 7efdcfcd1700 1 -- 172.21.15.156:0/3874283979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdc8103530 msgr2=0x7efdc8103990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:07:36.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.864+0000 7efdcfcd1700 1 --2- 172.21.15.156:0/3874283979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdc8103530 0x7efdc8103990 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efdc40099c0 tx=0x7efdc4009cd0 comp rx=0 tx=0).stop 2024-03-19T01:07:36.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.864+0000 7efdcfcd1700 1 -- 172.21.15.156:0/3874283979 shutdown_connections 2024-03-19T01:07:36.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.864+0000 7efdcfcd1700 1 --2- 172.21.15.156:0/3874283979 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdc8103ed0 0x7efdc810a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:36.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.864+0000 7efdcfcd1700 1 --2- 172.21.15.156:0/3874283979 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdc8102330 0x7efdc8102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:36.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.864+0000 7efdcfcd1700 1 --2- 172.21.15.156:0/3874283979 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdc8103530 0x7efdc8103990 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:36.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.864+0000 7efdcfcd1700 1 -- 172.21.15.156:0/3874283979 >> 172.21.15.156:0/3874283979 conn(0x7efdc80fd8f0 msgr2=0x7efdc80ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:07:36.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.865+0000 7efdcfcd1700 1 -- 172.21.15.156:0/3874283979 shutdown_connections 2024-03-19T01:07:36.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.865+0000 7efdcfcd1700 1 -- 172.21.15.156:0/3874283979 wait complete. 2024-03-19T01:07:36.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.865+0000 7efdcfcd1700 1 Processor -- start 2024-03-19T01:07:36.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.865+0000 7efdcfcd1700 1 -- start start 2024-03-19T01:07:36.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.865+0000 7efdcfcd1700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdc8102330 0x7efdc8198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:07:36.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.865+0000 7efdcfcd1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdc8103530 0x7efdc8198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:07:36.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.865+0000 7efdcfcd1700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdc8103ed0 0x7efdc819fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:07:36.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.865+0000 7efdcfcd1700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efdc810df20 con 0x7efdc8102330 2024-03-19T01:07:36.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.865+0000 7efdcfcd1700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efdc810dda0 con 0x7efdc8103530 2024-03-19T01:07:36.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.865+0000 7efdcfcd1700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efdc810e0a0 con 0x7efdc8103ed0 2024-03-19T01:07:36.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.865+0000 7efdcd26c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdc8103530 0x7efdc8198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:07:36.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.865+0000 7efdcd26c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdc8103530 0x7efdc8198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45440/0 (socket says 172.21.15.156:45440) 2024-03-19T01:07:36.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.865+0000 7efdcd26c700 1 -- 172.21.15.156:0/1489748953 learned_addr learned my addr 172.21.15.156:0/1489748953 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-19T01:07:36.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.866+0000 7efdcda6d700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdc8102330 0x7efdc8198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:07:36.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.866+0000 7efdce26e700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdc8103ed0 0x7efdc819fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:07:36.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.866+0000 7efdcd26c700 1 -- 172.21.15.156:0/1489748953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdc8103ed0 msgr2=0x7efdc819fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:07:36.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.866+0000 7efdcd26c700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdc8103ed0 0x7efdc819fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:36.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.866+0000 7efdcd26c700 1 -- 172.21.15.156:0/1489748953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdc8102330 msgr2=0x7efdc8198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:07:36.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.866+0000 7efdcd26c700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdc8102330 0x7efdc8198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:36.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.866+0000 7efdcd26c700 1 -- 172.21.15.156:0/1489748953 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc81a01a0 con 0x7efdc8103530 2024-03-19T01:07:36.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.866+0000 7efdcd26c700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdc8103530 0x7efdc8198a80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efdc4005910 tx=0x7efdc4009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:07:36.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.866+0000 7efdbeffd700 1 -- 172.21.15.156:0/1489748953 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc4027070 con 0x7efdc8103530 2024-03-19T01:07:36.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.866+0000 7efdbeffd700 1 -- 172.21.15.156:0/1489748953 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdc4004150 con 0x7efdc8103530 2024-03-19T01:07:36.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.866+0000 7efdcfcd1700 1 -- 172.21.15.156:0/1489748953 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdc81a0430 con 0x7efdc8103530 2024-03-19T01:07:36.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.866+0000 7efdbeffd700 1 -- 172.21.15.156:0/1489748953 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc4022e60 con 0x7efdc8103530 2024-03-19T01:07:36.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.866+0000 7efdcfcd1700 1 -- 172.21.15.156:0/1489748953 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdc81a08e0 con 0x7efdc8103530 2024-03-19T01:07:36.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.867+0000 7efdcfcd1700 1 -- 172.21.15.156:0/1489748953 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdc8068fd0 con 0x7efdc8103530 2024-03-19T01:07:36.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.868+0000 7efdbeffd700 1 -- 172.21.15.156:0/1489748953 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7efdc402a070 con 0x7efdc8103530 2024-03-19T01:07:36.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.869+0000 7efdbeffd700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efdb40a2910 0x7efdb40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:07:36.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.869+0000 7efdbeffd700 1 -- 172.21.15.156:0/1489748953 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7efdc40cebb0 con 0x7efdc8103530 2024-03-19T01:07:36.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.869+0000 7efdcda6d700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efdb40a2910 0x7efdb40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:07:36.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.870+0000 7efdcda6d700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efdb40a2910 0x7efdb40a4dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efdb8005950 tx=0x7efdb800a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:07:36.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:36.872+0000 7efdbeffd700 1 -- 172.21.15.156:0/1489748953 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efdc4013070 con 0x7efdc8103530 2024-03-19T01:07:36.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:36 smithi162 ceph-mon[67178]: Detected new or changed devices on smithi116 2024-03-19T01:07:36.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:36 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:36.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:36 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:36.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:36 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:36.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:36 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:36.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:36 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:36.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:36 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi116 to 5696M 2024-03-19T01:07:36.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:36 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:36.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:36 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:36.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:36 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:07:36.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:36 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:36.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:36 smithi116 ceph-mon[62701]: Detected new or changed devices on smithi116 2024-03-19T01:07:36.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:36 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:36.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:36 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:36.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:36 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:36.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:36 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:36.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:36 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:07:36.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:36 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi116 to 5696M 2024-03-19T01:07:36.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:36 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:36.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:36 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:36.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:36 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:07:36.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:36 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:37.077 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:37.076+0000 7efdcfcd1700 1 -- 172.21.15.156:0/1489748953 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7efdc8062750 con 0x7efdb40a2910 2024-03-19T01:07:37.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:37 smithi156 ceph-mon[66799]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:37.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:37 smithi156 ceph-mon[66799]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:07:37.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:37 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:07:37.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:37 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:07:37.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:37 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:37.800 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:37 smithi162 ceph-mon[67178]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:37.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:37 smithi162 ceph-mon[67178]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:07:37.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:37 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:07:37.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:37 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:07:37.801 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:37 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:37.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:37 smithi116 ceph-mon[62701]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:37.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:37 smithi116 ceph-mon[62701]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:07:37.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:37 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:07:37.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:37 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:07:37.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:37 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:40.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:40 smithi156 ceph-mon[66799]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:40.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:40 smithi156 ceph-mon[66799]: from='client.? 172.21.15.156:0/3268680289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e27814d-4a85-4dcd-8640-a09b2e55df23"}]: dispatch 2024-03-19T01:07:40.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:40 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e27814d-4a85-4dcd-8640-a09b2e55df23"}]: dispatch 2024-03-19T01:07:40.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:40 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e27814d-4a85-4dcd-8640-a09b2e55df23"}]': finished 2024-03-19T01:07:40.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:40 smithi156 ceph-mon[66799]: osdmap e48: 8 total, 7 up, 8 in 2024-03-19T01:07:40.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:40 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:07:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:40 smithi162 ceph-mon[67178]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:40 smithi162 ceph-mon[67178]: from='client.? 172.21.15.156:0/3268680289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e27814d-4a85-4dcd-8640-a09b2e55df23"}]: dispatch 2024-03-19T01:07:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:40 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e27814d-4a85-4dcd-8640-a09b2e55df23"}]: dispatch 2024-03-19T01:07:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:40 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e27814d-4a85-4dcd-8640-a09b2e55df23"}]': finished 2024-03-19T01:07:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:40 smithi162 ceph-mon[67178]: osdmap e48: 8 total, 7 up, 8 in 2024-03-19T01:07:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:40 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:07:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:40 smithi116 ceph-mon[62701]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:40 smithi116 ceph-mon[62701]: from='client.? 172.21.15.156:0/3268680289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e27814d-4a85-4dcd-8640-a09b2e55df23"}]: dispatch 2024-03-19T01:07:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:40 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e27814d-4a85-4dcd-8640-a09b2e55df23"}]: dispatch 2024-03-19T01:07:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:40 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e27814d-4a85-4dcd-8640-a09b2e55df23"}]': finished 2024-03-19T01:07:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:40 smithi116 ceph-mon[62701]: osdmap e48: 8 total, 7 up, 8 in 2024-03-19T01:07:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:40 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:07:41.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:41 smithi162 ceph-mon[67178]: from='client.? 172.21.15.156:0/3590222787' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:07:41.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:41 smithi116 ceph-mon[62701]: from='client.? 172.21.15.156:0/3590222787' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:07:41.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:41 smithi156 ceph-mon[66799]: from='client.? 172.21.15.156:0/3590222787' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:07:42.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:42 smithi162 ceph-mon[67178]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:42.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:42 smithi116 ceph-mon[62701]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:42 smithi156 ceph-mon[66799]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:44 smithi162 ceph-mon[67178]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:44.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:44 smithi116 ceph-mon[62701]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:44.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:44 smithi156 ceph-mon[66799]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:46.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:46 smithi162 ceph-mon[67178]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:46.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:46 smithi116 ceph-mon[62701]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:46.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:46 smithi156 ceph-mon[66799]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:48.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:48 smithi162 ceph-mon[67178]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:48.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:48 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-19T01:07:48.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:48 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:48.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:48 smithi116 ceph-mon[62701]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:48.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:48 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-19T01:07:48.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:48 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:48.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:48 smithi156 ceph-mon[66799]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:48.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:48 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-19T01:07:48.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:48 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:07:49.354 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:49 smithi156 ceph-mon[66799]: Deploying daemon osd.7 on smithi156 2024-03-19T01:07:49.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:49 smithi162 ceph-mon[67178]: Deploying daemon osd.7 on smithi156 2024-03-19T01:07:49.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:49 smithi116 ceph-mon[62701]: Deploying daemon osd.7 on smithi156 2024-03-19T01:07:50.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:50 smithi162 ceph-mon[67178]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:50.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:50 smithi116 ceph-mon[62701]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:50.469 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:50 smithi156 ceph-mon[66799]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:52.168 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:52 smithi156 ceph-mon[66799]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:52.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:52 smithi162 ceph-mon[67178]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:52.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:52 smithi116 ceph-mon[62701]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:54.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:54.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:54.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:54.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:53 smithi162 ceph-mon[67178]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:54.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:54.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:54.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:54.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:53 smithi116 ceph-mon[62701]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:54.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:54.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:54.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:07:54.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:53 smithi156 ceph-mon[66799]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:56.148 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:56 smithi156 ceph-mon[66799]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:56.239 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:56.238+0000 7efdbeffd700 1 -- 172.21.15.156:0/1489748953 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7efdc8062750 con 0x7efdb40a2910 2024-03-19T01:07:56.240 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 7 on host 'smithi156' 2024-03-19T01:07:56.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:56.240+0000 7efdcfcd1700 1 -- 172.21.15.156:0/1489748953 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efdb40a2910 msgr2=0x7efdb40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:07:56.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:56.240+0000 7efdcfcd1700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efdb40a2910 0x7efdb40a4dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efdb8005950 tx=0x7efdb800a300 comp rx=0 tx=0).stop 2024-03-19T01:07:56.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:56.240+0000 7efdcfcd1700 1 -- 172.21.15.156:0/1489748953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdc8103530 msgr2=0x7efdc8198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:07:56.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:56.240+0000 7efdcfcd1700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdc8103530 0x7efdc8198a80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efdc4005910 tx=0x7efdc4009580 comp rx=0 tx=0).stop 2024-03-19T01:07:56.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:56.240+0000 7efdcfcd1700 1 -- 172.21.15.156:0/1489748953 shutdown_connections 2024-03-19T01:07:56.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:56.240+0000 7efdcfcd1700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efdb40a2910 0x7efdb40a4dd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:56.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:56.240+0000 7efdcfcd1700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efdc8103ed0 0x7efdc819fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:56.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:56.240+0000 7efdcfcd1700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdc8102330 0x7efdc8198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:56.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:56.240+0000 7efdcfcd1700 1 --2- 172.21.15.156:0/1489748953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efdc8103530 0x7efdc8198a80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:07:56.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:56.240+0000 7efdcfcd1700 1 -- 172.21.15.156:0/1489748953 >> 172.21.15.156:0/1489748953 conn(0x7efdc80fd8f0 msgr2=0x7efdc8106900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:07:56.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:56.241+0000 7efdcfcd1700 1 -- 172.21.15.156:0/1489748953 shutdown_connections 2024-03-19T01:07:56.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:07:56.241+0000 7efdcfcd1700 1 -- 172.21.15.156:0/1489748953 wait complete. 2024-03-19T01:07:56.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:56 smithi162 ceph-mon[67178]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:56.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:56 smithi116 ceph-mon[62701]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:56.870 DEBUG:teuthology.orchestra.run.smithi156:osd.7> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.7.service 2024-03-19T01:07:56.873 INFO:tasks.cephadm:Deploying osd.8 on smithi162 with /dev/vg_nvme/lv_2... 2024-03-19T01:07:56.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-19T01:07:57.120 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:07:57.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:57 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:57 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:57 smithi156 ceph-mon[66799]: from='osd.7 [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T01:07:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:57 smithi156 ceph-mon[66799]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T01:07:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:57 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:57 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:57 smithi156 ceph-mon[66799]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:57.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:57 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:57.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:57 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:57.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:57 smithi162 ceph-mon[67178]: from='osd.7 [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T01:07:57.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:57 smithi162 ceph-mon[67178]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T01:07:57.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:57 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:57.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:57 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:57.488 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:57 smithi162 ceph-mon[67178]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:57 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:57 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:57 smithi116 ceph-mon[62701]: from='osd.7 [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T01:07:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:57 smithi116 ceph-mon[62701]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-19T01:07:57.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:57 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:57.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:57 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:07:57.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:57 smithi116 ceph-mon[62701]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:58.340 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:58 smithi162 ceph-mon[67178]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-19T01:07:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:58 smithi162 ceph-mon[67178]: osdmap e49: 8 total, 7 up, 8 in 2024-03-19T01:07:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:58 smithi162 ceph-mon[67178]: from='osd.7 [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:07:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:07:58.341 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:58 smithi162 ceph-mon[67178]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:07:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:58 smithi156 ceph-mon[66799]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-19T01:07:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:58 smithi156 ceph-mon[66799]: osdmap e49: 8 total, 7 up, 8 in 2024-03-19T01:07:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:58 smithi156 ceph-mon[66799]: from='osd.7 [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:07:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:07:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:58 smithi156 ceph-mon[66799]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:07:58.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:58 smithi116 ceph-mon[62701]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-19T01:07:58.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:58 smithi116 ceph-mon[62701]: osdmap e49: 8 total, 7 up, 8 in 2024-03-19T01:07:58.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:58 smithi116 ceph-mon[62701]: from='osd.7 [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:07:58.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:07:58.658 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:58 smithi116 ceph-mon[62701]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:07:58.938 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:07:58 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:07:58.629+0000 7fa814d9b700 -1 osd.7 0 waiting for initial osdmap 2024-03-19T01:07:58.938 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:07:58 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:07:58.639+0000 7fa80cb86700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:07:59.171 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-19T01:07:59.171 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-19T01:07:59.171 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-03-19T01:07:59.171 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-03-19T01:07:59.171 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112987 s, 928 MB/s 2024-03-19T01:07:59.171 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-03-19T01:07:59.472 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:59 smithi162 ceph-mon[67178]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-19T01:07:59.472 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:59 smithi162 ceph-mon[67178]: osdmap e50: 8 total, 7 up, 8 in 2024-03-19T01:07:59.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:07:59.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:07:59.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:59 smithi162 ceph-mon[67178]: from='osd.7 ' entity='osd.7' 2024-03-19T01:07:59.473 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:07:59 smithi162 ceph-mon[67178]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:59.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:59 smithi156 ceph-mon[66799]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-19T01:07:59.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:59 smithi156 ceph-mon[66799]: osdmap e50: 8 total, 7 up, 8 in 2024-03-19T01:07:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:07:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:07:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:59 smithi156 ceph-mon[66799]: from='osd.7 ' entity='osd.7' 2024-03-19T01:07:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:07:59 smithi156 ceph-mon[66799]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:59 smithi116 ceph-mon[62701]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-19T01:07:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:59 smithi116 ceph-mon[62701]: osdmap e50: 8 total, 7 up, 8 in 2024-03-19T01:07:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:07:59.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:07:59.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:59 smithi116 ceph-mon[62701]: from='osd.7 ' entity='osd.7' 2024-03-19T01:07:59.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:07:59 smithi116 ceph-mon[62701]: pgmap v176: 1 pgs: 1 unknown; 577 KiB data, 208 MiB used, 626 GiB / 626 GiB avail 2024-03-19T01:07:59.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_2 2024-03-19T01:08:00.071 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:08:00.377 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:00 smithi156 ceph-mon[66799]: purged_snaps scrub starts 2024-03-19T01:08:00.377 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:00 smithi156 ceph-mon[66799]: purged_snaps scrub ok 2024-03-19T01:08:00.377 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:00 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:08:00.377 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:00 smithi156 ceph-mon[66799]: osd.7 [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514] boot 2024-03-19T01:08:00.378 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:00 smithi156 ceph-mon[66799]: osdmap e51: 8 total, 8 up, 8 in 2024-03-19T01:08:00.378 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:00 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:08:00.378 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:00 smithi156 ceph-mon[66799]: osdmap e52: 8 total, 8 up, 8 in 2024-03-19T01:08:00.445 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:00 smithi162 ceph-mon[67178]: purged_snaps scrub starts 2024-03-19T01:08:00.445 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:00 smithi162 ceph-mon[67178]: purged_snaps scrub ok 2024-03-19T01:08:00.445 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:00 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:08:00.446 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:00 smithi162 ceph-mon[67178]: osd.7 [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514] boot 2024-03-19T01:08:00.446 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:00 smithi162 ceph-mon[67178]: osdmap e51: 8 total, 8 up, 8 in 2024-03-19T01:08:00.446 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:00 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:08:00.446 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:00 smithi162 ceph-mon[67178]: osdmap e52: 8 total, 8 up, 8 in 2024-03-19T01:08:00.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:00 smithi116 ceph-mon[62701]: purged_snaps scrub starts 2024-03-19T01:08:00.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:00 smithi116 ceph-mon[62701]: purged_snaps scrub ok 2024-03-19T01:08:00.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:00 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:08:00.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:00 smithi116 ceph-mon[62701]: osd.7 [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514] boot 2024-03-19T01:08:00.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:00 smithi116 ceph-mon[62701]: osdmap e51: 8 total, 8 up, 8 in 2024-03-19T01:08:00.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:00 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-19T01:08:00.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:00 smithi116 ceph-mon[62701]: osdmap e52: 8 total, 8 up, 8 in 2024-03-19T01:08:01.613 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: Detected new or changed devices on smithi156 2024-03-19T01:08:01.613 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.613 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.613 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:01.613 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:01.613 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:01.613 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi156 to 5693M 2024-03-19T01:08:01.613 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:01.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:01.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:01.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:01.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:01.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: osdmap e53: 8 total, 8 up, 8 in 2024-03-19T01:08:01.614 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:01 smithi162 ceph-mon[67178]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: Detected new or changed devices on smithi156 2024-03-19T01:08:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:01.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi156 to 5693M 2024-03-19T01:08:01.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:01.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:01.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:01.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:01.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:01.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: osdmap e53: 8 total, 8 up, 8 in 2024-03-19T01:08:01.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:01 smithi156 ceph-mon[66799]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:01.831 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.829+0000 7fe884303700 1 -- 172.21.15.162:0/2288796723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe87c102360 msgr2=0x7fe87c102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:08:01.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.829+0000 7fe884303700 1 --2- 172.21.15.162:0/2288796723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe87c102360 0x7fe87c102780 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe878009990 tx=0x7fe878009ca0 comp rx=0 tx=0).stop 2024-03-19T01:08:01.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.829+0000 7fe884303700 1 -- 172.21.15.162:0/2288796723 shutdown_connections 2024-03-19T01:08:01.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.829+0000 7fe884303700 1 --2- 172.21.15.162:0/2288796723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe87c103f00 0x7fe87c10a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:01.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.829+0000 7fe884303700 1 --2- 172.21.15.162:0/2288796723 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe87c102360 0x7fe87c102780 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:01.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.829+0000 7fe884303700 1 --2- 172.21.15.162:0/2288796723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c103560 0x7fe87c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:01.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.829+0000 7fe884303700 1 -- 172.21.15.162:0/2288796723 >> 172.21.15.162:0/2288796723 conn(0x7fe87c0fd920 msgr2=0x7fe87c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:08:01.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.829+0000 7fe884303700 1 -- 172.21.15.162:0/2288796723 shutdown_connections 2024-03-19T01:08:01.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.829+0000 7fe884303700 1 -- 172.21.15.162:0/2288796723 wait complete. 2024-03-19T01:08:01.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.829+0000 7fe884303700 1 Processor -- start 2024-03-19T01:08:01.832 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.829+0000 7fe884303700 1 -- start start 2024-03-19T01:08:01.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe884303700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c102360 0x7fe87c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:08:01.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe884303700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe87c103560 0x7fe87c198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:08:01.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe884303700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe87c103f00 0x7fe87c19fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:08:01.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe884303700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c10df50 con 0x7fe87c103f00 2024-03-19T01:08:01.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe884303700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c10ddd0 con 0x7fe87c102360 2024-03-19T01:08:01.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe884303700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c10e0d0 con 0x7fe87c103560 2024-03-19T01:08:01.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe88189e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe87c103560 0x7fe87c198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:08:01.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe88209f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c102360 0x7fe87c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:08:01.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe88189e700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe87c103560 0x7fe87c198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:59278/0 (socket says 172.21.15.162:59278) 2024-03-19T01:08:01.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe88189e700 1 -- 172.21.15.162:0/1595315589 learned_addr learned my addr 172.21.15.162:0/1595315589 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T01:08:01.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe8828a0700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe87c103f00 0x7fe87c19fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:08:01.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe88189e700 1 -- 172.21.15.162:0/1595315589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c102360 msgr2=0x7fe87c198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:08:01.833 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe88189e700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c102360 0x7fe87c198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:01.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe88189e700 1 -- 172.21.15.162:0/1595315589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe87c103f00 msgr2=0x7fe87c19fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:08:01.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe88189e700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe87c103f00 0x7fe87c19fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:01.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe88189e700 1 -- 172.21.15.162:0/1595315589 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c1a01f0 con 0x7fe87c103560 2024-03-19T01:08:01.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe8828a0700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe87c103f00 0x7fe87c19fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:08:01.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe88189e700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe87c103560 0x7fe87c198ad0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00d9b0 tx=0x7fe86c00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:08:01.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe8737fe700 1 -- 172.21.15.162:0/1595315589 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c017070 con 0x7fe87c103560 2024-03-19T01:08:01.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe884303700 1 -- 172.21.15.162:0/1595315589 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87c1a04e0 con 0x7fe87c103560 2024-03-19T01:08:01.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe8737fe700 1 -- 172.21.15.162:0/1595315589 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe86c005020 con 0x7fe87c103560 2024-03-19T01:08:01.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe8737fe700 1 -- 172.21.15.162:0/1595315589 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c009bd0 con 0x7fe87c103560 2024-03-19T01:08:01.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.830+0000 7fe884303700 1 -- 172.21.15.162:0/1595315589 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe87c1a0a20 con 0x7fe87c103560 2024-03-19T01:08:01.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.831+0000 7fe884303700 1 -- 172.21.15.162:0/1595315589 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe87c068fd0 con 0x7fe87c103560 2024-03-19T01:08:01.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.833+0000 7fe8737fe700 1 -- 172.21.15.162:0/1595315589 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fe86c0051d0 con 0x7fe87c103560 2024-03-19T01:08:01.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.834+0000 7fe8737fe700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe8680a2890 0x7fe8680a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:08:01.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.834+0000 7fe8737fe700 1 -- 172.21.15.162:0/1595315589 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7fe86c0ce280 con 0x7fe87c103560 2024-03-19T01:08:01.835 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.834+0000 7fe88209f700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe8680a2890 0x7fe8680a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:08:01.836 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.835+0000 7fe88209f700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe8680a2890 0x7fe8680a4d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe878003890 tx=0x7fe878005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:08:01.838 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:01.837+0000 7fe8737fe700 1 -- 172.21.15.162:0/1595315589 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe86c020020 con 0x7fe87c103560 2024-03-19T01:08:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: Detected new or changed devices on smithi156 2024-03-19T01:08:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi156 to 5693M 2024-03-19T01:08:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:01.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:01.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:01.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:01.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: osdmap e53: 8 total, 8 up, 8 in 2024-03-19T01:08:01.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:01 smithi116 ceph-mon[62701]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:02.046 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:02.046+0000 7fe884303700 1 -- 172.21.15.162:0/1595315589 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe87c062750 con 0x7fe8680a2890 2024-03-19T01:08:03.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:02 smithi156 ceph-mon[66799]: osdmap e54: 8 total, 8 up, 8 in 2024-03-19T01:08:03.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:02 smithi156 ceph-mon[66799]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:08:03.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:02 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:08:03.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:02 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:08:03.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:02 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:03.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:02 smithi162 ceph-mon[67178]: osdmap e54: 8 total, 8 up, 8 in 2024-03-19T01:08:03.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:02 smithi162 ceph-mon[67178]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:08:03.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:02 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:08:03.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:02 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:08:03.305 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:02 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:03.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:02 smithi116 ceph-mon[62701]: osdmap e54: 8 total, 8 up, 8 in 2024-03-19T01:08:03.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:02 smithi116 ceph-mon[62701]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:08:03.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:02 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:08:03.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:02 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:08:03.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:02 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:04.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:03 smithi156 ceph-mon[66799]: osdmap e55: 8 total, 8 up, 8 in 2024-03-19T01:08:04.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:03 smithi156 ceph-mon[66799]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:03 smithi162 ceph-mon[67178]: osdmap e55: 8 total, 8 up, 8 in 2024-03-19T01:08:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:03 smithi162 ceph-mon[67178]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:04.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:03 smithi116 ceph-mon[62701]: osdmap e55: 8 total, 8 up, 8 in 2024-03-19T01:08:04.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:03 smithi116 ceph-mon[62701]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:05.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:04 smithi156 ceph-mon[66799]: from='client.? 172.21.15.162:0/1917523099' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9660b3b0-ed99-4050-a339-1beea2450c79"}]: dispatch 2024-03-19T01:08:05.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:04 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9660b3b0-ed99-4050-a339-1beea2450c79"}]: dispatch 2024-03-19T01:08:05.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:04 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9660b3b0-ed99-4050-a339-1beea2450c79"}]': finished 2024-03-19T01:08:05.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:04 smithi156 ceph-mon[66799]: osdmap e56: 9 total, 8 up, 9 in 2024-03-19T01:08:05.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:04 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:04 smithi162 ceph-mon[67178]: from='client.? 172.21.15.162:0/1917523099' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9660b3b0-ed99-4050-a339-1beea2450c79"}]: dispatch 2024-03-19T01:08:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:04 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9660b3b0-ed99-4050-a339-1beea2450c79"}]: dispatch 2024-03-19T01:08:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:04 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9660b3b0-ed99-4050-a339-1beea2450c79"}]': finished 2024-03-19T01:08:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:04 smithi162 ceph-mon[67178]: osdmap e56: 9 total, 8 up, 9 in 2024-03-19T01:08:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:04 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:04 smithi116 ceph-mon[62701]: from='client.? 172.21.15.162:0/1917523099' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9660b3b0-ed99-4050-a339-1beea2450c79"}]: dispatch 2024-03-19T01:08:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:04 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9660b3b0-ed99-4050-a339-1beea2450c79"}]: dispatch 2024-03-19T01:08:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:04 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9660b3b0-ed99-4050-a339-1beea2450c79"}]': finished 2024-03-19T01:08:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:04 smithi116 ceph-mon[62701]: osdmap e56: 9 total, 8 up, 9 in 2024-03-19T01:08:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:05 smithi156 ceph-mon[66799]: pgmap v185: 1 pgs: 1 unknown; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:05 smithi156 ceph-mon[66799]: from='client.? 172.21.15.162:0/3657601539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:08:06.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:05 smithi162 ceph-mon[67178]: pgmap v185: 1 pgs: 1 unknown; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:06.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:05 smithi162 ceph-mon[67178]: from='client.? 172.21.15.162:0/3657601539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:08:06.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:05 smithi116 ceph-mon[62701]: pgmap v185: 1 pgs: 1 unknown; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:06.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:05 smithi116 ceph-mon[62701]: from='client.? 172.21.15.162:0/3657601539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:08:08.342 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:08 smithi162 ceph-mon[67178]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:08.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:08 smithi116 ceph-mon[62701]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:08.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:08 smithi156 ceph-mon[66799]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:10.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:10 smithi162 ceph-mon[67178]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:10.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:10 smithi116 ceph-mon[62701]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:10 smithi156 ceph-mon[66799]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:12 smithi162 ceph-mon[67178]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-19T01:08:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:12.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:12 smithi116 ceph-mon[62701]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:12.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:12 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-19T01:08:12.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:12 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:12.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:12 smithi156 ceph-mon[66799]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:12.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:12 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-19T01:08:12.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:12 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:13.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:13 smithi162 ceph-mon[67178]: Deploying daemon osd.8 on smithi162 2024-03-19T01:08:13.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:13 smithi116 ceph-mon[62701]: Deploying daemon osd.8 on smithi162 2024-03-19T01:08:13.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:13 smithi156 ceph-mon[66799]: Deploying daemon osd.8 on smithi162 2024-03-19T01:08:14.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:14 smithi162 ceph-mon[67178]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:14 smithi116 ceph-mon[62701]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:14.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:14 smithi156 ceph-mon[66799]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:16 smithi162 ceph-mon[67178]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:16 smithi116 ceph-mon[62701]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:16 smithi156 ceph-mon[66799]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:18.129 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:18 smithi162 ceph-mon[67178]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:18.129 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:18 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:18.129 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:18 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:18.129 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:18 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:18 smithi116 ceph-mon[62701]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:18 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:18 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:18 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:18 smithi156 ceph-mon[66799]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:18 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:18 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:18 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:20.233 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:20 smithi162 ceph-mon[67178]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:20.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:20 smithi116 ceph-mon[62701]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:20 smithi156 ceph-mon[66799]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:21.262 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 8 on host 'smithi162' 2024-03-19T01:08:21.263 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:21.261+0000 7fe8737fe700 1 -- 172.21.15.162:0/1595315589 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe87c062750 con 0x7fe8680a2890 2024-03-19T01:08:21.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:21.263+0000 7fe884303700 1 -- 172.21.15.162:0/1595315589 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe8680a2890 msgr2=0x7fe8680a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:08:21.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:21.263+0000 7fe884303700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe8680a2890 0x7fe8680a4d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe878003890 tx=0x7fe878005fd0 comp rx=0 tx=0).stop 2024-03-19T01:08:21.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:21.263+0000 7fe884303700 1 -- 172.21.15.162:0/1595315589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe87c103560 msgr2=0x7fe87c198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:08:21.264 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:21.263+0000 7fe884303700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe87c103560 0x7fe87c198ad0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00d9b0 tx=0x7fe86c00dcc0 comp rx=0 tx=0).stop 2024-03-19T01:08:21.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:21.263+0000 7fe884303700 1 -- 172.21.15.162:0/1595315589 shutdown_connections 2024-03-19T01:08:21.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:21.263+0000 7fe884303700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe87c103f00 0x7fe87c19fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:21.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:21.263+0000 7fe884303700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe87c102360 0x7fe87c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:21.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:21.263+0000 7fe884303700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe8680a2890 0x7fe8680a4d50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:21.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:21.263+0000 7fe884303700 1 --2- 172.21.15.162:0/1595315589 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe87c103560 0x7fe87c198ad0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:21.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:21.263+0000 7fe884303700 1 -- 172.21.15.162:0/1595315589 >> 172.21.15.162:0/1595315589 conn(0x7fe87c0fd920 msgr2=0x7fe87c106930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:08:21.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:21.263+0000 7fe884303700 1 -- 172.21.15.162:0/1595315589 shutdown_connections 2024-03-19T01:08:21.265 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:08:21.263+0000 7fe884303700 1 -- 172.21.15.162:0/1595315589 wait complete. 2024-03-19T01:08:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:21 smithi156 ceph-mon[66799]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:21 smithi156 ceph-mon[66799]: from='osd.8 [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T01:08:21.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:21 smithi156 ceph-mon[66799]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T01:08:21.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:21.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:21.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:21 smithi162 ceph-mon[67178]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:21.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:21.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:21 smithi162 ceph-mon[67178]: from='osd.8 [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T01:08:21.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:21 smithi162 ceph-mon[67178]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T01:08:21.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:21.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:21.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:21 smithi116 ceph-mon[62701]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:21.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:21.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:21.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:21 smithi116 ceph-mon[62701]: from='osd.8 [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T01:08:21.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:21 smithi116 ceph-mon[62701]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-19T01:08:21.968 DEBUG:teuthology.orchestra.run.smithi162:osd.8> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.8.service 2024-03-19T01:08:21.970 INFO:tasks.cephadm:Deploying osd.9 on smithi116 with /dev/vg_nvme/lv_1... 2024-03-19T01:08:21.970 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-19T01:08:22.221 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:08:23.382 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:23 smithi162 ceph-mon[67178]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-19T01:08:23.383 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:23 smithi162 ceph-mon[67178]: osdmap e57: 9 total, 8 up, 9 in 2024-03-19T01:08:23.383 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:23 smithi162 ceph-mon[67178]: from='osd.8 [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:08:23.383 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:23 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:23.383 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:23 smithi162 ceph-mon[67178]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:08:23.383 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:23 smithi162 ceph-mon[67178]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:23 smithi156 ceph-mon[66799]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-19T01:08:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:23 smithi156 ceph-mon[66799]: osdmap e57: 9 total, 8 up, 9 in 2024-03-19T01:08:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:23 smithi156 ceph-mon[66799]: from='osd.8 [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:08:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:23 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:23 smithi156 ceph-mon[66799]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:08:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:23 smithi156 ceph-mon[66799]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:23.514 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:23 smithi116 ceph-mon[62701]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-19T01:08:23.515 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:23 smithi116 ceph-mon[62701]: osdmap e57: 9 total, 8 up, 9 in 2024-03-19T01:08:23.515 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:23 smithi116 ceph-mon[62701]: from='osd.8 [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:08:23.515 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:23 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:23.515 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:23 smithi116 ceph-mon[62701]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:08:23.515 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:23 smithi116 ceph-mon[62701]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:24.149 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:08:23 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:08:23.690+0000 7f6f8d950700 -1 osd.8 0 waiting for initial osdmap 2024-03-19T01:08:24.149 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:08:23 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:08:23.703+0000 7f6f8573b700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:08:24.436 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:24 smithi116 ceph-mon[62701]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T01:08:24.437 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:24 smithi116 ceph-mon[62701]: osdmap e58: 9 total, 8 up, 9 in 2024-03-19T01:08:24.437 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:24 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:24.437 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:24 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:24.437 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:24 smithi116 ceph-mon[62701]: from='osd.8 ' entity='osd.8' 2024-03-19T01:08:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-19T01:08:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-19T01:08:24.437 INFO:teuthology.orchestra.run.smithi116.stderr: stderr: 10+0 records in 2024-03-19T01:08:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:10+0 records out 2024-03-19T01:08:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113154 s, 927 MB/s 2024-03-19T01:08:24.437 INFO:teuthology.orchestra.run.smithi116.stderr:--> Zapping successful for: 2024-03-19T01:08:24.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:24 smithi156 ceph-mon[66799]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T01:08:24.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:24 smithi156 ceph-mon[66799]: osdmap e58: 9 total, 8 up, 9 in 2024-03-19T01:08:24.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:24 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:24.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:24 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:24.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:24 smithi156 ceph-mon[66799]: from='osd.8 ' entity='osd.8' 2024-03-19T01:08:24.647 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:24 smithi162 ceph-mon[67178]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T01:08:24.647 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:24 smithi162 ceph-mon[67178]: osdmap e58: 9 total, 8 up, 9 in 2024-03-19T01:08:24.647 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:24 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:24.647 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:24 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:24.647 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:24 smithi162 ceph-mon[67178]: from='osd.8 ' entity='osd.8' 2024-03-19T01:08:25.125 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi116:vg_nvme/lv_1 2024-03-19T01:08:25.366 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:08:25.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: purged_snaps scrub starts 2024-03-19T01:08:25.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: purged_snaps scrub ok 2024-03-19T01:08:25.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:25.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: osd.8 [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340] boot 2024-03-19T01:08:25.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: osdmap e59: 9 total, 9 up, 9 in 2024-03-19T01:08:25.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:25.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: osdmap e60: 9 total, 9 up, 9 in 2024-03-19T01:08:25.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: Detected new or changed devices on smithi162 2024-03-19T01:08:25.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:25.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:25.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:25.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:25.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:25.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:25.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi162 to 5693M 2024-03-19T01:08:25.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:25.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:25.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:25.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:25.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: purged_snaps scrub starts 2024-03-19T01:08:25.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: purged_snaps scrub ok 2024-03-19T01:08:25.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:25.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: osd.8 [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340] boot 2024-03-19T01:08:25.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: osdmap e59: 9 total, 9 up, 9 in 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: osdmap e60: 9 total, 9 up, 9 in 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: Detected new or changed devices on smithi162 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi162 to 5693M 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:25.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:25.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: purged_snaps scrub starts 2024-03-19T01:08:25.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: purged_snaps scrub ok 2024-03-19T01:08:25.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:25.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: osd.8 [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340] boot 2024-03-19T01:08:25.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: osdmap e59: 9 total, 9 up, 9 in 2024-03-19T01:08:25.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-19T01:08:25.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: osdmap e60: 9 total, 9 up, 9 in 2024-03-19T01:08:25.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: Detected new or changed devices on smithi162 2024-03-19T01:08:25.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-19T01:08:25.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:25.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:25.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:25.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:25.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:25.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi162 to 5693M 2024-03-19T01:08:25.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:25.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:25.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:25.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:27.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.089+0000 7f698a0da700 1 -- 172.21.15.116:0/987639429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6984102390 msgr2=0x7f69841027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:08:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.089+0000 7f698a0da700 1 --2- 172.21.15.116:0/987639429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6984102390 0x7f69841027b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f69740099c0 tx=0x7f6974009cd0 comp rx=0 tx=0).stop 2024-03-19T01:08:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.089+0000 7f698a0da700 1 -- 172.21.15.116:0/987639429 shutdown_connections 2024-03-19T01:08:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.089+0000 7f698a0da700 1 --2- 172.21.15.116:0/987639429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6984103f30 0x7f698410a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.089+0000 7f698a0da700 1 --2- 172.21.15.116:0/987639429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6984102390 0x7f69841027b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.089+0000 7f698a0da700 1 --2- 172.21.15.116:0/987639429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6984103590 0x7f69841039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.089+0000 7f698a0da700 1 -- 172.21.15.116:0/987639429 >> 172.21.15.116:0/987639429 conn(0x7f69840fd930 msgr2=0x7f69840ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:08:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.089+0000 7f698a0da700 1 -- 172.21.15.116:0/987639429 shutdown_connections 2024-03-19T01:08:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.089+0000 7f698a0da700 1 -- 172.21.15.116:0/987639429 wait complete. 2024-03-19T01:08:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.089+0000 7f698a0da700 1 Processor -- start 2024-03-19T01:08:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.089+0000 7f698a0da700 1 -- start start 2024-03-19T01:08:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f698a0da700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6984102390 0x7f69841985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:08:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f698a0da700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6984103590 0x7f6984198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:08:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f698a0da700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6984103f30 0x7f698419fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:08:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f698a0da700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f698410dfd0 con 0x7f6984103f30 2024-03-19T01:08:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f698a0da700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f698410de50 con 0x7f6984102390 2024-03-19T01:08:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f698a0da700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f698410e150 con 0x7f6984103590 2024-03-19T01:08:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f6983fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6984103f30 0x7f698419fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:08:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f6983fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6984103f30 0x7f698419fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54716/0 (socket says 172.21.15.116:54716) 2024-03-19T01:08:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f6983fff700 1 -- 172.21.15.116:0/1493033029 learned_addr learned my addr 172.21.15.116:0/1493033029 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:08:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f69837fe700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6984102390 0x7f69841985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:08:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f697bfff700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6984103590 0x7f6984198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:08:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f6983fff700 1 -- 172.21.15.116:0/1493033029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6984103590 msgr2=0x7f6984198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:08:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f6983fff700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6984103590 0x7f6984198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f6983fff700 1 -- 172.21.15.116:0/1493033029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6984102390 msgr2=0x7f69841985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:08:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f6983fff700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6984102390 0x7f69841985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f6983fff700 1 -- 172.21.15.116:0/1493033029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69841a0210 con 0x7f6984103f30 2024-03-19T01:08:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f6983fff700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6984103f30 0x7f698419fb70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f697c007f10 tx=0x7f697c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:08:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f69817fa700 1 -- 172.21.15.116:0/1493033029 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c004bb0 con 0x7f6984103f30 2024-03-19T01:08:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f698a0da700 1 -- 172.21.15.116:0/1493033029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69840745a0 con 0x7f6984103f30 2024-03-19T01:08:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.090+0000 7f69817fa700 1 -- 172.21.15.116:0/1493033029 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f697c004d50 con 0x7f6984103f30 2024-03-19T01:08:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.091+0000 7f69817fa700 1 -- 172.21.15.116:0/1493033029 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c005a60 con 0x7f6984103f30 2024-03-19T01:08:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.091+0000 7f698a0da700 1 -- 172.21.15.116:0/1493033029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6984074b60 con 0x7f6984103f30 2024-03-19T01:08:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.091+0000 7f698a0da700 1 -- 172.21.15.116:0/1493033029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6984068fd0 con 0x7f6984103f30 2024-03-19T01:08:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.092+0000 7f69817fa700 1 -- 172.21.15.116:0/1493033029 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f697c01a020 con 0x7f6984103f30 2024-03-19T01:08:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.093+0000 7f69817fa700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f69700a2890 0x7f69700a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:08:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.093+0000 7f69837fe700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f69700a2890 0x7f69700a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:08:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.093+0000 7f69817fa700 1 -- 172.21.15.116:0/1493033029 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 5829+0+0 (secure 0 0 0) 0x7f697c0cdf10 con 0x7f6984103f30 2024-03-19T01:08:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.093+0000 7f69837fe700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f69700a2890 0x7f69700a4d50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f697400b580 tx=0x7f6974005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:08:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.097+0000 7f69817fa700 1 -- 172.21.15.116:0/1493033029 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f697c010070 con 0x7f6984103f30 2024-03-19T01:08:27.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:26 smithi116 ceph-mon[62701]: osdmap e61: 9 total, 9 up, 9 in 2024-03-19T01:08:27.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:26 smithi156 ceph-mon[66799]: osdmap e61: 9 total, 9 up, 9 in 2024-03-19T01:08:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:27.303+0000 7f698a0da700 1 -- 172.21.15.116:0/1493033029 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6984062810 con 0x7f69700a2890 2024-03-19T01:08:27.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:26 smithi162 ceph-mon[67178]: osdmap e61: 9 total, 9 up, 9 in 2024-03-19T01:08:28.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:27 smithi116 ceph-mon[62701]: osdmap e62: 9 total, 9 up, 9 in 2024-03-19T01:08:28.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:27 smithi116 ceph-mon[62701]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:28.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:08:28.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:08:28.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:28.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:27 smithi156 ceph-mon[66799]: osdmap e62: 9 total, 9 up, 9 in 2024-03-19T01:08:28.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:27 smithi156 ceph-mon[66799]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:28.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:08:28.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:08:28.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:27 smithi162 ceph-mon[67178]: osdmap e62: 9 total, 9 up, 9 in 2024-03-19T01:08:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:27 smithi162 ceph-mon[67178]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:08:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:08:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:29.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:28 smithi156 ceph-mon[66799]: from='client.14397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:08:29.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:28 smithi156 ceph-mon[66799]: osdmap e63: 9 total, 9 up, 9 in 2024-03-19T01:08:29.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:28 smithi116 ceph-mon[62701]: from='client.14397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:08:29.297 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:28 smithi116 ceph-mon[62701]: osdmap e63: 9 total, 9 up, 9 in 2024-03-19T01:08:29.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:28 smithi162 ceph-mon[67178]: from='client.14397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi116:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:08:29.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:28 smithi162 ceph-mon[67178]: osdmap e63: 9 total, 9 up, 9 in 2024-03-19T01:08:30.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:29 smithi156 ceph-mon[66799]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:30.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:29 smithi162 ceph-mon[67178]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:30.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:29 smithi116 ceph-mon[62701]: pgmap v204: 1 pgs: 1 remapped+peering; 577 KiB data, 275 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:31.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:30 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/4137152709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59d2e3c8-0173-48eb-8eed-303c1dc31822"}]: dispatch 2024-03-19T01:08:31.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:30 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/4137152709' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59d2e3c8-0173-48eb-8eed-303c1dc31822"}]': finished 2024-03-19T01:08:31.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:30 smithi156 ceph-mon[66799]: osdmap e64: 10 total, 9 up, 10 in 2024-03-19T01:08:31.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:30 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:31.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:30 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1044658222' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:08:31.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:30 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/4137152709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59d2e3c8-0173-48eb-8eed-303c1dc31822"}]: dispatch 2024-03-19T01:08:31.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:30 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/4137152709' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59d2e3c8-0173-48eb-8eed-303c1dc31822"}]': finished 2024-03-19T01:08:31.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:30 smithi162 ceph-mon[67178]: osdmap e64: 10 total, 9 up, 10 in 2024-03-19T01:08:31.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:30 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:31.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:30 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1044658222' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:08:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:30 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4137152709' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59d2e3c8-0173-48eb-8eed-303c1dc31822"}]: dispatch 2024-03-19T01:08:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:30 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4137152709' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59d2e3c8-0173-48eb-8eed-303c1dc31822"}]': finished 2024-03-19T01:08:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:30 smithi116 ceph-mon[62701]: osdmap e64: 10 total, 9 up, 10 in 2024-03-19T01:08:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:31.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:30 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1044658222' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:08:32.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:31 smithi156 ceph-mon[66799]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-19T01:08:32.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:31 smithi162 ceph-mon[67178]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-19T01:08:32.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:31 smithi116 ceph-mon[62701]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-19T01:08:34.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:34 smithi116 ceph-mon[62701]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-19T01:08:34.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:34 smithi162 ceph-mon[67178]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-19T01:08:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:34 smithi156 ceph-mon[66799]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-19T01:08:36.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:36 smithi162 ceph-mon[67178]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T01:08:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:36 smithi116 ceph-mon[62701]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T01:08:36.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:36 smithi156 ceph-mon[66799]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-19T01:08:38.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:38 smithi162 ceph-mon[67178]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-19T01:08:38.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:38 smithi116 ceph-mon[62701]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-19T01:08:38.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:38 smithi156 ceph-mon[66799]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-19T01:08:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:39 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-19T01:08:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:39 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:39.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-19T01:08:39.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:39 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:39.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:39 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-19T01:08:39.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:39 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:40 smithi162 ceph-mon[67178]: Deploying daemon osd.9 on smithi116 2024-03-19T01:08:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:40 smithi162 ceph-mon[67178]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T01:08:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:40 smithi116 ceph-mon[62701]: Deploying daemon osd.9 on smithi116 2024-03-19T01:08:40.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:40 smithi116 ceph-mon[62701]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T01:08:40.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:40 smithi156 ceph-mon[66799]: Deploying daemon osd.9 on smithi116 2024-03-19T01:08:40.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:40 smithi156 ceph-mon[66799]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-19T01:08:42.329 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:42 smithi116 ceph-mon[62701]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:42.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:42 smithi162 ceph-mon[67178]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:42 smithi156 ceph-mon[66799]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:44.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:44 smithi116 ceph-mon[62701]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:44 smithi162 ceph-mon[67178]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:44.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:44 smithi156 ceph-mon[66799]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:45.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:45.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:45.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:45.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:46.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:46 smithi162 ceph-mon[67178]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:46.403 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:46 smithi116 ceph-mon[62701]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:46.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:46 smithi156 ceph-mon[66799]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:48 smithi116 ceph-mon[62701]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:48 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:48 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:48 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:48 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:48 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:48.269+0000 7f69817fa700 1 -- 172.21.15.116:0/1493033029 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6984062810 con 0x7f69700a2890 2024-03-19T01:08:48.271 INFO:teuthology.orchestra.run.smithi116.stdout:Created osd(s) 9 on host 'smithi116' 2024-03-19T01:08:48.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:48.272+0000 7f698a0da700 1 -- 172.21.15.116:0/1493033029 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f69700a2890 msgr2=0x7f69700a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:08:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:48.272+0000 7f698a0da700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f69700a2890 0x7f69700a4d50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f697400b580 tx=0x7f6974005fb0 comp rx=0 tx=0).stop 2024-03-19T01:08:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:48.272+0000 7f698a0da700 1 -- 172.21.15.116:0/1493033029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6984103f30 msgr2=0x7f698419fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:08:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:48.272+0000 7f698a0da700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6984103f30 0x7f698419fb70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f697c007f10 tx=0x7f697c00d3b0 comp rx=0 tx=0).stop 2024-03-19T01:08:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:48.272+0000 7f698a0da700 1 -- 172.21.15.116:0/1493033029 shutdown_connections 2024-03-19T01:08:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:48.272+0000 7f698a0da700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6984103f30 0x7f698419fb70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:48.272+0000 7f698a0da700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6984102390 0x7f69841985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:48.272+0000 7f698a0da700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f69700a2890 0x7f69700a4d50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:48.272+0000 7f698a0da700 1 --2- 172.21.15.116:0/1493033029 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6984103590 0x7f6984198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:48.272+0000 7f698a0da700 1 -- 172.21.15.116:0/1493033029 >> 172.21.15.116:0/1493033029 conn(0x7f69840fd930 msgr2=0x7f6984106890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:08:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:48.272+0000 7f698a0da700 1 -- 172.21.15.116:0/1493033029 shutdown_connections 2024-03-19T01:08:48.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:08:48.272+0000 7f698a0da700 1 -- 172.21.15.116:0/1493033029 wait complete. 2024-03-19T01:08:48.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:48 smithi162 ceph-mon[67178]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:48.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:48 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:48.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:48 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:48.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:48 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:48.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:48 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:48.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:48 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:48.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:48 smithi156 ceph-mon[66799]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:48.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:48 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:48.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:48 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:48.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:48 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:48.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:48 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:48.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:48 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:48.882 DEBUG:teuthology.orchestra.run.smithi116:osd.9> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.9.service 2024-03-19T01:08:48.885 INFO:tasks.cephadm:Deploying osd.10 on smithi156 with /dev/vg_nvme/lv_1... 2024-03-19T01:08:48.885 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-19T01:08:49.135 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T01:08:49.502 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:49 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:49.502 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:49 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:49.503 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:49 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:49.503 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:49 smithi156 ceph-mon[66799]: from='osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T01:08:49.503 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:49 smithi156 ceph-mon[66799]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:49 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:49 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:49 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:49 smithi162 ceph-mon[67178]: from='osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T01:08:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:49 smithi162 ceph-mon[67178]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:49 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:08:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:49 smithi116 ceph-mon[62701]: from='osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-19T01:08:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:49 smithi116 ceph-mon[62701]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:50.485 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:50 smithi116 ceph-mon[62701]: from='osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-19T01:08:50.485 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:50 smithi116 ceph-mon[62701]: osdmap e65: 10 total, 9 up, 10 in 2024-03-19T01:08:50.485 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:50 smithi116 ceph-mon[62701]: from='osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T01:08:50.485 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:50 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:50.485 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:08:50 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:08:50.248+0000 7fd92f174700 -1 osd.9 0 waiting for initial osdmap 2024-03-19T01:08:50.485 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:08:50 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:08:50.262+0000 7fd92af67700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:08:50.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:50 smithi156 ceph-mon[66799]: from='osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-19T01:08:50.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:50 smithi156 ceph-mon[66799]: osdmap e65: 10 total, 9 up, 10 in 2024-03-19T01:08:50.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:50 smithi156 ceph-mon[66799]: from='osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T01:08:50.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:50 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:50.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:50 smithi162 ceph-mon[67178]: from='osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-19T01:08:50.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:50 smithi162 ceph-mon[67178]: osdmap e65: 10 total, 9 up, 10 in 2024-03-19T01:08:50.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:50 smithi162 ceph-mon[67178]: from='osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]: dispatch 2024-03-19T01:08:50.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:50 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:51.141 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-19T01:08:51.141 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-19T01:08:51.141 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-03-19T01:08:51.141 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-03-19T01:08:51.142 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00837509 s, 1.3 GB/s 2024-03-19T01:08:51.142 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-03-19T01:08:51.565 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:51 smithi156 ceph-mon[66799]: from='osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T01:08:51.566 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:51 smithi156 ceph-mon[66799]: osdmap e66: 10 total, 9 up, 10 in 2024-03-19T01:08:51.566 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:51 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:51.566 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:51 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:51.566 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:51 smithi156 ceph-mon[66799]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:51 smithi162 ceph-mon[67178]: from='osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T01:08:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:51 smithi162 ceph-mon[67178]: osdmap e66: 10 total, 9 up, 10 in 2024-03-19T01:08:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:51 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:51 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:51 smithi162 ceph-mon[67178]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:51.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:51 smithi116 ceph-mon[62701]: from='osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi116", "root=default"]}]': finished 2024-03-19T01:08:51.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:51 smithi116 ceph-mon[62701]: osdmap e66: 10 total, 9 up, 10 in 2024-03-19T01:08:51.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:51 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:51.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:51 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:51.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:51 smithi116 ceph-mon[62701]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 276 MiB used, 804 GiB / 805 GiB avail 2024-03-19T01:08:51.877 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi156:vg_nvme/lv_1 2024-03-19T01:08:52.124 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T01:08:52.561 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:52 smithi156 ceph-mon[66799]: purged_snaps scrub starts 2024-03-19T01:08:52.561 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:52 smithi156 ceph-mon[66799]: purged_snaps scrub ok 2024-03-19T01:08:52.561 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:52 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:52.561 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:52 smithi156 ceph-mon[66799]: osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116] boot 2024-03-19T01:08:52.561 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:52 smithi156 ceph-mon[66799]: osdmap e67: 10 total, 10 up, 10 in 2024-03-19T01:08:52.561 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:52 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:52.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:52 smithi162 ceph-mon[67178]: purged_snaps scrub starts 2024-03-19T01:08:52.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:52 smithi162 ceph-mon[67178]: purged_snaps scrub ok 2024-03-19T01:08:52.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:52 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:52.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:52 smithi162 ceph-mon[67178]: osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116] boot 2024-03-19T01:08:52.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:52 smithi162 ceph-mon[67178]: osdmap e67: 10 total, 10 up, 10 in 2024-03-19T01:08:52.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:52 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:52.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:52 smithi116 ceph-mon[62701]: purged_snaps scrub starts 2024-03-19T01:08:52.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:52 smithi116 ceph-mon[62701]: purged_snaps scrub ok 2024-03-19T01:08:52.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:52 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:52.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:52 smithi116 ceph-mon[62701]: osd.9 [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116] boot 2024-03-19T01:08:52.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:52 smithi116 ceph-mon[62701]: osdmap e67: 10 total, 10 up, 10 in 2024-03-19T01:08:52.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:52 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-19T01:08:53.587 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: Detected new or changed devices on smithi116 2024-03-19T01:08:53.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:53.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:53.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:53.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:53.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:53.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:53.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-19T01:08:53.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:53.588 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:53.589 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:53.589 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:53.589 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:08:53.589 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:53 smithi156 ceph-mon[66799]: osdmap e68: 10 total, 10 up, 10 in 2024-03-19T01:08:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: Detected new or changed devices on smithi116 2024-03-19T01:08:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:53.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-19T01:08:53.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:53.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:53.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:53.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:53.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:08:53.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:53 smithi162 ceph-mon[67178]: osdmap e68: 10 total, 10 up, 10 in 2024-03-19T01:08:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: Detected new or changed devices on smithi116 2024-03-19T01:08:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:08:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi116 to 4272M 2024-03-19T01:08:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:08:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:08:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:08:53.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:53 smithi116 ceph-mon[62701]: osdmap e68: 10 total, 10 up, 10 in 2024-03-19T01:08:53.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.942+0000 7f87762e6700 1 -- 172.21.15.156:0/583682194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87701080a0 msgr2=0x7f877010a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:08:53.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.942+0000 7f87762e6700 1 --2- 172.21.15.156:0/583682194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87701080a0 0x7f877010a510 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f876800b0a0 tx=0x7f876800b3b0 comp rx=0 tx=0).stop 2024-03-19T01:08:53.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.942+0000 7f87762e6700 1 -- 172.21.15.156:0/583682194 shutdown_connections 2024-03-19T01:08:53.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.942+0000 7f87762e6700 1 --2- 172.21.15.156:0/583682194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87701080a0 0x7f877010a510 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:53.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.942+0000 7f87762e6700 1 --2- 172.21.15.156:0/583682194 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8770102e40 0x7f8770105230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:53.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.942+0000 7f87762e6700 1 --2- 172.21.15.156:0/583682194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8770105770 0x7f8770107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:53.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.942+0000 7f87762e6700 1 -- 172.21.15.156:0/583682194 >> 172.21.15.156:0/583682194 conn(0x7f87700fc870 msgr2=0x7f87700fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:08:53.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.942+0000 7f87762e6700 1 -- 172.21.15.156:0/583682194 shutdown_connections 2024-03-19T01:08:53.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.942+0000 7f87762e6700 1 -- 172.21.15.156:0/583682194 wait complete. 2024-03-19T01:08:53.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f87762e6700 1 Processor -- start 2024-03-19T01:08:53.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f87762e6700 1 -- start start 2024-03-19T01:08:53.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f87762e6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8770102e40 0x7f87701027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:08:53.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f87762e6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8770105770 0x7f8770100df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:08:53.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f87762e6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87701080a0 0x7f8770101330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:08:53.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f87762e6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f877010df50 con 0x7f87701080a0 2024-03-19T01:08:53.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f87762e6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f877010ddd0 con 0x7f8770102e40 2024-03-19T01:08:53.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f87762e6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f877010e0d0 con 0x7f8770105770 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f876ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8770102e40 0x7f87701027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f876ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8770102e40 0x7f87701027a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:59360/0 (socket says 172.21.15.156:59360) 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f876ffff700 1 -- 172.21.15.156:0/2126133207 learned_addr learned my addr 172.21.15.156:0/2126133207 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f8774883700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87701080a0 0x7f8770101330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f876f7fe700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8770105770 0x7f8770100df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f876ffff700 1 -- 172.21.15.156:0/2126133207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8770105770 msgr2=0x7f8770100df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f876ffff700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8770105770 0x7f8770100df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f876ffff700 1 -- 172.21.15.156:0/2126133207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87701080a0 msgr2=0x7f8770101330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f876ffff700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87701080a0 0x7f8770101330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f876ffff700 1 -- 172.21.15.156:0/2126133207 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8770101be0 con 0x7f8770102e40 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.943+0000 7f876ffff700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8770102e40 0x7f87701027a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f876000c970 tx=0x7f876000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.944+0000 7f876d7fa700 1 -- 172.21.15.156:0/2126133207 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760007aa0 con 0x7f8770102e40 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.944+0000 7f87762e6700 1 -- 172.21.15.156:0/2126133207 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87701a2680 con 0x7f8770102e40 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.944+0000 7f876d7fa700 1 -- 172.21.15.156:0/2126133207 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8760004d10 con 0x7f8770102e40 2024-03-19T01:08:53.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.944+0000 7f876d7fa700 1 -- 172.21.15.156:0/2126133207 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876000f450 con 0x7f8770102e40 2024-03-19T01:08:53.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.944+0000 7f87762e6700 1 -- 172.21.15.156:0/2126133207 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87701a2b40 con 0x7f8770102e40 2024-03-19T01:08:53.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.945+0000 7f87762e6700 1 -- 172.21.15.156:0/2126133207 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8770068fd0 con 0x7f8770102e40 2024-03-19T01:08:53.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.946+0000 7f876d7fa700 1 -- 172.21.15.156:0/2126133207 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f8760004750 con 0x7f8770102e40 2024-03-19T01:08:53.948 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.946+0000 7f876d7fa700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f87580a2890 0x7f87580a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:08:53.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.947+0000 7f876d7fa700 1 -- 172.21.15.156:0/2126133207 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f87600ce050 con 0x7f8770102e40 2024-03-19T01:08:53.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.947+0000 7f876f7fe700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f87580a2890 0x7f87580a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:08:53.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.947+0000 7f876f7fe700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f87580a2890 0x7f87580a4d50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8764005950 tx=0x7f87640058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:08:53.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:53.951+0000 7f876d7fa700 1 -- 172.21.15.156:0/2126133207 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f876009b980 con 0x7f8770102e40 2024-03-19T01:08:54.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:08:54.155+0000 7f87762e6700 1 -- 172.21.15.156:0/2126133207 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f8770062750 con 0x7f87580a2890 2024-03-19T01:08:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:08:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:08:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:54.597 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:08:54.597 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:08:54.597 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:54.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:08:54.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:08:54.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:08:55.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:55 smithi162 ceph-mon[67178]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:08:55.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:55 smithi162 ceph-mon[67178]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:08:55.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:55 smithi116 ceph-mon[62701]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:08:55.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:55 smithi116 ceph-mon[62701]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:08:55.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:55 smithi156 ceph-mon[66799]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:08:55.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:55 smithi156 ceph-mon[66799]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:08:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:57 smithi162 ceph-mon[67178]: from='client.? 172.21.15.156:0/491892113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7abd5362-d32e-48a0-9de5-d23ea5ba4391"}]: dispatch 2024-03-19T01:08:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:57 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7abd5362-d32e-48a0-9de5-d23ea5ba4391"}]: dispatch 2024-03-19T01:08:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:57 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7abd5362-d32e-48a0-9de5-d23ea5ba4391"}]': finished 2024-03-19T01:08:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:57 smithi162 ceph-mon[67178]: osdmap e69: 11 total, 10 up, 11 in 2024-03-19T01:08:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:57 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:08:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:57 smithi116 ceph-mon[62701]: from='client.? 172.21.15.156:0/491892113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7abd5362-d32e-48a0-9de5-d23ea5ba4391"}]: dispatch 2024-03-19T01:08:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:57 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7abd5362-d32e-48a0-9de5-d23ea5ba4391"}]: dispatch 2024-03-19T01:08:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:57 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7abd5362-d32e-48a0-9de5-d23ea5ba4391"}]': finished 2024-03-19T01:08:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:57 smithi116 ceph-mon[62701]: osdmap e69: 11 total, 10 up, 11 in 2024-03-19T01:08:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:57 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:08:57.454 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:57 smithi156 ceph-mon[66799]: from='client.? 172.21.15.156:0/491892113' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7abd5362-d32e-48a0-9de5-d23ea5ba4391"}]: dispatch 2024-03-19T01:08:57.454 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:57 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7abd5362-d32e-48a0-9de5-d23ea5ba4391"}]: dispatch 2024-03-19T01:08:57.454 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:57 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7abd5362-d32e-48a0-9de5-d23ea5ba4391"}]': finished 2024-03-19T01:08:57.454 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:57 smithi156 ceph-mon[66799]: osdmap e69: 11 total, 10 up, 11 in 2024-03-19T01:08:57.454 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:57 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:08:58.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:58 smithi162 ceph-mon[67178]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:08:58.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:58 smithi162 ceph-mon[67178]: from='client.? 172.21.15.156:0/3668421591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:08:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:58 smithi116 ceph-mon[62701]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:08:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:58 smithi116 ceph-mon[62701]: from='client.? 172.21.15.156:0/3668421591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:08:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:58 smithi156 ceph-mon[66799]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:08:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:58 smithi156 ceph-mon[66799]: from='client.? 172.21.15.156:0/3668421591' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:08:59.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:08:59 smithi162 ceph-mon[67178]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:08:59.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:08:59 smithi116 ceph-mon[62701]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:08:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:08:59 smithi156 ceph-mon[66799]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:02.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:02 smithi156 ceph-mon[66799]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:02.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:02 smithi162 ceph-mon[67178]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:02.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:02 smithi116 ceph-mon[62701]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:04.204 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:04 smithi156 ceph-mon[66799]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:04.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:04 smithi162 ceph-mon[67178]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:04.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:04 smithi116 ceph-mon[62701]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-19T01:09:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-19T01:09:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:05.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-19T01:09:05.417 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:06.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:06 smithi162 ceph-mon[67178]: Deploying daemon osd.10 on smithi156 2024-03-19T01:09:06.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:06 smithi162 ceph-mon[67178]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:06.401 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:06 smithi156 ceph-mon[66799]: Deploying daemon osd.10 on smithi156 2024-03-19T01:09:06.401 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:06 smithi156 ceph-mon[66799]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:06.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:06 smithi116 ceph-mon[62701]: Deploying daemon osd.10 on smithi156 2024-03-19T01:09:06.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:06 smithi116 ceph-mon[62701]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:08.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:08 smithi156 ceph-mon[66799]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:08.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:08 smithi162 ceph-mon[67178]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:08.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:08 smithi116 ceph-mon[62701]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:10.298 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:10 smithi156 ceph-mon[66799]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:10 smithi162 ceph-mon[67178]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:10.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:10 smithi116 ceph-mon[62701]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:11.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:11.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:11.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:11.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:11 smithi156 ceph-mon[66799]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:11 smithi162 ceph-mon[67178]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:11.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:11.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:11.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:11.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:11 smithi116 ceph-mon[62701]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:13.923 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 10 on host 'smithi156' 2024-03-19T01:09:13.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:09:13.922+0000 7f876d7fa700 1 -- 172.21.15.156:0/2126133207 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f8770062750 con 0x7f87580a2890 2024-03-19T01:09:13.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:09:13.924+0000 7f87762e6700 1 -- 172.21.15.156:0/2126133207 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f87580a2890 msgr2=0x7f87580a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:13.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:09:13.924+0000 7f87762e6700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f87580a2890 0x7f87580a4d50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8764005950 tx=0x7f87640058e0 comp rx=0 tx=0).stop 2024-03-19T01:09:13.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:09:13.924+0000 7f87762e6700 1 -- 172.21.15.156:0/2126133207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8770102e40 msgr2=0x7f87701027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:13.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:09:13.924+0000 7f87762e6700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8770102e40 0x7f87701027a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f876000c970 tx=0x7f876000cc80 comp rx=0 tx=0).stop 2024-03-19T01:09:13.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:09:13.924+0000 7f87762e6700 1 -- 172.21.15.156:0/2126133207 shutdown_connections 2024-03-19T01:09:13.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:09:13.924+0000 7f87762e6700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87701080a0 0x7f8770101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:13.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:09:13.924+0000 7f87762e6700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8770102e40 0x7f87701027a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:13.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:09:13.924+0000 7f87762e6700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f87580a2890 0x7f87580a4d50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:13.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:09:13.924+0000 7f87762e6700 1 --2- 172.21.15.156:0/2126133207 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8770105770 0x7f8770100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:13.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:09:13.924+0000 7f87762e6700 1 -- 172.21.15.156:0/2126133207 >> 172.21.15.156:0/2126133207 conn(0x7f87700fc870 msgr2=0x7f8770103760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:13.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:09:13.924+0000 7f87762e6700 1 -- 172.21.15.156:0/2126133207 shutdown_connections 2024-03-19T01:09:13.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-03-19T01:09:13.924+0000 7f87762e6700 1 -- 172.21.15.156:0/2126133207 wait complete. 2024-03-19T01:09:14.175 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:14 smithi156 ceph-mon[66799]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:14.175 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:14.175 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:14 smithi162 ceph-mon[67178]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:14 smithi116 ceph-mon[62701]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:14.576 DEBUG:teuthology.orchestra.run.smithi156:osd.10> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.10.service 2024-03-19T01:09:14.578 INFO:tasks.cephadm:Deploying osd.11 on smithi162 with /dev/vg_nvme/lv_1... 2024-03-19T01:09:14.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-19T01:09:14.828 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:09:15.261 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:15 smithi162 ceph-mon[67178]: from='osd.10 [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T01:09:15.262 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:15 smithi162 ceph-mon[67178]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T01:09:15.262 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:15 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:15.262 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:15 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:15.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:15 smithi116 ceph-mon[62701]: from='osd.10 [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T01:09:15.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:15 smithi116 ceph-mon[62701]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T01:09:15.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:15 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:15.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:15 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:15.418 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:15 smithi156 ceph-mon[66799]: from='osd.10 [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T01:09:15.419 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:15 smithi156 ceph-mon[66799]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-19T01:09:15.419 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:15 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:15.419 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:15 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:16.238 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:16 smithi162 ceph-mon[67178]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:16.238 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:16 smithi162 ceph-mon[67178]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-19T01:09:16.238 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:16 smithi162 ceph-mon[67178]: osdmap e70: 11 total, 10 up, 11 in 2024-03-19T01:09:16.238 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:16 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:16.238 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:16 smithi162 ceph-mon[67178]: from='osd.10 [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:09:16.238 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:16 smithi162 ceph-mon[67178]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:09:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:16 smithi116 ceph-mon[62701]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:16 smithi116 ceph-mon[62701]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-19T01:09:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:16 smithi116 ceph-mon[62701]: osdmap e70: 11 total, 10 up, 11 in 2024-03-19T01:09:16.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:16 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:16.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:16 smithi116 ceph-mon[62701]: from='osd.10 [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:09:16.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:16 smithi116 ceph-mon[62701]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:09:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:16 smithi156 ceph-mon[66799]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:16 smithi156 ceph-mon[66799]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-19T01:09:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:16 smithi156 ceph-mon[66799]: osdmap e70: 11 total, 10 up, 11 in 2024-03-19T01:09:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:16 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:16 smithi156 ceph-mon[66799]: from='osd.10 [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:09:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:16 smithi156 ceph-mon[66799]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-03-19T01:09:16.772 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:09:16 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:09:16.579+0000 7f8a522c9700 -1 osd.10 0 waiting for initial osdmap 2024-03-19T01:09:16.773 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:09:16 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:09:16.591+0000 7f8a4a0b4700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:09:17.014 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-19T01:09:17.014 INFO:teuthology.orchestra.run.smithi162.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-19T01:09:17.014 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10+0 records in 2024-03-19T01:09:17.014 INFO:teuthology.orchestra.run.smithi162.stderr:10+0 records out 2024-03-19T01:09:17.014 INFO:teuthology.orchestra.run.smithi162.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127098 s, 825 MB/s 2024-03-19T01:09:17.014 INFO:teuthology.orchestra.run.smithi162.stderr:--> Zapping successful for: 2024-03-19T01:09:17.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:17 smithi162 ceph-mon[67178]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-19T01:09:17.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:17 smithi162 ceph-mon[67178]: osdmap e71: 11 total, 10 up, 11 in 2024-03-19T01:09:17.319 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:17.320 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:17.320 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:17 smithi162 ceph-mon[67178]: from='osd.10 ' entity='osd.10' 2024-03-19T01:09:17.320 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:17 smithi116 ceph-mon[62701]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-19T01:09:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:17 smithi116 ceph-mon[62701]: osdmap e71: 11 total, 10 up, 11 in 2024-03-19T01:09:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:17 smithi116 ceph-mon[62701]: from='osd.10 ' entity='osd.10' 2024-03-19T01:09:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:17.467 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:17 smithi156 ceph-mon[66799]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-03-19T01:09:17.468 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:17 smithi156 ceph-mon[66799]: osdmap e71: 11 total, 10 up, 11 in 2024-03-19T01:09:17.468 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:17 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:17.468 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:17 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:17.468 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:17 smithi156 ceph-mon[66799]: from='osd.10 ' entity='osd.10' 2024-03-19T01:09:17.468 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:17 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:17.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi162:vg_nvme/lv_1 2024-03-19T01:09:17.948 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:09:18.335 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:18 smithi162 ceph-mon[67178]: purged_snaps scrub starts 2024-03-19T01:09:18.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:18 smithi162 ceph-mon[67178]: purged_snaps scrub ok 2024-03-19T01:09:18.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:18 smithi162 ceph-mon[67178]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:18.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:18 smithi162 ceph-mon[67178]: osd.10 [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631] boot 2024-03-19T01:09:18.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:18 smithi162 ceph-mon[67178]: osdmap e72: 11 total, 11 up, 11 in 2024-03-19T01:09:18.336 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:18 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:18 smithi116 ceph-mon[62701]: purged_snaps scrub starts 2024-03-19T01:09:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:18 smithi116 ceph-mon[62701]: purged_snaps scrub ok 2024-03-19T01:09:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:18 smithi116 ceph-mon[62701]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:18 smithi116 ceph-mon[62701]: osd.10 [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631] boot 2024-03-19T01:09:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:18 smithi116 ceph-mon[62701]: osdmap e72: 11 total, 11 up, 11 in 2024-03-19T01:09:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:18 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:18.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:18 smithi156 ceph-mon[66799]: purged_snaps scrub starts 2024-03-19T01:09:18.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:18 smithi156 ceph-mon[66799]: purged_snaps scrub ok 2024-03-19T01:09:18.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:18 smithi156 ceph-mon[66799]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 303 MiB used, 894 GiB / 894 GiB avail 2024-03-19T01:09:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:18 smithi156 ceph-mon[66799]: osd.10 [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631] boot 2024-03-19T01:09:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:18 smithi156 ceph-mon[66799]: osdmap e72: 11 total, 11 up, 11 in 2024-03-19T01:09:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:18 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-19T01:09:19.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: osdmap e73: 11 total, 11 up, 11 in 2024-03-19T01:09:19.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:19.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:19.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: osdmap e73: 11 total, 11 up, 11 in 2024-03-19T01:09:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:19.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:19.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:19.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:19.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:19.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:19.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: osdmap e73: 11 total, 11 up, 11 in 2024-03-19T01:09:19.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:19.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:19.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:19.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:19.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:19.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:19.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:19.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:19.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:19.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:19.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.690+0000 7f6a12c68700 1 -- 172.21.15.162:0/3804092288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a0c103ee0 msgr2=0x7f6a0c10a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:19.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.690+0000 7f6a12c68700 1 --2- 172.21.15.162:0/3804092288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a0c103ee0 0x7f6a0c10a790 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6a0400b0a0 tx=0x7f6a0400b3b0 comp rx=0 tx=0).stop 2024-03-19T01:09:19.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.691+0000 7f6a12c68700 1 -- 172.21.15.162:0/3804092288 shutdown_connections 2024-03-19T01:09:19.704 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.691+0000 7f6a12c68700 1 --2- 172.21.15.162:0/3804092288 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a0c103ee0 0x7f6a0c10a790 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:19.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.691+0000 7f6a12c68700 1 --2- 172.21.15.162:0/3804092288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a0c102340 0x7f6a0c102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:19.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.691+0000 7f6a12c68700 1 --2- 172.21.15.162:0/3804092288 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a0c103540 0x7f6a0c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:19.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.691+0000 7f6a12c68700 1 -- 172.21.15.162:0/3804092288 >> 172.21.15.162:0/3804092288 conn(0x7f6a0c0fd920 msgr2=0x7f6a0c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:19.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.691+0000 7f6a12c68700 1 -- 172.21.15.162:0/3804092288 shutdown_connections 2024-03-19T01:09:19.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.691+0000 7f6a12c68700 1 -- 172.21.15.162:0/3804092288 wait complete. 2024-03-19T01:09:19.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.691+0000 7f6a12c68700 1 Processor -- start 2024-03-19T01:09:19.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.691+0000 7f6a12c68700 1 -- start start 2024-03-19T01:09:19.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.691+0000 7f6a12c68700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a0c102340 0x7f6a0c198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:19.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.691+0000 7f6a12c68700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a0c103540 0x7f6a0c198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:19.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.691+0000 7f6a12c68700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a0c103ee0 0x7f6a0c19fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:19.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.691+0000 7f6a12c68700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0c10e020 con 0x7f6a0c103ee0 2024-03-19T01:09:19.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a12c68700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0c10dea0 con 0x7f6a0c103540 2024-03-19T01:09:19.705 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a12c68700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0c10e1a0 con 0x7f6a0c102340 2024-03-19T01:09:19.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a10a04700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a0c102340 0x7f6a0c198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:19.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a10a04700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a0c102340 0x7f6a0c198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:36134/0 (socket says 172.21.15.162:36134) 2024-03-19T01:09:19.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a10a04700 1 -- 172.21.15.162:0/4008530067 learned_addr learned my addr 172.21.15.162:0/4008530067 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T01:09:19.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a11205700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a0c103ee0 0x7f6a0c19fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:19.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a0bfff700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a0c103540 0x7f6a0c198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:19.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a10a04700 1 -- 172.21.15.162:0/4008530067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a0c103540 msgr2=0x7f6a0c198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:19.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a10a04700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a0c103540 0x7f6a0c198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:19.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a10a04700 1 -- 172.21.15.162:0/4008530067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a0c103ee0 msgr2=0x7f6a0c19fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:19.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a10a04700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a0c103ee0 0x7f6a0c19fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:19.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a10a04700 1 -- 172.21.15.162:0/4008530067 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0c1a02d0 con 0x7f6a0c102340 2024-03-19T01:09:19.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a10a04700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a0c102340 0x7f6a0c198670 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a0000b530 tx=0x7f6a0000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:19.706 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a09ffb700 1 -- 172.21.15.162:0/4008530067 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a00015020 con 0x7f6a0c102340 2024-03-19T01:09:19.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a09ffb700 1 -- 172.21.15.162:0/4008530067 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a00011d10 con 0x7f6a0c102340 2024-03-19T01:09:19.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a12c68700 1 -- 172.21.15.162:0/4008530067 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a0c1a05c0 con 0x7f6a0c102340 2024-03-19T01:09:19.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a09ffb700 1 -- 172.21.15.162:0/4008530067 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a0000f2e0 con 0x7f6a0c102340 2024-03-19T01:09:19.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.692+0000 7f6a12c68700 1 -- 172.21.15.162:0/4008530067 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a0c1a0bd0 con 0x7f6a0c102340 2024-03-19T01:09:19.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.693+0000 7f6a12c68700 1 -- 172.21.15.162:0/4008530067 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a0c068fd0 con 0x7f6a0c102340 2024-03-19T01:09:19.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.694+0000 7f6a09ffb700 1 -- 172.21.15.162:0/4008530067 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f6a00018020 con 0x7f6a0c102340 2024-03-19T01:09:19.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.695+0000 7f6a09ffb700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f69fc0a2910 0x7f69fc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:19.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.695+0000 7f6a09ffb700 1 -- 172.21.15.162:0/4008530067 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 6723+0+0 (secure 0 0 0) 0x7f6a000cdb90 con 0x7f6a0c102340 2024-03-19T01:09:19.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.695+0000 7f6a0bfff700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f69fc0a2910 0x7f69fc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:19.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.695+0000 7f6a0bfff700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f69fc0a2910 0x7f69fc0a4dd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f69f8006fd0 tx=0x7f69f8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:19.707 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.699+0000 7f6a09ffb700 1 -- 172.21.15.162:0/4008530067 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6a0001f020 con 0x7f6a0c102340 2024-03-19T01:09:19.907 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:19.907+0000 7f6a12c68700 1 -- 172.21.15.162:0/4008530067 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6a0c062750 con 0x7f69fc0a2910 2024-03-19T01:09:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:20 smithi162 ceph-mon[67178]: Detected new or changed devices on smithi156 2024-03-19T01:09:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:20 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi156 to 4270M 2024-03-19T01:09:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:20 smithi162 ceph-mon[67178]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:20 smithi162 ceph-mon[67178]: osdmap e74: 11 total, 11 up, 11 in 2024-03-19T01:09:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:20 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:09:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:20 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:09:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:20 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:20 smithi162 ceph-mon[67178]: osdmap e75: 11 total, 11 up, 11 in 2024-03-19T01:09:20.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:20 smithi116 ceph-mon[62701]: Detected new or changed devices on smithi156 2024-03-19T01:09:20.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:20 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi156 to 4270M 2024-03-19T01:09:20.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:20 smithi116 ceph-mon[62701]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:20.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:20 smithi116 ceph-mon[62701]: osdmap e74: 11 total, 11 up, 11 in 2024-03-19T01:09:20.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:20 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:09:20.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:20 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:09:20.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:20 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:20.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:20 smithi116 ceph-mon[62701]: osdmap e75: 11 total, 11 up, 11 in 2024-03-19T01:09:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:20 smithi156 ceph-mon[66799]: Detected new or changed devices on smithi156 2024-03-19T01:09:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:20 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi156 to 4270M 2024-03-19T01:09:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:20 smithi156 ceph-mon[66799]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:20 smithi156 ceph-mon[66799]: osdmap e74: 11 total, 11 up, 11 in 2024-03-19T01:09:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:20 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-19T01:09:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:20 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-19T01:09:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:20 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:20 smithi156 ceph-mon[66799]: osdmap e75: 11 total, 11 up, 11 in 2024-03-19T01:09:21.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:21 smithi162 ceph-mon[67178]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:09:21.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:21 smithi162 ceph-mon[67178]: osdmap e76: 11 total, 11 up, 11 in 2024-03-19T01:09:21.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:21 smithi116 ceph-mon[62701]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:09:21.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:21 smithi116 ceph-mon[62701]: osdmap e76: 11 total, 11 up, 11 in 2024-03-19T01:09:21.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:21 smithi156 ceph-mon[66799]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:09:21.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:21 smithi156 ceph-mon[66799]: osdmap e76: 11 total, 11 up, 11 in 2024-03-19T01:09:22.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:22 smithi162 ceph-mon[67178]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-03-19T01:09:22.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:22 smithi116 ceph-mon[62701]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-03-19T01:09:22.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:22 smithi156 ceph-mon[66799]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-03-19T01:09:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:23 smithi162 ceph-mon[67178]: from='client.? 172.21.15.162:0/1851421903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4fafaa0-173c-4b54-b5b1-10b925873fa0"}]: dispatch 2024-03-19T01:09:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:23 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4fafaa0-173c-4b54-b5b1-10b925873fa0"}]: dispatch 2024-03-19T01:09:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:23 smithi162 ceph-mon[67178]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4fafaa0-173c-4b54-b5b1-10b925873fa0"}]': finished 2024-03-19T01:09:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:23 smithi162 ceph-mon[67178]: osdmap e77: 12 total, 11 up, 12 in 2024-03-19T01:09:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:23 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:23 smithi116 ceph-mon[62701]: from='client.? 172.21.15.162:0/1851421903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4fafaa0-173c-4b54-b5b1-10b925873fa0"}]: dispatch 2024-03-19T01:09:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:23 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4fafaa0-173c-4b54-b5b1-10b925873fa0"}]: dispatch 2024-03-19T01:09:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:23 smithi116 ceph-mon[62701]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4fafaa0-173c-4b54-b5b1-10b925873fa0"}]': finished 2024-03-19T01:09:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:23 smithi116 ceph-mon[62701]: osdmap e77: 12 total, 11 up, 12 in 2024-03-19T01:09:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:23 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:23 smithi156 ceph-mon[66799]: from='client.? 172.21.15.162:0/1851421903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4fafaa0-173c-4b54-b5b1-10b925873fa0"}]: dispatch 2024-03-19T01:09:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:23 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4fafaa0-173c-4b54-b5b1-10b925873fa0"}]: dispatch 2024-03-19T01:09:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:23 smithi156 ceph-mon[66799]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4fafaa0-173c-4b54-b5b1-10b925873fa0"}]': finished 2024-03-19T01:09:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:23 smithi156 ceph-mon[66799]: osdmap e77: 12 total, 11 up, 12 in 2024-03-19T01:09:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:23 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:24.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:24 smithi162 ceph-mon[67178]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-19T01:09:24.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:24 smithi162 ceph-mon[67178]: from='client.? 172.21.15.162:0/914611906' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:09:24.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:24 smithi116 ceph-mon[62701]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-19T01:09:24.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:24 smithi116 ceph-mon[62701]: from='client.? 172.21.15.162:0/914611906' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:09:24.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:24 smithi156 ceph-mon[66799]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-19T01:09:24.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:24 smithi156 ceph-mon[66799]: from='client.? 172.21.15.162:0/914611906' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-19T01:09:25.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:25 smithi162 ceph-mon[67178]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-19T01:09:25.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:25 smithi116 ceph-mon[62701]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-19T01:09:25.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:25 smithi156 ceph-mon[66799]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 339 MiB used, 983 GiB / 983 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-19T01:09:28.350 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:28 smithi162 ceph-mon[67178]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-19T01:09:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:28 smithi116 ceph-mon[62701]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-19T01:09:28.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:28 smithi156 ceph-mon[66799]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-19T01:09:30.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:30 smithi162 ceph-mon[67178]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:30.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:30 smithi116 ceph-mon[62701]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:30.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:30 smithi156 ceph-mon[66799]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:31.256 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:31 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-19T01:09:31.256 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:31 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-19T01:09:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:31 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-19T01:09:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:31 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:32.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:32 smithi162 ceph-mon[67178]: Deploying daemon osd.11 on smithi162 2024-03-19T01:09:32.285 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:32 smithi162 ceph-mon[67178]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:32.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:32 smithi116 ceph-mon[62701]: Deploying daemon osd.11 on smithi162 2024-03-19T01:09:32.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:32 smithi116 ceph-mon[62701]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:32.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:32 smithi156 ceph-mon[66799]: Deploying daemon osd.11 on smithi162 2024-03-19T01:09:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:32 smithi156 ceph-mon[66799]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:34.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:34 smithi162 ceph-mon[67178]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:34.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:34 smithi116 ceph-mon[62701]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:34 smithi156 ceph-mon[66799]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:36.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:36 smithi162 ceph-mon[67178]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:36 smithi116 ceph-mon[62701]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:36.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:36 smithi156 ceph-mon[66799]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:37 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:37 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:37 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:37 smithi162 ceph-mon[67178]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:37.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:37 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:37.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:37 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:37.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:37 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:37.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:37 smithi116 ceph-mon[62701]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:37.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:37 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:37.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:37 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:37.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:37 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:37.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:37 smithi156 ceph-mon[66799]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:40.192 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:40 smithi162 ceph-mon[67178]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:40.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:40 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:40.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:40 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:40.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:40 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:40.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:40 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:40.193 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:40 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:40.272 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:40.271+0000 7f6a09ffb700 1 -- 172.21.15.162:0/4008530067 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f6a0c062750 con 0x7f69fc0a2910 2024-03-19T01:09:40.272 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 11 on host 'smithi162' 2024-03-19T01:09:40.275 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:40.274+0000 7f6a12c68700 1 -- 172.21.15.162:0/4008530067 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f69fc0a2910 msgr2=0x7f69fc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:40.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:40.274+0000 7f6a12c68700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f69fc0a2910 0x7f69fc0a4dd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f69f8006fd0 tx=0x7f69f8008040 comp rx=0 tx=0).stop 2024-03-19T01:09:40.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:40.274+0000 7f6a12c68700 1 -- 172.21.15.162:0/4008530067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a0c102340 msgr2=0x7f6a0c198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:40.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:40.274+0000 7f6a12c68700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a0c102340 0x7f6a0c198670 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6a0000b530 tx=0x7f6a0000b8f0 comp rx=0 tx=0).stop 2024-03-19T01:09:40.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:40.275+0000 7f6a12c68700 1 -- 172.21.15.162:0/4008530067 shutdown_connections 2024-03-19T01:09:40.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:40.275+0000 7f6a12c68700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a0c103ee0 0x7f6a0c19fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:40.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:40.275+0000 7f6a12c68700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f69fc0a2910 0x7f69fc0a4dd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:40.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:40.275+0000 7f6a12c68700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a0c102340 0x7f6a0c198670 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:40.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:40.275+0000 7f6a12c68700 1 --2- 172.21.15.162:0/4008530067 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a0c103540 0x7f6a0c198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:40.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:40.275+0000 7f6a12c68700 1 -- 172.21.15.162:0/4008530067 >> 172.21.15.162:0/4008530067 conn(0x7f6a0c0fd920 msgr2=0x7f6a0c106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:40.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:40.275+0000 7f6a12c68700 1 -- 172.21.15.162:0/4008530067 shutdown_connections 2024-03-19T01:09:40.276 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:40.275+0000 7f6a12c68700 1 -- 172.21.15.162:0/4008530067 wait complete. 2024-03-19T01:09:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:40 smithi116 ceph-mon[62701]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:40 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:40 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:40 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:40 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:40 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:40.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:40 smithi156 ceph-mon[66799]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:40.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:40 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:40.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:40 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:40.426 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:40 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:40.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:40 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:40.427 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:40 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:40.955 DEBUG:teuthology.orchestra.run.smithi162:osd.11> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.11.service 2024-03-19T01:09:40.957 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-19T01:09:40.957 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-19T01:09:41.190 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:41 smithi162 ceph-mon[67178]: from='osd.11 [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T01:09:41.190 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:41 smithi162 ceph-mon[67178]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T01:09:41.190 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:41 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:41.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:41 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:41.191 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:41 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:09:41.331 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:41 smithi116 ceph-mon[62701]: from='osd.11 [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T01:09:41.331 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:41 smithi116 ceph-mon[62701]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T01:09:41.331 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:41 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:41.331 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:41 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:41.331 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:41 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:41.441 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:41 smithi156 ceph-mon[66799]: from='osd.11 [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T01:09:41.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:41 smithi156 ceph-mon[66799]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-19T01:09:41.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:41 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:41.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:41 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:41.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:41 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:42.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:42 smithi162 ceph-mon[67178]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:42.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:42 smithi162 ceph-mon[67178]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-19T01:09:42.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:42 smithi162 ceph-mon[67178]: osdmap e78: 12 total, 11 up, 12 in 2024-03-19T01:09:42.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:42 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:42.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:42 smithi162 ceph-mon[67178]: from='osd.11 [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:09:42.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:42 smithi162 ceph-mon[67178]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:09:42.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:09:42 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:09:42.115+0000 7fb7b47c8700 -1 osd.11 0 waiting for initial osdmap 2024-03-19T01:09:42.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:09:42 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:09:42.128+0000 7fb7b05bb700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-19T01:09:42.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:42 smithi116 ceph-mon[62701]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:42.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:42 smithi116 ceph-mon[62701]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-19T01:09:42.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:42 smithi116 ceph-mon[62701]: osdmap e78: 12 total, 11 up, 12 in 2024-03-19T01:09:42.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:42 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:42.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:42 smithi116 ceph-mon[62701]: from='osd.11 [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:09:42.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:42 smithi116 ceph-mon[62701]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:09:42.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:42 smithi156 ceph-mon[66799]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:42 smithi156 ceph-mon[66799]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-19T01:09:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:42 smithi156 ceph-mon[66799]: osdmap e78: 12 total, 11 up, 12 in 2024-03-19T01:09:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:42 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:42 smithi156 ceph-mon[66799]: from='osd.11 [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:09:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:42 smithi156 ceph-mon[66799]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]: dispatch 2024-03-19T01:09:43.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.033+0000 7f9fd64f2700 1 -- 172.21.15.116:0/3810759624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fd0103530 msgr2=0x7f9fd0103990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:43.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.033+0000 7f9fd64f2700 1 --2- 172.21.15.116:0/3810759624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fd0103530 0x7f9fd0103990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9fc40099c0 tx=0x7f9fc4009cd0 comp rx=0 tx=0).stop 2024-03-19T01:09:43.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.033+0000 7f9fd64f2700 1 -- 172.21.15.116:0/3810759624 shutdown_connections 2024-03-19T01:09:43.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.033+0000 7f9fd64f2700 1 --2- 172.21.15.116:0/3810759624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fd0103ed0 0x7f9fd010a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:43.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.033+0000 7f9fd64f2700 1 --2- 172.21.15.116:0/3810759624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd0102330 0x7f9fd0102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:43.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.033+0000 7f9fd64f2700 1 --2- 172.21.15.116:0/3810759624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fd0103530 0x7f9fd0103990 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:43.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.033+0000 7f9fd64f2700 1 -- 172.21.15.116:0/3810759624 >> 172.21.15.116:0/3810759624 conn(0x7f9fd00fd8f0 msgr2=0x7f9fd00ffd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:43.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.033+0000 7f9fd64f2700 1 -- 172.21.15.116:0/3810759624 shutdown_connections 2024-03-19T01:09:43.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.033+0000 7f9fd64f2700 1 -- 172.21.15.116:0/3810759624 wait complete. 2024-03-19T01:09:43.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fd64f2700 1 Processor -- start 2024-03-19T01:09:43.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fd64f2700 1 -- start start 2024-03-19T01:09:43.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fd64f2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd0102330 0x7f9fd0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:43.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fd64f2700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fd0103530 0x7f9fd0198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fd64f2700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fd0103ed0 0x7f9fd019fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fd64f2700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd010df00 con 0x7f9fd0103530 2024-03-19T01:09:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fd64f2700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd010dd80 con 0x7f9fd0102330 2024-03-19T01:09:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fd64f2700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd010e080 con 0x7f9fd0103ed0 2024-03-19T01:09:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fcf7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fd0103530 0x7f9fd0198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fcf7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fd0103530 0x7f9fd0198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45164/0 (socket says 172.21.15.116:45164) 2024-03-19T01:09:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fcf7fe700 1 -- 172.21.15.116:0/2124147015 learned_addr learned my addr 172.21.15.116:0/2124147015 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:09:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fcffff700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd0102330 0x7f9fd0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fd4a8f700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fd0103ed0 0x7f9fd019fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fcf7fe700 1 -- 172.21.15.116:0/2124147015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fd0103ed0 msgr2=0x7f9fd019fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fcf7fe700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fd0103ed0 0x7f9fd019fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fcf7fe700 1 -- 172.21.15.116:0/2124147015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd0102330 msgr2=0x7f9fd0198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:43.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fcf7fe700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd0102330 0x7f9fd0198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:43.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fcf7fe700 1 -- 172.21.15.116:0/2124147015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd01a01b0 con 0x7f9fd0103530 2024-03-19T01:09:43.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.034+0000 7f9fcffff700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd0102330 0x7f9fd0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:09:43.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.035+0000 7f9fcf7fe700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fd0103530 0x7f9fd0198a90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4000c00 tx=0x7f9fc400bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:43.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.035+0000 7f9fcd7fa700 1 -- 172.21.15.116:0/2124147015 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4009e60 con 0x7f9fd0103530 2024-03-19T01:09:43.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.035+0000 7f9fd64f2700 1 -- 172.21.15.116:0/2124147015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd01a0440 con 0x7f9fd0103530 2024-03-19T01:09:43.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.035+0000 7f9fcd7fa700 1 -- 172.21.15.116:0/2124147015 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fc400f460 con 0x7f9fd0103530 2024-03-19T01:09:43.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.035+0000 7f9fcd7fa700 1 -- 172.21.15.116:0/2124147015 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc4017650 con 0x7f9fd0103530 2024-03-19T01:09:43.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.035+0000 7f9fd64f2700 1 -- 172.21.15.116:0/2124147015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fd01a0890 con 0x7f9fd0103530 2024-03-19T01:09:43.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.036+0000 7f9fd64f2700 1 -- 172.21.15.116:0/2124147015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fd0068fd0 con 0x7f9fd0103530 2024-03-19T01:09:43.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.037+0000 7f9fcd7fa700 1 -- 172.21.15.116:0/2124147015 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f9fc400f910 con 0x7f9fd0103530 2024-03-19T01:09:43.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.037+0000 7f9fcd7fa700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9fb80a2890 0x7f9fb80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:43.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.038+0000 7f9fcffff700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9fb80a2890 0x7f9fb80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:43.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.038+0000 7f9fcd7fa700 1 -- 172.21.15.116:0/2124147015 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 6847+0+0 (secure 0 0 0) 0x7f9fc40cf340 con 0x7f9fd0103530 2024-03-19T01:09:43.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.038+0000 7f9fcffff700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9fb80a2890 0x7f9fb80a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0103390 tx=0x7f9fc0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:43.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.041+0000 7f9fcd7fa700 1 -- 172.21.15.116:0/2124147015 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9fc4013070 con 0x7f9fd0103530 2024-03-19T01:09:43.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.243+0000 7f9fd64f2700 1 -- 172.21.15.116:0/2124147015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f9fd0062f20 con 0x7f9fd0103530 2024-03-19T01:09:43.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.244+0000 7f9fcd7fa700 1 -- 172.21.15.116:0/2124147015 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f9fc409ca30 con 0x7f9fd0103530 2024-03-19T01:09:43.245 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:09:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.246+0000 7f9fd64f2700 1 -- 172.21.15.116:0/2124147015 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9fb80a2890 msgr2=0x7f9fb80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.246+0000 7f9fd64f2700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9fb80a2890 0x7f9fb80a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0103390 tx=0x7f9fc0009260 comp rx=0 tx=0).stop 2024-03-19T01:09:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.246+0000 7f9fd64f2700 1 -- 172.21.15.116:0/2124147015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fd0103530 msgr2=0x7f9fd0198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.246+0000 7f9fd64f2700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fd0103530 0x7f9fd0198a90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9fc4000c00 tx=0x7f9fc400bee0 comp rx=0 tx=0).stop 2024-03-19T01:09:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.246+0000 7f9fd64f2700 1 -- 172.21.15.116:0/2124147015 shutdown_connections 2024-03-19T01:09:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.246+0000 7f9fd64f2700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9fb80a2890 0x7f9fb80a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.246+0000 7f9fd64f2700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fd0103ed0 0x7f9fd019fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.246+0000 7f9fd64f2700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fd0102330 0x7f9fd0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.246+0000 7f9fd64f2700 1 --2- 172.21.15.116:0/2124147015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fd0103530 0x7f9fd0198a90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.246+0000 7f9fd64f2700 1 -- 172.21.15.116:0/2124147015 >> 172.21.15.116:0/2124147015 conn(0x7f9fd00fd8f0 msgr2=0x7f9fd0106830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.246+0000 7f9fd64f2700 1 -- 172.21.15.116:0/2124147015 shutdown_connections 2024-03-19T01:09:43.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:43.246+0000 7f9fd64f2700 1 -- 172.21.15.116:0/2124147015 wait complete. 2024-03-19T01:09:43.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:43 smithi162 ceph-mon[67178]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T01:09:43.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:43 smithi162 ceph-mon[67178]: osdmap e79: 12 total, 11 up, 12 in 2024-03-19T01:09:43.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:43.371 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:43.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:43 smithi116 ceph-mon[62701]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T01:09:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:43 smithi116 ceph-mon[62701]: osdmap e79: 12 total, 11 up, 12 in 2024-03-19T01:09:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:43.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:43 smithi156 ceph-mon[66799]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-03-19T01:09:43.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:43 smithi156 ceph-mon[66799]: osdmap e79: 12 total, 11 up, 12 in 2024-03-19T01:09:43.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:43.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:43.872 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1710810583,"num_in_osds":12,"osd_in_since":1710810562,"num_remapped_pgs":0} 2024-03-19T01:09:43.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T01:09:44.113 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:09:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:44 smithi162 ceph-mon[67178]: purged_snaps scrub starts 2024-03-19T01:09:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:44 smithi162 ceph-mon[67178]: purged_snaps scrub ok 2024-03-19T01:09:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:44 smithi162 ceph-mon[67178]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:44 smithi162 ceph-mon[67178]: osd.11 [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941] boot 2024-03-19T01:09:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:44 smithi162 ceph-mon[67178]: osdmap e80: 12 total, 12 up, 12 in 2024-03-19T01:09:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:44 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:44 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2124147015' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T01:09:44.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:44 smithi116 ceph-mon[62701]: purged_snaps scrub starts 2024-03-19T01:09:44.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:44 smithi116 ceph-mon[62701]: purged_snaps scrub ok 2024-03-19T01:09:44.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:44 smithi116 ceph-mon[62701]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:44.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:44 smithi116 ceph-mon[62701]: osd.11 [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941] boot 2024-03-19T01:09:44.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:44 smithi116 ceph-mon[62701]: osdmap e80: 12 total, 12 up, 12 in 2024-03-19T01:09:44.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:44 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:44.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:44 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2124147015' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T01:09:44.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:44 smithi156 ceph-mon[66799]: purged_snaps scrub starts 2024-03-19T01:09:44.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:44 smithi156 ceph-mon[66799]: purged_snaps scrub ok 2024-03-19T01:09:44.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:44 smithi156 ceph-mon[66799]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 340 MiB used, 983 GiB / 983 GiB avail 2024-03-19T01:09:44.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:44 smithi156 ceph-mon[66799]: osd.11 [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941] boot 2024-03-19T01:09:44.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:44 smithi156 ceph-mon[66799]: osdmap e80: 12 total, 12 up, 12 in 2024-03-19T01:09:44.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:44 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-19T01:09:44.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:44 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2124147015' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-19T01:09:45.363 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:45 smithi116 ceph-mon[62701]: osdmap e81: 12 total, 12 up, 12 in 2024-03-19T01:09:45.363 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:45.363 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:45.363 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:45.363 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:45.363 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:45.363 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:45.364 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:45.364 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:45.364 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:45.364 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:45 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:45 smithi162 ceph-mon[67178]: osdmap e81: 12 total, 12 up, 12 in 2024-03-19T01:09:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:45.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:45.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:45 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:45 smithi156 ceph-mon[66799]: osdmap e81: 12 total, 12 up, 12 in 2024-03-19T01:09:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:09:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:45 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:45.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.877+0000 7f6d947af700 1 -- 172.21.15.116:0/3435730973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d8c1014e0 msgr2=0x7f6d8c1018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:45.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.877+0000 7f6d947af700 1 --2- 172.21.15.116:0/3435730973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d8c1014e0 0x7f6d8c1018c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6d880099c0 tx=0x7f6d88009cd0 comp rx=0 tx=0).stop 2024-03-19T01:09:45.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 -- 172.21.15.116:0/3435730973 shutdown_connections 2024-03-19T01:09:45.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 --2- 172.21.15.116:0/3435730973 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d8c1065c0 0x7f6d8c10e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:45.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 --2- 172.21.15.116:0/3435730973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d8c1014e0 0x7f6d8c1018c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 --2- 172.21.15.116:0/3435730973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d8c101e90 0x7f6d8c105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 -- 172.21.15.116:0/3435730973 >> 172.21.15.116:0/3435730973 conn(0x7f6d8c078340 msgr2=0x7f6d8c078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 -- 172.21.15.116:0/3435730973 shutdown_connections 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 -- 172.21.15.116:0/3435730973 wait complete. 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 Processor -- start 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 -- start start 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d8c1014e0 0x7f6d8c19e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d8c101e90 0x7f6d8c19ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d8c1065c0 0x7f6d8c1989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8c112700 con 0x7f6d8c101e90 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8c112580 con 0x7f6d8c1014e0 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.878+0000 7f6d947af700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8c112880 con 0x7f6d8c1065c0 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d91d4a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d8c101e90 0x7f6d8c19ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d91d4a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d8c101e90 0x7f6d8c19ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45182/0 (socket says 172.21.15.116:45182) 2024-03-19T01:09:45.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d91d4a700 1 -- 172.21.15.116:0/3831706605 learned_addr learned my addr 172.21.15.116:0/3831706605 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d9254b700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d8c1014e0 0x7f6d8c19e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d92d4c700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d8c1065c0 0x7f6d8c1989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d91d4a700 1 -- 172.21.15.116:0/3831706605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d8c1065c0 msgr2=0x7f6d8c1989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d91d4a700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d8c1065c0 0x7f6d8c1989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d91d4a700 1 -- 172.21.15.116:0/3831706605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d8c1014e0 msgr2=0x7f6d8c19e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d91d4a700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d8c1014e0 0x7f6d8c19e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d91d4a700 1 -- 172.21.15.116:0/3831706605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d8c199100 con 0x7f6d8c101e90 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d91d4a700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d8c101e90 0x7f6d8c19ecd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c00c8f0 tx=0x7f6d7c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d837fe700 1 -- 172.21.15.116:0/3831706605 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c007940 con 0x7f6d8c101e90 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d947af700 1 -- 172.21.15.116:0/3831706605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d8c1993f0 con 0x7f6d8c101e90 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d837fe700 1 -- 172.21.15.116:0/3831706605 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d7c00f450 con 0x7f6d8c101e90 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d837fe700 1 -- 172.21.15.116:0/3831706605 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c018660 con 0x7f6d8c101e90 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.879+0000 7f6d947af700 1 -- 172.21.15.116:0/3831706605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d8c1a5660 con 0x7f6d8c101e90 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.880+0000 7f6d947af700 1 -- 172.21.15.116:0/3831706605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d8c068fd0 con 0x7f6d8c101e90 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.881+0000 7f6d837fe700 1 -- 172.21.15.116:0/3831706605 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f6d7c00f600 con 0x7f6d8c101e90 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.882+0000 7f6d837fe700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6d780a2890 0x7f6d780a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:45.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.882+0000 7f6d9254b700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6d780a2890 0x7f6d780a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:45.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.882+0000 7f6d837fe700 1 -- 172.21.15.116:0/3831706605 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6d7c0ce430 con 0x7f6d8c101e90 2024-03-19T01:09:45.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.882+0000 7f6d9254b700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6d780a2890 0x7f6d780a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6d88000c00 tx=0x7f6d88005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:45.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:45.887+0000 7f6d837fe700 1 -- 172.21.15.116:0/3831706605 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6d7c014030 con 0x7f6d8c101e90 2024-03-19T01:09:46.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.081+0000 7f6d947af700 1 -- 172.21.15.116:0/3831706605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6d8c199e80 con 0x7f6d8c101e90 2024-03-19T01:09:46.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.082+0000 7f6d837fe700 1 -- 172.21.15.116:0/3831706605 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18516 (secure 0 0 0) 0x7f6d7c09b970 con 0x7f6d8c101e90 2024-03-19T01:09:46.083 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:09:46.083 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:09:44.118744+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:09:46.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.084+0000 7f6d947af700 1 -- 172.21.15.116:0/3831706605 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6d780a2890 msgr2=0x7f6d780a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:46.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.084+0000 7f6d947af700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6d780a2890 0x7f6d780a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6d88000c00 tx=0x7f6d88005fb0 comp rx=0 tx=0).stop 2024-03-19T01:09:46.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.084+0000 7f6d947af700 1 -- 172.21.15.116:0/3831706605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d8c101e90 msgr2=0x7f6d8c19ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:46.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.084+0000 7f6d947af700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d8c101e90 0x7f6d8c19ecd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c00c8f0 tx=0x7f6d7c00cc00 comp rx=0 tx=0).stop 2024-03-19T01:09:46.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.084+0000 7f6d947af700 1 -- 172.21.15.116:0/3831706605 shutdown_connections 2024-03-19T01:09:46.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.084+0000 7f6d947af700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6d780a2890 0x7f6d780a4d50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:46.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.084+0000 7f6d947af700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d8c1065c0 0x7f6d8c1989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:46.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.084+0000 7f6d947af700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d8c1014e0 0x7f6d8c19e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:46.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.084+0000 7f6d947af700 1 --2- 172.21.15.116:0/3831706605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d8c101e90 0x7f6d8c19ecd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:46.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.084+0000 7f6d947af700 1 -- 172.21.15.116:0/3831706605 >> 172.21.15.116:0/3831706605 conn(0x7f6d8c078340 msgr2=0x7f6d8c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:46.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.084+0000 7f6d947af700 1 -- 172.21.15.116:0/3831706605 shutdown_connections 2024-03-19T01:09:46.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:46.084+0000 7f6d947af700 1 -- 172.21.15.116:0/3831706605 wait complete. 2024-03-19T01:09:46.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:46 smithi162 ceph-mon[67178]: Detected new or changed devices on smithi162 2024-03-19T01:09:46.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:46 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi162 to 4270M 2024-03-19T01:09:46.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:46 smithi162 ceph-mon[67178]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:46.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:46 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3831706605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:09:46.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:46 smithi116 ceph-mon[62701]: Detected new or changed devices on smithi162 2024-03-19T01:09:46.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:46 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi162 to 4270M 2024-03-19T01:09:46.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:46 smithi116 ceph-mon[62701]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:46.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:46 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3831706605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:09:46.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:46 smithi156 ceph-mon[66799]: Detected new or changed devices on smithi162 2024-03-19T01:09:46.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:46 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi162 to 4270M 2024-03-19T01:09:46.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:46 smithi156 ceph-mon[66799]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:46.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:46 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3831706605' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:09:46.809 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:05:53.301933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:09:46.809 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-19T01:09:47.048 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:09:48.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:48 smithi116 ceph-mon[62701]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:48.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:48 smithi162 ceph-mon[67178]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:48.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:48 smithi156 ceph-mon[66799]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:48.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.834+0000 7f11ee10a700 1 -- 172.21.15.116:0/424610721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e8108360 msgr2=0x7f11e8108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:48.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.834+0000 7f11ee10a700 1 --2- 172.21.15.116:0/424610721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e8108360 0x7f11e8108740 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f11dc0099c0 tx=0x7f11dc009cd0 comp rx=0 tx=0).stop 2024-03-19T01:09:48.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.834+0000 7f11ee10a700 1 -- 172.21.15.116:0/424610721 shutdown_connections 2024-03-19T01:09:48.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.834+0000 7f11ee10a700 1 --2- 172.21.15.116:0/424610721 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11e806a960 0x7f11e810f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:48.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.834+0000 7f11ee10a700 1 --2- 172.21.15.116:0/424610721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e8108360 0x7f11e8108740 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:48.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.834+0000 7f11ee10a700 1 --2- 172.21.15.116:0/424610721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11e8069fc0 0x7f11e806a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:48.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.834+0000 7f11ee10a700 1 -- 172.21.15.116:0/424610721 >> 172.21.15.116:0/424610721 conn(0x7f11e8078140 msgr2=0x7f11e8078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:48.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.834+0000 7f11ee10a700 1 -- 172.21.15.116:0/424610721 shutdown_connections 2024-03-19T01:09:48.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.834+0000 7f11ee10a700 1 -- 172.21.15.116:0/424610721 wait complete. 2024-03-19T01:09:48.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.835+0000 7f11ee10a700 1 Processor -- start 2024-03-19T01:09:48.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.835+0000 7f11ee10a700 1 -- start start 2024-03-19T01:09:48.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.835+0000 7f11ee10a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11e8069fc0 0x7f11e819e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:48.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.835+0000 7f11ee10a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e806a960 0x7f11e819ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:48.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.835+0000 7f11ee10a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11e8108360 0x7f11e81988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:48.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.835+0000 7f11ee10a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11e8112620 con 0x7f11e806a960 2024-03-19T01:09:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.835+0000 7f11ee10a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f11e81124a0 con 0x7f11e8108360 2024-03-19T01:09:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.835+0000 7f11ee10a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11e81127a0 con 0x7f11e8069fc0 2024-03-19T01:09:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e806a960 0x7f11e819ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e6ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e806a960 0x7f11e819ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50912/0 (socket says 172.21.15.116:50912) 2024-03-19T01:09:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e6ffd700 1 -- 172.21.15.116:0/2442676144 learned_addr learned my addr 172.21.15.116:0/2442676144 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:09:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e7fff700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11e8108360 0x7f11e81988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e77fe700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11e8069fc0 0x7f11e819e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e6ffd700 1 -- 172.21.15.116:0/2442676144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11e8069fc0 msgr2=0x7f11e819e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e6ffd700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11e8069fc0 0x7f11e819e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e6ffd700 1 -- 172.21.15.116:0/2442676144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11e8108360 msgr2=0x7f11e81988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e6ffd700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11e8108360 0x7f11e81988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e6ffd700 1 -- 172.21.15.116:0/2442676144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e8199040 con 0x7f11e806a960 2024-03-19T01:09:48.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e6ffd700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e806a960 0x7f11e819ec60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f11d800b530 tx=0x7f11d800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e4ff9700 1 -- 172.21.15.116:0/2442676144 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d8015020 con 0x7f11e806a960 2024-03-19T01:09:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11ee10a700 1 -- 172.21.15.116:0/2442676144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e8199330 con 0x7f11e806a960 2024-03-19T01:09:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e4ff9700 1 -- 172.21.15.116:0/2442676144 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11d8011d10 con 0x7f11e806a960 2024-03-19T01:09:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11e4ff9700 1 -- 172.21.15.116:0/2442676144 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d800f2e0 con 0x7f11e806a960 2024-03-19T01:09:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.836+0000 7f11ee10a700 1 -- 172.21.15.116:0/2442676144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e81a55f0 con 0x7f11e806a960 2024-03-19T01:09:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.837+0000 7f11ee10a700 1 -- 172.21.15.116:0/2442676144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11e8068fd0 con 0x7f11e806a960 2024-03-19T01:09:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.838+0000 7f11e4ff9700 1 -- 172.21.15.116:0/2442676144 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f11d8018020 con 0x7f11e806a960 2024-03-19T01:09:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.839+0000 7f11e4ff9700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f11d00a2890 0x7f11d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.839+0000 7f11e77fe700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f11d00a2890 0x7f11d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.839+0000 7f11e4ff9700 1 -- 172.21.15.116:0/2442676144 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f11d80ce230 con 0x7f11e806a960 2024-03-19T01:09:48.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.839+0000 7f11e77fe700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f11d00a2890 0x7f11d00a4d50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f11dc000c00 tx=0x7f11dc005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:48.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:48.843+0000 7f11e4ff9700 1 -- 172.21.15.116:0/2442676144 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f11d801f020 con 0x7f11e806a960 2024-03-19T01:09:49.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.036+0000 7f11ee10a700 1 -- 172.21.15.116:0/2442676144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f11e8199e10 con 0x7f11e806a960 2024-03-19T01:09:49.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.037+0000 7f11e4ff9700 1 -- 172.21.15.116:0/2442676144 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f11d809b770 con 0x7f11e806a960 2024-03-19T01:09:49.038 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T01:09:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.039+0000 7f11ee10a700 1 -- 172.21.15.116:0/2442676144 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f11d00a2890 msgr2=0x7f11d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.039+0000 7f11ee10a700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f11d00a2890 0x7f11d00a4d50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f11dc000c00 tx=0x7f11dc005fb0 comp rx=0 tx=0).stop 2024-03-19T01:09:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.039+0000 7f11ee10a700 1 -- 172.21.15.116:0/2442676144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e806a960 msgr2=0x7f11e819ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.039+0000 7f11ee10a700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e806a960 0x7f11e819ec60 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f11d800b530 tx=0x7f11d800b840 comp rx=0 tx=0).stop 2024-03-19T01:09:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.039+0000 7f11ee10a700 1 -- 172.21.15.116:0/2442676144 shutdown_connections 2024-03-19T01:09:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.039+0000 7f11ee10a700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11e8108360 0x7f11e81988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.039+0000 7f11ee10a700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f11d00a2890 0x7f11d00a4d50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.039+0000 7f11ee10a700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11e8069fc0 0x7f11e819e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.039+0000 7f11ee10a700 1 --2- 172.21.15.116:0/2442676144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e806a960 0x7f11e819ec60 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:49.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.039+0000 7f11ee10a700 1 -- 172.21.15.116:0/2442676144 >> 172.21.15.116:0/2442676144 conn(0x7f11e8078140 msgr2=0x7f11e80ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:49.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.039+0000 7f11ee10a700 1 -- 172.21.15.116:0/2442676144 shutdown_connections 2024-03-19T01:09:49.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:49.039+0000 7f11ee10a700 1 -- 172.21.15.116:0/2442676144 wait complete. 2024-03-19T01:09:49.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:49 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2442676144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:09:49.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:49 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2442676144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:09:49.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:49 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2442676144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:09:49.882 INFO:tasks.cephadm:Adding mds.a on smithi116 2024-03-19T01:09:49.883 INFO:tasks.cephadm:Adding mds.d on smithi116 2024-03-19T01:09:49.883 INFO:tasks.cephadm:Adding mds.g on smithi116 2024-03-19T01:09:49.883 INFO:tasks.cephadm:Adding mds.j on smithi116 2024-03-19T01:09:49.883 INFO:tasks.cephadm:Adding mds.b on smithi156 2024-03-19T01:09:49.883 INFO:tasks.cephadm:Adding mds.e on smithi156 2024-03-19T01:09:49.883 INFO:tasks.cephadm:Adding mds.h on smithi156 2024-03-19T01:09:49.883 INFO:tasks.cephadm:Adding mds.k on smithi156 2024-03-19T01:09:49.883 INFO:tasks.cephadm:Adding mds.c on smithi162 2024-03-19T01:09:49.883 INFO:tasks.cephadm:Adding mds.f on smithi162 2024-03-19T01:09:49.883 INFO:tasks.cephadm:Adding mds.i on smithi162 2024-03-19T01:09:49.884 INFO:tasks.cephadm:Adding mds.l on smithi162 2024-03-19T01:09:49.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l' 2024-03-19T01:09:50.133 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:09:50.389 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:50 smithi162 ceph-mon[67178]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:50.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:50 smithi116 ceph-mon[62701]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:50.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:50 smithi156 ceph-mon[66799]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:51.737 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.735+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/468275694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bd4101270 msgr2=0x7f5bd4101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:51.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.735+0000 7f5bdc1bd700 1 --2- 172.21.15.162:0/468275694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bd4101270 0x7f5bd4101690 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5bc4009990 tx=0x7f5bc4009ca0 comp rx=0 tx=0).stop 2024-03-19T01:09:51.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.735+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/468275694 shutdown_connections 2024-03-19T01:09:51.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.735+0000 7f5bdc1bd700 1 --2- 172.21.15.162:0/468275694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd4102e10 0x7f5bd41096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:51.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.735+0000 7f5bdc1bd700 1 --2- 172.21.15.162:0/468275694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bd4101270 0x7f5bd4101690 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:51.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.735+0000 7f5bdc1bd700 1 --2- 172.21.15.162:0/468275694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd4102470 0x7f5bd41028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:51.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.735+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/468275694 >> 172.21.15.162:0/468275694 conn(0x7f5bd40fc830 msgr2=0x7f5bd40fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:51.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.736+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/468275694 shutdown_connections 2024-03-19T01:09:51.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.736+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/468275694 wait complete. 2024-03-19T01:09:51.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.736+0000 7f5bdc1bd700 1 Processor -- start 2024-03-19T01:09:51.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.736+0000 7f5bdc1bd700 1 -- start start 2024-03-19T01:09:51.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.736+0000 7f5bdc1bd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bd4101270 0x7f5bd406b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:51.738 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.736+0000 7f5bdc1bd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd4102470 0x7f5bd419d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.736+0000 7f5bdc1bd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd4102e10 0x7f5bd419f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.736+0000 7f5bdc1bd700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd410ce60 con 0x7f5bd4102e10 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.736+0000 7f5bdc1bd700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd410cce0 con 0x7f5bd4102470 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.736+0000 7f5bdc1bd700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5bd410cfe0 con 0x7f5bd4101270 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.736+0000 7f5bd9f59700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bd4101270 0x7f5bd406b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.736+0000 7f5bd9f59700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bd4101270 0x7f5bd406b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.162:48354/0 (socket says 172.21.15.162:48354) 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.736+0000 7f5bd9f59700 1 -- 172.21.15.162:0/755346305 learned_addr learned my addr 172.21.15.162:0/755346305 (peer_addr_for_me v2:172.21.15.162:0/0) 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.737+0000 7f5bda75a700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd4102e10 0x7f5bd419f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.737+0000 7f5bd9758700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd4102470 0x7f5bd419d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.737+0000 7f5bd9f59700 1 -- 172.21.15.162:0/755346305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd4102470 msgr2=0x7f5bd419d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.737+0000 7f5bd9f59700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd4102470 0x7f5bd419d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.737+0000 7f5bd9f59700 1 -- 172.21.15.162:0/755346305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd4102e10 msgr2=0x7f5bd419f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.737+0000 7f5bd9f59700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd4102e10 0x7f5bd419f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:51.739 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.737+0000 7f5bd9f59700 1 -- 172.21.15.162:0/755346305 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd41a01e0 con 0x7f5bd4101270 2024-03-19T01:09:51.740 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.737+0000 7f5bd9f59700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bd4101270 0x7f5bd406b5b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40039c0 tx=0x7f5bc40047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:51.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.737+0000 7f5bcaffd700 1 -- 172.21.15.162:0/755346305 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4027070 con 0x7f5bd4101270 2024-03-19T01:09:51.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.737+0000 7f5bcaffd700 1 -- 172.21.15.162:0/755346305 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bc4004150 con 0x7f5bd4101270 2024-03-19T01:09:51.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.737+0000 7f5bcaffd700 1 -- 172.21.15.162:0/755346305 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc4022e00 con 0x7f5bd4101270 2024-03-19T01:09:51.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.737+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/755346305 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bd41a0470 con 0x7f5bd4101270 2024-03-19T01:09:51.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.737+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/755346305 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bd41a08c0 con 0x7f5bd4101270 2024-03-19T01:09:51.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.738+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/755346305 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bd4068fd0 con 0x7f5bd4101270 2024-03-19T01:09:51.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.739+0000 7f5bcaffd700 1 -- 172.21.15.162:0/755346305 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f5bc402a070 con 0x7f5bd4101270 2024-03-19T01:09:51.741 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.740+0000 7f5bcaffd700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5bc00a2890 0x7f5bc00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:51.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.740+0000 7f5bcaffd700 1 -- 172.21.15.162:0/755346305 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5bc40cf740 con 0x7f5bd4101270 2024-03-19T01:09:51.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.740+0000 7f5bd9758700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5bc00a2890 0x7f5bc00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:51.742 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.740+0000 7f5bd9758700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5bc00a2890 0x7f5bc00a4d50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5bd406bb30 tx=0x7f5bd000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:51.744 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.743+0000 7f5bcaffd700 1 -- 172.21.15.162:0/755346305 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5bc4013070 con 0x7f5bd4101270 2024-03-19T01:09:51.950 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.949+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/755346305 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}) -- 0x7f5bd4062a10 con 0x7f5bc00a2890 2024-03-19T01:09:51.958 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.958+0000 7f5bcaffd700 1 -- 172.21.15.162:0/755346305 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f5bd4062a10 con 0x7f5bc00a2890 2024-03-19T01:09:51.959 INFO:teuthology.orchestra.run.smithi162.stdout:Scheduled mds.all update... 2024-03-19T01:09:51.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.960+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/755346305 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5bc00a2890 msgr2=0x7f5bc00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:51.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.960+0000 7f5bdc1bd700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5bc00a2890 0x7f5bc00a4d50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5bd406bb30 tx=0x7f5bd000b410 comp rx=0 tx=0).stop 2024-03-19T01:09:51.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.960+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/755346305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bd4101270 msgr2=0x7f5bd406b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:51.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.960+0000 7f5bdc1bd700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bd4101270 0x7f5bd406b5b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5bc40039c0 tx=0x7f5bc40047b0 comp rx=0 tx=0).stop 2024-03-19T01:09:51.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.960+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/755346305 shutdown_connections 2024-03-19T01:09:51.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.960+0000 7f5bdc1bd700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bd4102e10 0x7f5bd419f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:51.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.960+0000 7f5bdc1bd700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5bc00a2890 0x7f5bc00a4d50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:51.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.960+0000 7f5bdc1bd700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5bd4101270 0x7f5bd406b5b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:51.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.960+0000 7f5bdc1bd700 1 --2- 172.21.15.162:0/755346305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5bd4102470 0x7f5bd419d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:51.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.960+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/755346305 >> 172.21.15.162:0/755346305 conn(0x7f5bd40fc830 msgr2=0x7f5bd4105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:51.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.960+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/755346305 shutdown_connections 2024-03-19T01:09:51.961 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-19T01:09:51.960+0000 7f5bdc1bd700 1 -- 172.21.15.162:0/755346305 wait complete. 2024-03-19T01:09:52.286 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:52 smithi162 ceph-mon[67178]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:52.286 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:52 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:52.287 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:52 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:52.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:52 smithi116 ceph-mon[62701]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:52.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:52 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:52.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:52 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:52.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:52 smithi156 ceph-mon[66799]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:52.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:52 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:52.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:52 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:09:52.715 DEBUG:teuthology.orchestra.run.smithi116:mds.a> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.a.service 2024-03-19T01:09:52.761 DEBUG:teuthology.orchestra.run.smithi116:mds.d> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.d.service 2024-03-19T01:09:52.804 DEBUG:teuthology.orchestra.run.smithi116:mds.g> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.g.service 2024-03-19T01:09:52.846 DEBUG:teuthology.orchestra.run.smithi116:mds.j> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.j.service 2024-03-19T01:09:52.889 DEBUG:teuthology.orchestra.run.smithi156:mds.b> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.b.service 2024-03-19T01:09:52.891 DEBUG:teuthology.orchestra.run.smithi156:mds.e> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.e.service 2024-03-19T01:09:52.935 DEBUG:teuthology.orchestra.run.smithi156:mds.h> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.h.service 2024-03-19T01:09:52.978 DEBUG:teuthology.orchestra.run.smithi156:mds.k> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.k.service 2024-03-19T01:09:53.020 DEBUG:teuthology.orchestra.run.smithi162:mds.c> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.c.service 2024-03-19T01:09:53.023 DEBUG:teuthology.orchestra.run.smithi162:mds.f> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.f.service 2024-03-19T01:09:53.066 DEBUG:teuthology.orchestra.run.smithi162:mds.i> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.i.service 2024-03-19T01:09:53.109 DEBUG:teuthology.orchestra.run.smithi162:mds.l> sudo journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.l.service 2024-03-19T01:09:53.151 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-19T01:09:53.152 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:09:53.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:53 smithi156 ceph-mon[66799]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:09:53.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:53 smithi156 ceph-mon[66799]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-03-19T01:09:53.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:53.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:53.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:53.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:09:53.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:09:53.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:53 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.259+0000 7efc16c59640 1 -- 172.21.15.116:0/2397153623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc10101930 msgr2=0x7efc10101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:53.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.259+0000 7efc16c59640 1 --2- 172.21.15.116:0/2397153623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc10101930 0x7efc10101db0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009900 tx=0x7efbfc02bc30 comp rx=0 tx=0).stop 2024-03-19T01:09:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.259+0000 7efc16c59640 1 -- 172.21.15.116:0/2397153623 shutdown_connections 2024-03-19T01:09:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.259+0000 7efc16c59640 1 --2- 172.21.15.116:0/2397153623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc101022f0 0x7efc10108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.259+0000 7efc16c59640 1 --2- 172.21.15.116:0/2397153623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc10101930 0x7efc10101db0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.259+0000 7efc16c59640 1 --2- 172.21.15.116:0/2397153623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc10100740 0x7efc10100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.259+0000 7efc16c59640 1 -- 172.21.15.116:0/2397153623 >> 172.21.15.116:0/2397153623 conn(0x7efc100fbcb0 msgr2=0x7efc100fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.259+0000 7efc16c59640 1 -- 172.21.15.116:0/2397153623 shutdown_connections 2024-03-19T01:09:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.259+0000 7efc16c59640 1 -- 172.21.15.116:0/2397153623 wait complete. 2024-03-19T01:09:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc16c59640 1 Processor -- start 2024-03-19T01:09:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc16c59640 1 -- start start 2024-03-19T01:09:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc16c59640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc10100740 0x7efc100694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc16c59640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc10101930 0x7efc1019ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc16c59640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc101022f0 0x7efc101a1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:53.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc16c59640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efc1010c3c0 con 0x7efc101022f0 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc16c59640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efc1010c240 con 0x7efc10101930 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc16c59640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7efc1010c540 con 0x7efc10100740 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc151f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc101022f0 0x7efc101a1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc149f5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc10100740 0x7efc100694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc151f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc101022f0 0x7efc101a1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50928/0 (socket says 172.21.15.116:50928) 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc151f6640 1 -- 172.21.15.116:0/2842895506 learned_addr learned my addr 172.21.15.116:0/2842895506 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc0ffff640 1 --2- 172.21.15.116:0/2842895506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc10101930 0x7efc1019ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc151f6640 1 -- 172.21.15.116:0/2842895506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc10100740 msgr2=0x7efc100694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc151f6640 1 --2- 172.21.15.116:0/2842895506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc10100740 0x7efc100694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc151f6640 1 -- 172.21.15.116:0/2842895506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc10101930 msgr2=0x7efc1019ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc151f6640 1 --2- 172.21.15.116:0/2842895506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc10101930 0x7efc1019ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.260+0000 7efc151f6640 1 -- 172.21.15.116:0/2842895506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc101a19f0 con 0x7efc101022f0 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.261+0000 7efc151f6640 1 --2- 172.21.15.116:0/2842895506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc101022f0 0x7efc101a1200 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efc0800eb10 tx=0x7efc0800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.261+0000 7efc0dffb640 1 -- 172.21.15.116:0/2842895506 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08004b20 con 0x7efc101022f0 2024-03-19T01:09:53.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.261+0000 7efc0dffb640 1 -- 172.21.15.116:0/2842895506 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc08004cc0 con 0x7efc101022f0 2024-03-19T01:09:53.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.261+0000 7efc16c59640 1 -- 172.21.15.116:0/2842895506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc101a1c80 con 0x7efc101022f0 2024-03-19T01:09:53.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.261+0000 7efc0dffb640 1 -- 172.21.15.116:0/2842895506 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08010760 con 0x7efc101022f0 2024-03-19T01:09:53.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.261+0000 7efc16c59640 1 -- 172.21.15.116:0/2842895506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc101a21a0 con 0x7efc101022f0 2024-03-19T01:09:53.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.262+0000 7efc16c59640 1 -- 172.21.15.116:0/2842895506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc10107a60 con 0x7efc101022f0 2024-03-19T01:09:53.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.263+0000 7efc0dffb640 1 -- 172.21.15.116:0/2842895506 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7efc08005020 con 0x7efc101022f0 2024-03-19T01:09:53.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.263+0000 7efc0dffb640 1 --2- 172.21.15.116:0/2842895506 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efc000a2830 0x7efc000a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:53.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.264+0000 7efc149f5640 1 --2- 172.21.15.116:0/2842895506 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efc000a2830 0x7efc000a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:53.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.264+0000 7efc0dffb640 1 -- 172.21.15.116:0/2842895506 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7efc080cea50 con 0x7efc101022f0 2024-03-19T01:09:53.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.264+0000 7efc149f5640 1 --2- 172.21.15.116:0/2842895506 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efc000a2830 0x7efc000a4cf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efc04005fd0 tx=0x7efc0400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:53.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.268+0000 7efc0dffb640 1 -- 172.21.15.116:0/2842895506 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efc0809bf90 con 0x7efc101022f0 2024-03-19T01:09:53.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:53 smithi162 ceph-mon[67178]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:09:53.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:53 smithi162 ceph-mon[67178]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-03-19T01:09:53.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:53.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:53.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:53.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:09:53.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:09:53.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:53 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:53.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:53 smithi116 ceph-mon[62701]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:09:53.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:53 smithi116 ceph-mon[62701]: Saving service mds.all spec with placement smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-03-19T01:09:53.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:53.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:09:53.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:53.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:09:53.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:09:53.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:53 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:53.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.449+0000 7efc16c59640 1 -- 172.21.15.116:0/2842895506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc1002d430 con 0x7efc101022f0 2024-03-19T01:09:53.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.450+0000 7efc0dffb640 1 -- 172.21.15.116:0/2842895506 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18516 (secure 0 0 0) 0x7efc08084e20 con 0x7efc101022f0 2024-03-19T01:09:53.451 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:09:53.452 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:09:44.118744+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:09:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.453+0000 7efc16c59640 1 -- 172.21.15.116:0/2842895506 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efc000a2830 msgr2=0x7efc000a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.453+0000 7efc16c59640 1 --2- 172.21.15.116:0/2842895506 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efc000a2830 0x7efc000a4cf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efc04005fd0 tx=0x7efc0400b410 comp rx=0 tx=0).stop 2024-03-19T01:09:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.453+0000 7efc16c59640 1 -- 172.21.15.116:0/2842895506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc101022f0 msgr2=0x7efc101a1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.453+0000 7efc16c59640 1 --2- 172.21.15.116:0/2842895506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc101022f0 0x7efc101a1200 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7efc0800eb10 tx=0x7efc0800c2d0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.453+0000 7efc16c59640 1 -- 172.21.15.116:0/2842895506 shutdown_connections 2024-03-19T01:09:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.453+0000 7efc16c59640 1 --2- 172.21.15.116:0/2842895506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efc101022f0 0x7efc101a1200 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.453+0000 7efc16c59640 1 --2- 172.21.15.116:0/2842895506 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7efc000a2830 0x7efc000a4cf0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.453+0000 7efc16c59640 1 --2- 172.21.15.116:0/2842895506 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efc10101930 0x7efc1019ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.453+0000 7efc16c59640 1 --2- 172.21.15.116:0/2842895506 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7efc10100740 0x7efc100694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.453+0000 7efc16c59640 1 -- 172.21.15.116:0/2842895506 >> 172.21.15.116:0/2842895506 conn(0x7efc100fbcb0 msgr2=0x7efc10104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:53.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.453+0000 7efc16c59640 1 -- 172.21.15.116:0/2842895506 shutdown_connections 2024-03-19T01:09:53.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.453+0000 7efc16c59640 1 -- 172.21.15.116:0/2842895506 wait complete. 2024-03-19T01:09:53.465 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:05:53.301933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:09:53.465 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:09:53.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.568+0000 7f3ec71a9640 1 -- 172.21.15.116:0/1588688278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec0103400 msgr2=0x7f3ec0109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:53.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.568+0000 7f3ec71a9640 1 --2- 172.21.15.116:0/1588688278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec0103400 0x7f3ec0109c90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8009ca0 tx=0x7f3eb802c0f0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.568+0000 7f3ec71a9640 1 -- 172.21.15.116:0/1588688278 shutdown_connections 2024-03-19T01:09:53.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.568+0000 7f3ec71a9640 1 --2- 172.21.15.116:0/1588688278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec0103400 0x7f3ec0109c90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.568+0000 7f3ec71a9640 1 --2- 172.21.15.116:0/1588688278 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ec0102a40 0x7f3ec0102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.568+0000 7f3ec71a9640 1 --2- 172.21.15.116:0/1588688278 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ec0101850 0x7f3ec0101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.568+0000 7f3ec71a9640 1 -- 172.21.15.116:0/1588688278 >> 172.21.15.116:0/1588688278 conn(0x7f3ec00fcdc0 msgr2=0x7f3ec00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:53.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.568+0000 7f3ec71a9640 1 -- 172.21.15.116:0/1588688278 shutdown_connections 2024-03-19T01:09:53.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.568+0000 7f3ec71a9640 1 -- 172.21.15.116:0/1588688278 wait complete. 2024-03-19T01:09:53.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec71a9640 1 Processor -- start 2024-03-19T01:09:53.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec71a9640 1 -- start start 2024-03-19T01:09:53.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec71a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ec0101850 0x7f3ec0197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec71a9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ec0102a40 0x7f3ec01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec71a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec0103400 0x7f3ec019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec71a9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec010d530 con 0x7f3ec0103400 2024-03-19T01:09:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec71a9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec010d3b0 con 0x7f3ec0102a40 2024-03-19T01:09:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec71a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec010d6b0 con 0x7f3ec0101850 2024-03-19T01:09:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec5746640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec0103400 0x7f3ec019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec5746640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec0103400 0x7f3ec019f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50934/0 (socket says 172.21.15.116:50934) 2024-03-19T01:09:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec5746640 1 -- 172.21.15.116:0/4109471582 learned_addr learned my addr 172.21.15.116:0/4109471582 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:09:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec4f45640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ec0101850 0x7f3ec0197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ebffff640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ec0102a40 0x7f3ec01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec5746640 1 -- 172.21.15.116:0/4109471582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ec0101850 msgr2=0x7f3ec0197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec5746640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ec0101850 0x7f3ec0197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec5746640 1 -- 172.21.15.116:0/4109471582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ec0102a40 msgr2=0x7f3ec01980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec5746640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ec0102a40 0x7f3ec01980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.569+0000 7f3ec5746640 1 -- 172.21.15.116:0/4109471582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec019f7c0 con 0x7f3ec0103400 2024-03-19T01:09:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.570+0000 7f3ec5746640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec0103400 0x7f3ec019f150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8037040 tx=0x7f3eb8032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.570+0000 7f3ebdffb640 1 -- 172.21.15.116:0/4109471582 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8039040 con 0x7f3ec0103400 2024-03-19T01:09:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.570+0000 7f3ec71a9640 1 -- 172.21.15.116:0/4109471582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec019fa50 con 0x7f3ec0103400 2024-03-19T01:09:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.570+0000 7f3ebdffb640 1 -- 172.21.15.116:0/4109471582 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3eb8044070 con 0x7f3ec0103400 2024-03-19T01:09:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.570+0000 7f3ebdffb640 1 -- 172.21.15.116:0/4109471582 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8004260 con 0x7f3ec0103400 2024-03-19T01:09:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.570+0000 7f3ec71a9640 1 -- 172.21.15.116:0/4109471582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec019ff90 con 0x7f3ec0103400 2024-03-19T01:09:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.571+0000 7f3ec71a9640 1 -- 172.21.15.116:0/4109471582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ec0108b70 con 0x7f3ec0103400 2024-03-19T01:09:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.572+0000 7f3ebdffb640 1 -- 172.21.15.116:0/4109471582 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f3eb8003970 con 0x7f3ec0103400 2024-03-19T01:09:53.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.573+0000 7f3ebdffb640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3ea80a2800 0x7f3ea80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:53.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.573+0000 7f3ec4f45640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3ea80a2800 0x7f3ea80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:53.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.573+0000 7f3ebdffb640 1 -- 172.21.15.116:0/4109471582 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3eb80f0cc0 con 0x7f3ec0103400 2024-03-19T01:09:53.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.573+0000 7f3ec4f45640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3ea80a2800 0x7f3ea80a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3ec01028a0 tx=0x7f3eb000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:53.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.577+0000 7f3ebdffb640 1 -- 172.21.15.116:0/4109471582 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3eb80be200 con 0x7f3ec0103400 2024-03-19T01:09:53.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.751+0000 7f3ec71a9640 1 -- 172.21.15.116:0/4109471582 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3ec002d1f0 con 0x7f3ec0103400 2024-03-19T01:09:53.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.752+0000 7f3ebdffb640 1 -- 172.21.15.116:0/4109471582 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f3eb80bdb00 con 0x7f3ec0103400 2024-03-19T01:09:53.753 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T01:09:53.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.754+0000 7f3ec71a9640 1 -- 172.21.15.116:0/4109471582 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3ea80a2800 msgr2=0x7f3ea80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:53.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.754+0000 7f3ec71a9640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3ea80a2800 0x7f3ea80a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3ec01028a0 tx=0x7f3eb000a2d0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.754+0000 7f3ec71a9640 1 -- 172.21.15.116:0/4109471582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec0103400 msgr2=0x7f3ec019f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:53.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.754+0000 7f3ec71a9640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec0103400 0x7f3ec019f150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8037040 tx=0x7f3eb8032f90 comp rx=0 tx=0).stop 2024-03-19T01:09:53.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.754+0000 7f3ec71a9640 1 -- 172.21.15.116:0/4109471582 shutdown_connections 2024-03-19T01:09:53.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.754+0000 7f3ec71a9640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ec0103400 0x7f3ec019f150 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.754+0000 7f3ec71a9640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3ea80a2800 0x7f3ea80a4cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.754+0000 7f3ec71a9640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ec0102a40 0x7f3ec01980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.754+0000 7f3ec71a9640 1 --2- 172.21.15.116:0/4109471582 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3ec0101850 0x7f3ec0197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.754+0000 7f3ec71a9640 1 -- 172.21.15.116:0/4109471582 >> 172.21.15.116:0/4109471582 conn(0x7f3ec00fcdc0 msgr2=0x7f3ec0105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:53.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.754+0000 7f3ec71a9640 1 -- 172.21.15.116:0/4109471582 shutdown_connections 2024-03-19T01:09:53.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.754+0000 7f3ec71a9640 1 -- 172.21.15.116:0/4109471582 wait complete. 2024-03-19T01:09:53.765 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:09:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.866+0000 7fb8bd537640 1 -- 172.21.15.116:0/786839306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b8100760 msgr2=0x7fb8b8100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.866+0000 7fb8bd537640 1 --2- 172.21.15.116:0/786839306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b8100760 0x7fb8b8100b80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8009900 tx=0x7fb8a802bc30 comp rx=0 tx=0).stop 2024-03-19T01:09:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.866+0000 7fb8bd537640 1 -- 172.21.15.116:0/786839306 shutdown_connections 2024-03-19T01:09:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.866+0000 7fb8bd537640 1 --2- 172.21.15.116:0/786839306 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8b8102310 0x7fb8b8108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.866+0000 7fb8bd537640 1 --2- 172.21.15.116:0/786839306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8b8101950 0x7fb8b8101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.866+0000 7fb8bd537640 1 --2- 172.21.15.116:0/786839306 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b8100760 0x7fb8b8100b80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.866+0000 7fb8bd537640 1 -- 172.21.15.116:0/786839306 >> 172.21.15.116:0/786839306 conn(0x7fb8b80fbcd0 msgr2=0x7fb8b80fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.866+0000 7fb8bd537640 1 -- 172.21.15.116:0/786839306 shutdown_connections 2024-03-19T01:09:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.866+0000 7fb8bd537640 1 -- 172.21.15.116:0/786839306 wait complete. 2024-03-19T01:09:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.866+0000 7fb8bd537640 1 Processor -- start 2024-03-19T01:09:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8bd537640 1 -- start start 2024-03-19T01:09:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8bd537640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8b8100760 0x7fb8b8197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8bd537640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8b8101950 0x7fb8b8198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8bd537640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b8102310 0x7fb8b819f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8bd537640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b810c3e0 con 0x7fb8b8102310 2024-03-19T01:09:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8bd537640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b810c260 con 0x7fb8b8100760 2024-03-19T01:09:53.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8bd537640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b810c560 con 0x7fb8b8101950 2024-03-19T01:09:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b8102310 0x7fb8b819f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8b67fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8b8101950 0x7fb8b8198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b8102310 0x7fb8b819f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50938/0 (socket says 172.21.15.116:50938) 2024-03-19T01:09:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8b77fe640 1 -- 172.21.15.116:0/126766766 learned_addr learned my addr 172.21.15.116:0/126766766 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:09:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8b6ffd640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8b8100760 0x7fb8b8197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8b77fe640 1 -- 172.21.15.116:0/126766766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8b8101950 msgr2=0x7fb8b8198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8b77fe640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8b8101950 0x7fb8b8198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8b77fe640 1 -- 172.21.15.116:0/126766766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8b8100760 msgr2=0x7fb8b8197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8b77fe640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8b8100760 0x7fb8b8197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8b77fe640 1 -- 172.21.15.116:0/126766766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b819f7d0 con 0x7fb8b8102310 2024-03-19T01:09:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8b6ffd640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8b8100760 0x7fb8b8197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:09:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb8b77fe640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b8102310 0x7fb8b819f160 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000eaa0 tx=0x7fb8b000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:53.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.867+0000 7fb89ffff640 1 -- 172.21.15.116:0/126766766 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b000cdb0 con 0x7fb8b8102310 2024-03-19T01:09:53.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.868+0000 7fb89ffff640 1 -- 172.21.15.116:0/126766766 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8b0005020 con 0x7fb8b8102310 2024-03-19T01:09:53.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.868+0000 7fb89ffff640 1 -- 172.21.15.116:0/126766766 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b00116b0 con 0x7fb8b8102310 2024-03-19T01:09:53.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.868+0000 7fb8bd537640 1 -- 172.21.15.116:0/126766766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b819fac0 con 0x7fb8b8102310 2024-03-19T01:09:53.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.868+0000 7fb8bd537640 1 -- 172.21.15.116:0/126766766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b81a0060 con 0x7fb8b8102310 2024-03-19T01:09:53.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.868+0000 7fb8bd537640 1 -- 172.21.15.116:0/126766766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b81a0290 con 0x7fb8b8102310 2024-03-19T01:09:53.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.870+0000 7fb89ffff640 1 -- 172.21.15.116:0/126766766 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fb8b0005560 con 0x7fb8b8102310 2024-03-19T01:09:53.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.870+0000 7fb89ffff640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb8a00a2880 0x7fb8a00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:53.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.870+0000 7fb8b6ffd640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb8a00a2880 0x7fb8a00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:53.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.870+0000 7fb89ffff640 1 -- 172.21.15.116:0/126766766 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb8b00ce760 con 0x7fb8b8102310 2024-03-19T01:09:53.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.871+0000 7fb8b6ffd640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb8a00a2880 0x7fb8a00a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8036040 tx=0x7fb8a8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:53.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:53.874+0000 7fb89ffff640 1 -- 172.21.15.116:0/126766766 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb8b0016030 con 0x7fb8b8102310 2024-03-19T01:09:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.089+0000 7fb8bd537640 1 -- 172.21.15.116:0/126766766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8b81a0460 con 0x7fb8b8102310 2024-03-19T01:09:54.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.089+0000 7fb89ffff640 1 -- 172.21.15.116:0/126766766 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fb8b009bca0 con 0x7fb8b8102310 2024-03-19T01:09:54.090 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:09:54.091 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-19T01:09:54.091 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 1 2024-03-19T01:09:54.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.091+0000 7fb8bd537640 1 -- 172.21.15.116:0/126766766 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb8a00a2880 msgr2=0x7fb8a00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:54.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.091+0000 7fb8bd537640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb8a00a2880 0x7fb8a00a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb8a8036040 tx=0x7fb8a8000bc0 comp rx=0 tx=0).stop 2024-03-19T01:09:54.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.091+0000 7fb8bd537640 1 -- 172.21.15.116:0/126766766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b8102310 msgr2=0x7fb8b819f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:54.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.091+0000 7fb8bd537640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b8102310 0x7fb8b819f160 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000eaa0 tx=0x7fb8b000ef60 comp rx=0 tx=0).stop 2024-03-19T01:09:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.091+0000 7fb8bd537640 1 -- 172.21.15.116:0/126766766 shutdown_connections 2024-03-19T01:09:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.091+0000 7fb8bd537640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8b8102310 0x7fb8b819f160 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.091+0000 7fb8bd537640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb8b8101950 0x7fb8b8198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.091+0000 7fb8bd537640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb8a00a2880 0x7fb8a00a4d40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.091+0000 7fb8bd537640 1 --2- 172.21.15.116:0/126766766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8b8100760 0x7fb8b8197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.091+0000 7fb8bd537640 1 -- 172.21.15.116:0/126766766 >> 172.21.15.116:0/126766766 conn(0x7fb8b80fbcd0 msgr2=0x7fb8b80fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.092+0000 7fb8bd537640 1 -- 172.21.15.116:0/126766766 shutdown_connections 2024-03-19T01:09:54.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:54.092+0000 7fb8bd537640 1 -- 172.21.15.116:0/126766766 wait complete. 2024-03-19T01:09:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:54 smithi162 ceph-mon[67178]: Deploying daemon mds.l on smithi162 2024-03-19T01:09:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:54 smithi162 ceph-mon[67178]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:54 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2842895506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:09:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:54 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/4109471582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:09:54.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:54 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/126766766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:09:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:54 smithi116 ceph-mon[62701]: Deploying daemon mds.l on smithi162 2024-03-19T01:09:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:54 smithi116 ceph-mon[62701]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:54 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2842895506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:09:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:54 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4109471582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:09:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:54 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/126766766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:09:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:54 smithi156 ceph-mon[66799]: Deploying daemon mds.l on smithi162 2024-03-19T01:09:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:54 smithi156 ceph-mon[66799]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:54 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2842895506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:09:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:54 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/4109471582' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:09:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:54 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/126766766' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:09:55.149 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:09:55 smithi162 systemd[1]: Starting Ceph mds.l for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:09:55.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:55 smithi116 ceph-mon[62701]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:55.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:55 smithi156 ceph-mon[66799]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:55.482 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:55 smithi162 ceph-mon[67178]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:55.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:09:55 smithi162 podman[97335]: 2024-03-19 01:09:55.131173262 +0000 UTC m=+0.017292483 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:09:55.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:09:55 smithi162 podman[97335]: 2024-03-19 01:09:55.255290245 +0000 UTC m=+0.141409448 container create e7c84b2d954c7ed2a87087912d7eff4eafb7311a415d5c5d8f46303dd34ed6ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-l, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-19T01:09:55.483 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:09:55 smithi162 podman[97335]: 2024-03-19 01:09:55.482773638 +0000 UTC m=+0.368892847 container init e7c84b2d954c7ed2a87087912d7eff4eafb7311a415d5c5d8f46303dd34ed6ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-l, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-19T01:09:55.899 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:09:55 smithi162 podman[97335]: 2024-03-19 01:09:55.485568566 +0000 UTC m=+0.371687783 container start e7c84b2d954c7ed2a87087912d7eff4eafb7311a415d5c5d8f46303dd34ed6ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-l, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=) 2024-03-19T01:09:55.899 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:09:55 smithi162 ceph-mds[97354]: -- 172.21.15.162:0/1094307480 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560962699a40 con 0x56096266d000 2024-03-19T01:09:55.899 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:09:55 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-l[97350]: starting mds.l at 2024-03-19T01:09:55.899 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:09:55 smithi162 bash[97335]: e7c84b2d954c7ed2a87087912d7eff4eafb7311a415d5c5d8f46303dd34ed6ad 2024-03-19T01:09:55.899 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:09:55 smithi162 systemd[1]: Started Ceph mds.l for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:09:56.103 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:09:56.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.215+0000 7fdabd844640 1 -- 172.21.15.116:0/117718892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101930 msgr2=0x7fdab8101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:56.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.215+0000 7fdabd844640 1 --2- 172.21.15.116:0/117718892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101930 0x7fdab8101db0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fdaac009900 tx=0x7fdaac02bc30 comp rx=0 tx=0).stop 2024-03-19T01:09:56.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.215+0000 7fdabd844640 1 -- 172.21.15.116:0/117718892 shutdown_connections 2024-03-19T01:09:56.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.215+0000 7fdabd844640 1 --2- 172.21.15.116:0/117718892 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdab81022f0 0x7fdab8108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:56.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.215+0000 7fdabd844640 1 --2- 172.21.15.116:0/117718892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101930 0x7fdab8101db0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:56.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.215+0000 7fdabd844640 1 --2- 172.21.15.116:0/117718892 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdab8100740 0x7fdab8100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:56.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.215+0000 7fdabd844640 1 -- 172.21.15.116:0/117718892 >> 172.21.15.116:0/117718892 conn(0x7fdab80fbcb0 msgr2=0x7fdab80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:56.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.215+0000 7fdabd844640 1 -- 172.21.15.116:0/117718892 shutdown_connections 2024-03-19T01:09:56.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.215+0000 7fdabd844640 1 -- 172.21.15.116:0/117718892 wait complete. 2024-03-19T01:09:56.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.215+0000 7fdabd844640 1 Processor -- start 2024-03-19T01:09:56.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.215+0000 7fdabd844640 1 -- start start 2024-03-19T01:09:56.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdabd844640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdab8100740 0x7fdab8199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:56.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdabd844640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101930 0x7fdab819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:56.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdabd844640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdab81022f0 0x7fdab81a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:56.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdabd844640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdab810d4c0 con 0x7fdab8101930 2024-03-19T01:09:56.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdabd844640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdab810d340 con 0x7fdab81022f0 2024-03-19T01:09:56.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdabd844640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdab810d640 con 0x7fdab8100740 2024-03-19T01:09:56.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdab67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101930 0x7fdab819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:56.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdab67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101930 0x7fdab819a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50962/0 (socket says 172.21.15.116:50962) 2024-03-19T01:09:56.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdab67fc640 1 -- 172.21.15.116:0/1588338635 learned_addr learned my addr 172.21.15.116:0/1588338635 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:09:56.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdab6ffd640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdab8100740 0x7fdab8199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:56.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdab77fe640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdab81022f0 0x7fdab81a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:56.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdab67fc640 1 -- 172.21.15.116:0/1588338635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdab8100740 msgr2=0x7fdab8199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:56.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdab67fc640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdab8100740 0x7fdab8199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:56.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdab67fc640 1 -- 172.21.15.116:0/1588338635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdab81022f0 msgr2=0x7fdab81a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:56.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdab67fc640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdab81022f0 0x7fdab81a12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:56.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdab67fc640 1 -- 172.21.15.116:0/1588338635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab81a1950 con 0x7fdab8101930 2024-03-19T01:09:56.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdab67fc640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101930 0x7fdab819a280 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdaac009230 tx=0x7fdaac005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:56.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdabc842640 1 -- 172.21.15.116:0/1588338635 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac041070 con 0x7fdab8101930 2024-03-19T01:09:56.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdabd844640 1 -- 172.21.15.116:0/1588338635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab81a1be0 con 0x7fdab8101930 2024-03-19T01:09:56.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdabc842640 1 -- 172.21.15.116:0/1588338635 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdaac0044d0 con 0x7fdab8101930 2024-03-19T01:09:56.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdabc842640 1 -- 172.21.15.116:0/1588338635 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac034440 con 0x7fdab8101930 2024-03-19T01:09:56.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.216+0000 7fdabd844640 1 -- 172.21.15.116:0/1588338635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab81a20d0 con 0x7fdab8101930 2024-03-19T01:09:56.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.217+0000 7fdabd844640 1 -- 172.21.15.116:0/1588338635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab81a22a0 con 0x7fdab8101930 2024-03-19T01:09:56.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.219+0000 7fdabc842640 1 -- 172.21.15.116:0/1588338635 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fdaac044070 con 0x7fdab8101930 2024-03-19T01:09:56.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.219+0000 7fdabc842640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdaa00a2800 0x7fdaa00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:56.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.219+0000 7fdab6ffd640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdaa00a2800 0x7fdaa00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:56.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.219+0000 7fdabc842640 1 -- 172.21.15.116:0/1588338635 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdaac0f05d0 con 0x7fdab8101930 2024-03-19T01:09:56.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.220+0000 7fdab6ffd640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdaa00a2800 0x7fdaa00a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdab819b210 tx=0x7fdaa8007600 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:56.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.223+0000 7fdabc842640 1 -- 172.21.15.116:0/1588338635 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdaac0bdb10 con 0x7fdab8101930 2024-03-19T01:09:56.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.434+0000 7fdabd844640 1 -- 172.21.15.116:0/1588338635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdab81a2470 con 0x7fdab8101930 2024-03-19T01:09:56.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.435+0000 7fdabc842640 1 -- 172.21.15.116:0/1588338635 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fdaac0bd410 con 0x7fdab8101930 2024-03-19T01:09:56.436 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:09:56.437 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-19T01:09:56.437 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 2 2024-03-19T01:09:56.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.437+0000 7fdabd844640 1 -- 172.21.15.116:0/1588338635 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdaa00a2800 msgr2=0x7fdaa00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:56.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.437+0000 7fdabd844640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdaa00a2800 0x7fdaa00a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdab819b210 tx=0x7fdaa8007600 comp rx=0 tx=0).stop 2024-03-19T01:09:56.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.437+0000 7fdabd844640 1 -- 172.21.15.116:0/1588338635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101930 msgr2=0x7fdab819a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.437+0000 7fdabd844640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101930 0x7fdab819a280 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdaac009230 tx=0x7fdaac005d50 comp rx=0 tx=0).stop 2024-03-19T01:09:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.437+0000 7fdabd844640 1 -- 172.21.15.116:0/1588338635 shutdown_connections 2024-03-19T01:09:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.437+0000 7fdabd844640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdaa00a2800 0x7fdaa00a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.437+0000 7fdabd844640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdab81022f0 0x7fdab81a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:56.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.437+0000 7fdabd844640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdab8101930 0x7fdab819a280 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.437+0000 7fdabd844640 1 --2- 172.21.15.116:0/1588338635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdab8100740 0x7fdab8199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.437+0000 7fdabd844640 1 -- 172.21.15.116:0/1588338635 >> 172.21.15.116:0/1588338635 conn(0x7fdab80fbcb0 msgr2=0x7fdab810a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.437+0000 7fdabd844640 1 -- 172.21.15.116:0/1588338635 shutdown_connections 2024-03-19T01:09:56.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:56.437+0000 7fdabd844640 1 -- 172.21.15.116:0/1588338635 wait complete. 2024-03-19T01:09:56.555 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:56 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.162:6832/3113382416,v1:172.21.15.162:6833/3113382416] up:boot 2024-03-19T01:09:56.555 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:56 smithi156 ceph-mon[66799]: fsmap 1 up:standby 2024-03-19T01:09:56.555 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:56 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T01:09:56.555 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:56 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:56.556 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:56 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:56.556 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:56 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:56.556 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:56 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:09:56.556 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:56 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:09:56.556 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:56 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:56.556 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:56 smithi156 ceph-mon[66799]: Deploying daemon mds.k on smithi156 2024-03-19T01:09:56.557 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:56 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1588338635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:09:56.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:56 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.162:6832/3113382416,v1:172.21.15.162:6833/3113382416] up:boot 2024-03-19T01:09:56.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:56 smithi162 ceph-mon[67178]: fsmap 1 up:standby 2024-03-19T01:09:56.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:56 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T01:09:56.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:56 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:56.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:56 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:56.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:56 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:56.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:56 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:09:56.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:56 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:09:56.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:56 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:56.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:56 smithi162 ceph-mon[67178]: Deploying daemon mds.k on smithi156 2024-03-19T01:09:56.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:56 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1588338635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:09:56.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:56 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.162:6832/3113382416,v1:172.21.15.162:6833/3113382416] up:boot 2024-03-19T01:09:56.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:56 smithi116 ceph-mon[62701]: fsmap 1 up:standby 2024-03-19T01:09:56.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T01:09:56.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:56.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:56.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:56.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:09:56.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:09:56.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:56.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:56 smithi116 ceph-mon[62701]: Deploying daemon mds.k on smithi156 2024-03-19T01:09:56.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:56 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1588338635' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:09:57.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:57 smithi156 ceph-mon[66799]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:57.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:57 smithi162 ceph-mon[67178]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:57.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:57 smithi116 ceph-mon[62701]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:58.188 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:09:57 smithi156 systemd[1]: Starting Ceph mds.k for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:09:58.440 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:09:58 smithi156 podman[96358]: 2024-03-19 01:09:58.090467581 +0000 UTC m=+0.017765545 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:09:58.440 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:09:58 smithi156 podman[96358]: 2024-03-19 01:09:58.203819337 +0000 UTC m=+0.131117315 container create 128f16c9a1babe1a67640a1f0710cde7f1356d1a84ac146e0d1d351fcd3e6674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-k, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:09:58.440 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:09:58 smithi156 podman[96358]: 2024-03-19 01:09:58.439682483 +0000 UTC m=+0.366980449 container init 128f16c9a1babe1a67640a1f0710cde7f1356d1a84ac146e0d1d351fcd3e6674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-k, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:09:58.449 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:09:58.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.563+0000 7f6bca8c6640 1 -- 172.21.15.116:0/3393268281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc4100740 msgr2=0x7f6bc4100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.563+0000 7f6bca8c6640 1 --2- 172.21.15.116:0/3393268281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc4100740 0x7f6bc4100b60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4009900 tx=0x7f6bb402bc30 comp rx=0 tx=0).stop 2024-03-19T01:09:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.564+0000 7f6bca8c6640 1 -- 172.21.15.116:0/3393268281 shutdown_connections 2024-03-19T01:09:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.564+0000 7f6bca8c6640 1 --2- 172.21.15.116:0/3393268281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bc41022f0 0x7f6bc4108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.564+0000 7f6bca8c6640 1 --2- 172.21.15.116:0/3393268281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bc4101930 0x7f6bc4101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.564+0000 7f6bca8c6640 1 --2- 172.21.15.116:0/3393268281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc4100740 0x7f6bc4100b60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.564+0000 7f6bca8c6640 1 -- 172.21.15.116:0/3393268281 >> 172.21.15.116:0/3393268281 conn(0x7f6bc40fbcb0 msgr2=0x7f6bc40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.564+0000 7f6bca8c6640 1 -- 172.21.15.116:0/3393268281 shutdown_connections 2024-03-19T01:09:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.564+0000 7f6bca8c6640 1 -- 172.21.15.116:0/3393268281 wait complete. 2024-03-19T01:09:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.564+0000 7f6bca8c6640 1 Processor -- start 2024-03-19T01:09:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.564+0000 7f6bca8c6640 1 -- start start 2024-03-19T01:09:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bca8c6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bc4100740 0x7f6bc4197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:58.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bca8c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc4101930 0x7f6bc41981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bca8c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bc41022f0 0x7f6bc419f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bca8c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc410c410 con 0x7f6bc4101930 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bca8c6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc410c290 con 0x7f6bc4100740 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bca8c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc410c590 con 0x7f6bc41022f0 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc4101930 0x7f6bc41981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc4101930 0x7f6bc41981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41020/0 (socket says 172.21.15.116:41020) 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc37fe640 1 -- 172.21.15.116:0/2535570534 learned_addr learned my addr 172.21.15.116:0/2535570534 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc3fff640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bc4100740 0x7f6bc4197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc8e63640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bc41022f0 0x7f6bc419f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc37fe640 1 -- 172.21.15.116:0/2535570534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bc41022f0 msgr2=0x7f6bc419f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc37fe640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bc41022f0 0x7f6bc419f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc37fe640 1 -- 172.21.15.116:0/2535570534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bc4100740 msgr2=0x7f6bc4197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc37fe640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bc4100740 0x7f6bc4197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:58.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc37fe640 1 -- 172.21.15.116:0/2535570534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc419f890 con 0x7f6bc4101930 2024-03-19T01:09:58.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc37fe640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc4101930 0x7f6bc41981c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6bb800b4a0 tx=0x7f6bb800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:58.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc17fa640 1 -- 172.21.15.116:0/2535570534 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8014070 con 0x7f6bc4101930 2024-03-19T01:09:58.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bca8c6640 1 -- 172.21.15.116:0/2535570534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bc419fb80 con 0x7f6bc4101930 2024-03-19T01:09:58.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc17fa640 1 -- 172.21.15.116:0/2535570534 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bb8010af0 con 0x7f6bc4101930 2024-03-19T01:09:58.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bc17fa640 1 -- 172.21.15.116:0/2535570534 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bb8017d10 con 0x7f6bc4101930 2024-03-19T01:09:58.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.565+0000 7f6bca8c6640 1 -- 172.21.15.116:0/2535570534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bc41a00a0 con 0x7f6bc4101930 2024-03-19T01:09:58.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.566+0000 7f6bca8c6640 1 -- 172.21.15.116:0/2535570534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bc41a0270 con 0x7f6bc4101930 2024-03-19T01:09:58.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.567+0000 7f6bc17fa640 1 -- 172.21.15.116:0/2535570534 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f6bb8017070 con 0x7f6bc4101930 2024-03-19T01:09:58.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.568+0000 7f6bc17fa640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6bac0a2800 0x7f6bac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:09:58.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.568+0000 7f6bc3fff640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6bac0a2800 0x7f6bac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:09:58.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.568+0000 7f6bc17fa640 1 -- 172.21.15.116:0/2535570534 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6bb80cde00 con 0x7f6bc4101930 2024-03-19T01:09:58.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.569+0000 7f6bc3fff640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6bac0a2800 0x7f6bac0a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4000c00 tx=0x7f6bb4005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:09:58.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.572+0000 7f6bc17fa640 1 -- 172.21.15.116:0/2535570534 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6bb809b340 con 0x7f6bc4101930 2024-03-19T01:09:58.724 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:09:58 smithi156 podman[96358]: 2024-03-19 01:09:58.442566378 +0000 UTC m=+0.369864336 container start 128f16c9a1babe1a67640a1f0710cde7f1356d1a84ac146e0d1d351fcd3e6674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-k, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-19T01:09:58.724 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:09:58 smithi156 ceph-mds[96377]: -- 172.21.15.156:0/756303458 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b7a7d67a40 con 0x55b7a7d3ac00 2024-03-19T01:09:58.724 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:09:58 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-k[96373]: starting mds.k at 2024-03-19T01:09:58.724 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:09:58 smithi156 bash[96358]: 128f16c9a1babe1a67640a1f0710cde7f1356d1a84ac146e0d1d351fcd3e6674 2024-03-19T01:09:58.724 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:09:58 smithi156 systemd[1]: Started Ceph mds.k for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:09:58.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.793+0000 7f6bca8c6640 1 -- 172.21.15.116:0/2535570534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6bc41a0440 con 0x7f6bc4101930 2024-03-19T01:09:58.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.794+0000 7f6bc17fa640 1 -- 172.21.15.116:0/2535570534 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f6bb809ac40 con 0x7f6bc4101930 2024-03-19T01:09:58.795 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:09:58.795 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-19T01:09:58.795 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 3 2024-03-19T01:09:58.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.796+0000 7f6bca8c6640 1 -- 172.21.15.116:0/2535570534 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6bac0a2800 msgr2=0x7f6bac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.796+0000 7f6bca8c6640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6bac0a2800 0x7f6bac0a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4000c00 tx=0x7f6bb4005fd0 comp rx=0 tx=0).stop 2024-03-19T01:09:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.796+0000 7f6bca8c6640 1 -- 172.21.15.116:0/2535570534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc4101930 msgr2=0x7f6bc41981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:09:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.796+0000 7f6bca8c6640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc4101930 0x7f6bc41981c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6bb800b4a0 tx=0x7f6bb800b960 comp rx=0 tx=0).stop 2024-03-19T01:09:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.796+0000 7f6bca8c6640 1 -- 172.21.15.116:0/2535570534 shutdown_connections 2024-03-19T01:09:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.796+0000 7f6bca8c6640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bc41022f0 0x7f6bc419f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.796+0000 7f6bca8c6640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bc4101930 0x7f6bc41981c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.796+0000 7f6bca8c6640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6bac0a2800 0x7f6bac0a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.796+0000 7f6bca8c6640 1 --2- 172.21.15.116:0/2535570534 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bc4100740 0x7f6bc4197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:09:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.796+0000 7f6bca8c6640 1 -- 172.21.15.116:0/2535570534 >> 172.21.15.116:0/2535570534 conn(0x7f6bc40fbcb0 msgr2=0x7f6bc40689d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:09:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.796+0000 7f6bca8c6640 1 -- 172.21.15.116:0/2535570534 shutdown_connections 2024-03-19T01:09:58.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:09:58.796+0000 7f6bca8c6640 1 -- 172.21.15.116:0/2535570534 wait complete. 2024-03-19T01:09:59.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:59 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.156:6832/1647822466,v1:172.21.15.156:6833/1647822466] up:boot 2024-03-19T01:09:59.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:59 smithi162 ceph-mon[67178]: fsmap 2 up:standby 2024-03-19T01:09:59.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T01:09:59.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:59.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:59.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:59.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:09:59.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:09:59.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:59.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:59 smithi162 ceph-mon[67178]: Deploying daemon mds.j on smithi116 2024-03-19T01:09:59.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:59 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2535570534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:09:59.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:09:59 smithi162 ceph-mon[67178]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:59.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:59 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.156:6832/1647822466,v1:172.21.15.156:6833/1647822466] up:boot 2024-03-19T01:09:59.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:59 smithi116 ceph-mon[62701]: fsmap 2 up:standby 2024-03-19T01:09:59.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T01:09:59.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:59.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:59.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:59.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:09:59.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:09:59.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:59.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:59 smithi116 ceph-mon[62701]: Deploying daemon mds.j on smithi116 2024-03-19T01:09:59.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:59 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2535570534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:09:59.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:09:59 smithi116 ceph-mon[62701]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:09:59.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:59 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.156:6832/1647822466,v1:172.21.15.156:6833/1647822466] up:boot 2024-03-19T01:09:59.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:59 smithi156 ceph-mon[66799]: fsmap 2 up:standby 2024-03-19T01:09:59.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T01:09:59.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:59.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:59.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:09:59.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:09:59.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:09:59.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:09:59.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:59 smithi156 ceph-mon[66799]: Deploying daemon mds.j on smithi116 2024-03-19T01:09:59.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:59 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2535570534' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:09:59.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:09:59 smithi156 ceph-mon[66799]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:00.808 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:00.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:00 smithi162 ceph-mon[67178]: overall HEALTH_OK 2024-03-19T01:10:00.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:00 smithi116 ceph-mon[62701]: overall HEALTH_OK 2024-03-19T01:10:00.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:00 smithi156 ceph-mon[66799]: overall HEALTH_OK 2024-03-19T01:10:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.973+0000 7f2d6c95b640 1 -- 172.21.15.116:0/3725630685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d68102ac0 msgr2=0x7f2d68102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.973+0000 7f2d6c95b640 1 --2- 172.21.15.116:0/3725630685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d68102ac0 0x7f2d68102f40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c009930 tx=0x7f2d5c02bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.973+0000 7f2d6c95b640 1 -- 172.21.15.116:0/3725630685 shutdown_connections 2024-03-19T01:10:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.973+0000 7f2d6c95b640 1 --2- 172.21.15.116:0/3725630685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68103480 0x7f2d68109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.973+0000 7f2d6c95b640 1 --2- 172.21.15.116:0/3725630685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d68102ac0 0x7f2d68102f40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.973+0000 7f2d6c95b640 1 --2- 172.21.15.116:0/3725630685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d680ffaf0 0x7f2d680fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.973+0000 7f2d6c95b640 1 -- 172.21.15.116:0/3725630685 >> 172.21.15.116:0/3725630685 conn(0x7f2d680fb420 msgr2=0x7f2d680fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.973+0000 7f2d6c95b640 1 -- 172.21.15.116:0/3725630685 shutdown_connections 2024-03-19T01:10:00.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.973+0000 7f2d6c95b640 1 -- 172.21.15.116:0/3725630685 wait complete. 2024-03-19T01:10:00.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6c95b640 1 Processor -- start 2024-03-19T01:10:00.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6c95b640 1 -- start start 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6c95b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d680ffaf0 0x7f2d68197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6c95b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d68102ac0 0x7f2d68198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6c95b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68103480 0x7f2d6819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6c95b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6810d4b0 con 0x7f2d680ffaf0 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6c95b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6810d330 con 0x7f2d68102ac0 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6c95b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6810d630 con 0x7f2d68103480 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d680ffaf0 0x7f2d68197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d680ffaf0 0x7f2d68197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41034/0 (socket says 172.21.15.116:41034) 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6659c640 1 -- 172.21.15.116:0/1782175409 learned_addr learned my addr 172.21.15.116:0/1782175409 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d66d9d640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68103480 0x7f2d6819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d65d9b640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d68102ac0 0x7f2d68198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6659c640 1 -- 172.21.15.116:0/1782175409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68103480 msgr2=0x7f2d6819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6659c640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68103480 0x7f2d6819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:00.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6659c640 1 -- 172.21.15.116:0/1782175409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d68102ac0 msgr2=0x7f2d68198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6659c640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d68102ac0 0x7f2d68198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d6659c640 1 -- 172.21.15.116:0/1782175409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d68100930 con 0x7f2d680ffaf0 2024-03-19T01:10:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.974+0000 7f2d65d9b640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d68102ac0 0x7f2d68198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:10:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.975+0000 7f2d6659c640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d680ffaf0 0x7f2d68197b20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800d950 tx=0x7f2d5800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.975+0000 7f2d577fe640 1 -- 172.21.15.116:0/1782175409 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d58017020 con 0x7f2d680ffaf0 2024-03-19T01:10:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.975+0000 7f2d6c95b640 1 -- 172.21.15.116:0/1782175409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d68100c20 con 0x7f2d680ffaf0 2024-03-19T01:10:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.975+0000 7f2d577fe640 1 -- 172.21.15.116:0/1782175409 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d58009c30 con 0x7f2d680ffaf0 2024-03-19T01:10:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.975+0000 7f2d577fe640 1 -- 172.21.15.116:0/1782175409 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5800f740 con 0x7f2d680ffaf0 2024-03-19T01:10:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.975+0000 7f2d6c95b640 1 -- 172.21.15.116:0/1782175409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d68101170 con 0x7f2d680ffaf0 2024-03-19T01:10:00.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.976+0000 7f2d6c95b640 1 -- 172.21.15.116:0/1782175409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d68102600 con 0x7f2d680ffaf0 2024-03-19T01:10:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.977+0000 7f2d577fe640 1 -- 172.21.15.116:0/1782175409 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f2d5801a020 con 0x7f2d680ffaf0 2024-03-19T01:10:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.978+0000 7f2d577fe640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2d500a2800 0x7f2d500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.978+0000 7f2d65d9b640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2d500a2800 0x7f2d500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.978+0000 7f2d577fe640 1 -- 172.21.15.116:0/1782175409 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2d580ce090 con 0x7f2d680ffaf0 2024-03-19T01:10:00.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.978+0000 7f2d65d9b640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2d500a2800 0x7f2d500a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0096c0 tx=0x7f2d5c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:00.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:00.982+0000 7f2d577fe640 1 -- 172.21.15.116:0/1782175409 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2d58021070 con 0x7f2d680ffaf0 2024-03-19T01:10:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.205+0000 7f2d6c95b640 1 -- 172.21.15.116:0/1782175409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d68102800 con 0x7f2d680ffaf0 2024-03-19T01:10:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.206+0000 7f2d577fe640 1 -- 172.21.15.116:0/1782175409 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f2d68102800 con 0x7f2d680ffaf0 2024-03-19T01:10:01.207 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:01.207 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-19T01:10:01.207 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 3 2024-03-19T01:10:01.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.208+0000 7f2d6c95b640 1 -- 172.21.15.116:0/1782175409 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2d500a2800 msgr2=0x7f2d500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:01.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.208+0000 7f2d6c95b640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2d500a2800 0x7f2d500a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0096c0 tx=0x7f2d5c005fd0 comp rx=0 tx=0).stop 2024-03-19T01:10:01.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.208+0000 7f2d6c95b640 1 -- 172.21.15.116:0/1782175409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d680ffaf0 msgr2=0x7f2d68197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:01.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.208+0000 7f2d6c95b640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d680ffaf0 0x7f2d68197b20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800d950 tx=0x7f2d5800de10 comp rx=0 tx=0).stop 2024-03-19T01:10:01.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.208+0000 7f2d6c95b640 1 -- 172.21.15.116:0/1782175409 shutdown_connections 2024-03-19T01:10:01.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.208+0000 7f2d6c95b640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d68103480 0x7f2d6819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:01.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.208+0000 7f2d6c95b640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2d500a2800 0x7f2d500a4cc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:01.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.208+0000 7f2d6c95b640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d68102ac0 0x7f2d68198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:01.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.208+0000 7f2d6c95b640 1 --2- 172.21.15.116:0/1782175409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d680ffaf0 0x7f2d68197b20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:01.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.208+0000 7f2d6c95b640 1 -- 172.21.15.116:0/1782175409 >> 172.21.15.116:0/1782175409 conn(0x7f2d680fb420 msgr2=0x7f2d68105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:01.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.208+0000 7f2d6c95b640 1 -- 172.21.15.116:0/1782175409 shutdown_connections 2024-03-19T01:10:01.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:01.208+0000 7f2d6c95b640 1 -- 172.21.15.116:0/1782175409 wait complete. 2024-03-19T01:10:01.219 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 01:10:01 smithi116 podman[103008]: 2024-03-19 01:10:01.032246982 +0000 UTC m=+0.018310006 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:10:01.652 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 01:10:01 smithi116 podman[103008]: 2024-03-19 01:10:01.353250046 +0000 UTC m=+0.339313061 container init cd92888c4567cd245e61110af2ccabc38940d753a73bdb4e38247b0752695d29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-j, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-19T01:10:01.652 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 01:10:01 smithi116 podman[103008]: 2024-03-19 01:10:01.35599559 +0000 UTC m=+0.342058585 container start cd92888c4567cd245e61110af2ccabc38940d753a73bdb4e38247b0752695d29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-19T01:10:01.652 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 01:10:01 smithi116 ceph-mds[103029]: -- 172.21.15.116:0/3375109700 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e57418da40 con 0x55e574160800 2024-03-19T01:10:01.652 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 01:10:01 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-j[103025]: starting mds.j at 2024-03-19T01:10:01.652 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 01:10:01 smithi116 bash[103008]: cd92888c4567cd245e61110af2ccabc38940d753a73bdb4e38247b0752695d29 2024-03-19T01:10:01.652 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 01:10:01 smithi116 systemd[1]: Started Ceph mds.j for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:10:01.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:01 smithi116 ceph-mon[62701]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:01.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:01 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1782175409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:01.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:01.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:01.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:01.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:01.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:01 smithi156 ceph-mon[66799]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:01.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:01 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1782175409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:01.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:01.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:01.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:01.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:01.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:01.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:02.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:01 smithi162 ceph-mon[67178]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:02.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:01 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1782175409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:02.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:02.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:02.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:02.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:02.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:02.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:02.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:02 smithi116 ceph-mon[62701]: Deploying daemon mds.d on smithi116 2024-03-19T01:10:02.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:02 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.116:6834/909235,v1:172.21.15.116:6835/909235] up:boot 2024-03-19T01:10:02.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:02 smithi116 ceph-mon[62701]: fsmap 3 up:standby 2024-03-19T01:10:02.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:02 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T01:10:02.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:02 smithi156 ceph-mon[66799]: Deploying daemon mds.d on smithi116 2024-03-19T01:10:02.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:02 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.116:6834/909235,v1:172.21.15.116:6835/909235] up:boot 2024-03-19T01:10:02.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:02 smithi156 ceph-mon[66799]: fsmap 3 up:standby 2024-03-19T01:10:02.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:02 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T01:10:03.155 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:02 smithi162 ceph-mon[67178]: Deploying daemon mds.d on smithi116 2024-03-19T01:10:03.155 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:02 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.116:6834/909235,v1:172.21.15.116:6835/909235] up:boot 2024-03-19T01:10:03.155 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:02 smithi162 ceph-mon[67178]: fsmap 3 up:standby 2024-03-19T01:10:03.155 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:02 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-19T01:10:03.221 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.329+0000 7f1f09b9d640 1 -- 172.21.15.116:0/1089563967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f04102850 msgr2=0x7f1f04102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.329+0000 7f1f09b9d640 1 --2- 172.21.15.116:0/1089563967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f04102850 0x7f1f04102cd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1ef0009900 tx=0x7f1ef002bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.329+0000 7f1f09b9d640 1 -- 172.21.15.116:0/1089563967 shutdown_connections 2024-03-19T01:10:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.329+0000 7f1f09b9d640 1 --2- 172.21.15.116:0/1089563967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f04103210 0x7f1f04109aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f09b9d640 1 --2- 172.21.15.116:0/1089563967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f04102850 0x7f1f04102cd0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f09b9d640 1 --2- 172.21.15.116:0/1089563967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f04101660 0x7f1f04101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f09b9d640 1 -- 172.21.15.116:0/1089563967 >> 172.21.15.116:0/1089563967 conn(0x7f1f040fcbb0 msgr2=0x7f1f040ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f09b9d640 1 -- 172.21.15.116:0/1089563967 shutdown_connections 2024-03-19T01:10:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f09b9d640 1 -- 172.21.15.116:0/1089563967 wait complete. 2024-03-19T01:10:03.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f09b9d640 1 Processor -- start 2024-03-19T01:10:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f09b9d640 1 -- start start 2024-03-19T01:10:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f09b9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f04101660 0x7f1f04197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f09b9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f04102850 0x7f1f04197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f09b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f04103210 0x7f1f0419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f09b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0410d360 con 0x7f1f04103210 2024-03-19T01:10:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f09b9d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0410d1e0 con 0x7f1f04101660 2024-03-19T01:10:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f09b9d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0410d4e0 con 0x7f1f04102850 2024-03-19T01:10:03.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f0939c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f04103210 0x7f1f0419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f0939c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f04103210 0x7f1f0419efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41074/0 (socket says 172.21.15.116:41074) 2024-03-19T01:10:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f0939c640 1 -- 172.21.15.116:0/3348156673 learned_addr learned my addr 172.21.15.116:0/3348156673 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.330+0000 7f1f03fff640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f04102850 0x7f1f04197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.331+0000 7f1f08b9b640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f04101660 0x7f1f04197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.331+0000 7f1f0939c640 1 -- 172.21.15.116:0/3348156673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f04102850 msgr2=0x7f1f04197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.331+0000 7f1f0939c640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f04102850 0x7f1f04197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.331+0000 7f1f0939c640 1 -- 172.21.15.116:0/3348156673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f04101660 msgr2=0x7f1f04197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.331+0000 7f1f0939c640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f04101660 0x7f1f04197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.331+0000 7f1f0939c640 1 -- 172.21.15.116:0/3348156673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f0419f610 con 0x7f1f04103210 2024-03-19T01:10:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.331+0000 7f1f08b9b640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f04101660 0x7f1f04197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:10:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.331+0000 7f1f0939c640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f04103210 0x7f1f0419efa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00eaa0 tx=0x7f1efc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.331+0000 7f1f01ffb640 1 -- 172.21.15.116:0/3348156673 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1efc00cdb0 con 0x7f1f04103210 2024-03-19T01:10:03.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.331+0000 7f1f09b9d640 1 -- 172.21.15.116:0/3348156673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f0419f900 con 0x7f1f04103210 2024-03-19T01:10:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.331+0000 7f1f01ffb640 1 -- 172.21.15.116:0/3348156673 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1efc005020 con 0x7f1f04103210 2024-03-19T01:10:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.331+0000 7f1f01ffb640 1 -- 172.21.15.116:0/3348156673 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1efc0116b0 con 0x7f1f04103210 2024-03-19T01:10:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.331+0000 7f1f09b9d640 1 -- 172.21.15.116:0/3348156673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f0419fe20 con 0x7f1f04103210 2024-03-19T01:10:03.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.332+0000 7f1f09b9d640 1 -- 172.21.15.116:0/3348156673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f04108980 con 0x7f1f04103210 2024-03-19T01:10:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.333+0000 7f1f01ffb640 1 -- 172.21.15.116:0/3348156673 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f1efc005560 con 0x7f1f04103210 2024-03-19T01:10:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.333+0000 7f1f01ffb640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1ef40a2800 0x7f1ef40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.334+0000 7f1f08b9b640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1ef40a2800 0x7f1ef40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.334+0000 7f1f01ffb640 1 -- 172.21.15.116:0/3348156673 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f1efc0ce830 con 0x7f1f04103210 2024-03-19T01:10:03.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.334+0000 7f1f08b9b640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1ef40a2800 0x7f1ef40a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8006fd0 tx=0x7f1ef8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:03.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.338+0000 7f1f01ffb640 1 -- 172.21.15.116:0/3348156673 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1efc016030 con 0x7f1f04103210 2024-03-19T01:10:03.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.561+0000 7f1f09b9d640 1 -- 172.21.15.116:0/3348156673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f0402d3a0 con 0x7f1f04103210 2024-03-19T01:10:03.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.565+0000 7f1f01ffb640 1 -- 172.21.15.116:0/3348156673 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3040 (secure 0 0 0) 0x7f1efc09bd70 con 0x7f1f04103210 2024-03-19T01:10:03.566 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:03.566 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-19T01:10:03.567 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 4 2024-03-19T01:10:03.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.568+0000 7f1f09b9d640 1 -- 172.21.15.116:0/3348156673 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1ef40a2800 msgr2=0x7f1ef40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:03.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.568+0000 7f1f09b9d640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1ef40a2800 0x7f1ef40a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8006fd0 tx=0x7f1ef8008040 comp rx=0 tx=0).stop 2024-03-19T01:10:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.568+0000 7f1f09b9d640 1 -- 172.21.15.116:0/3348156673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f04103210 msgr2=0x7f1f0419efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.568+0000 7f1f09b9d640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f04103210 0x7f1f0419efa0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1efc00eaa0 tx=0x7f1efc00ef60 comp rx=0 tx=0).stop 2024-03-19T01:10:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.569+0000 7f1f09b9d640 1 -- 172.21.15.116:0/3348156673 shutdown_connections 2024-03-19T01:10:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.569+0000 7f1f09b9d640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f04103210 0x7f1f0419efa0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.569+0000 7f1f09b9d640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f04102850 0x7f1f04197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.569+0000 7f1f09b9d640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1ef40a2800 0x7f1ef40a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.569+0000 7f1f09b9d640 1 --2- 172.21.15.116:0/3348156673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f04101660 0x7f1f04197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.569+0000 7f1f09b9d640 1 -- 172.21.15.116:0/3348156673 >> 172.21.15.116:0/3348156673 conn(0x7f1f040fcbb0 msgr2=0x7f1f04105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.569+0000 7f1f09b9d640 1 -- 172.21.15.116:0/3348156673 shutdown_connections 2024-03-19T01:10:03.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:03.569+0000 7f1f09b9d640 1 -- 172.21.15.116:0/3348156673 wait complete. 2024-03-19T01:10:03.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:03 smithi116 ceph-mon[62701]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:03.894 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:03 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3348156673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:03.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:03 smithi156 ceph-mon[66799]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:03.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:03 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3348156673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:04.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:03 smithi162 ceph-mon[67178]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:04.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:03 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3348156673' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:04.155 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 01:10:04 smithi116 podman[103360]: 2024-03-19 01:10:04.116117556 +0000 UTC m=+0.116893946 container create d66a0cad8ef2e659edd959d578248de9a5a60db789d638175c7062b76bc5a97c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-d, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-19T01:10:04.656 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 01:10:04 smithi116 podman[103360]: 2024-03-19 01:10:04.360775911 +0000 UTC m=+0.361552300 container init d66a0cad8ef2e659edd959d578248de9a5a60db789d638175c7062b76bc5a97c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-d, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-19T01:10:04.656 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 01:10:04 smithi116 podman[103360]: 2024-03-19 01:10:04.36373475 +0000 UTC m=+0.364511150 container start d66a0cad8ef2e659edd959d578248de9a5a60db789d638175c7062b76bc5a97c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-19T01:10:04.656 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 01:10:04 smithi116 ceph-mds[103380]: -- 172.21.15.116:0/1740848871 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5555a6ee7a40 con 0x5555a6eba800 2024-03-19T01:10:04.656 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 01:10:04 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-d[103376]: starting mds.d at 2024-03-19T01:10:04.656 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 01:10:04 smithi116 bash[103360]: d66a0cad8ef2e659edd959d578248de9a5a60db789d638175c7062b76bc5a97c 2024-03-19T01:10:04.656 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 01:10:04 smithi116 systemd[1]: Started Ceph mds.d for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:10:05.581 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:05.593 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:05.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:05.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:05.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:05.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:05.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:05.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:05 smithi156 ceph-mon[66799]: Deploying daemon mds.e on smithi156 2024-03-19T01:10:05.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:05 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.116:6836/3764874477,v1:172.21.15.116:6837/3764874477] up:boot 2024-03-19T01:10:05.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:05 smithi156 ceph-mon[66799]: fsmap 4 up:standby 2024-03-19T01:10:05.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T01:10:05.594 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:05 smithi156 ceph-mon[66799]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.697+0000 7f6d5db37640 1 -- 172.21.15.116:0/387136355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d58101840 msgr2=0x7f6d58101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.697+0000 7f6d5db37640 1 --2- 172.21.15.116:0/387136355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d58101840 0x7f6d58101c60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009900 tx=0x7f6d4802bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.697+0000 7f6d5db37640 1 -- 172.21.15.116:0/387136355 shutdown_connections 2024-03-19T01:10:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.697+0000 7f6d5db37640 1 --2- 172.21.15.116:0/387136355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d581033f0 0x7f6d58109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.697+0000 7f6d5db37640 1 --2- 172.21.15.116:0/387136355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d58102a30 0x7f6d58102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:05.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.697+0000 7f6d5db37640 1 --2- 172.21.15.116:0/387136355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d58101840 0x7f6d58101c60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.697+0000 7f6d5db37640 1 -- 172.21.15.116:0/387136355 >> 172.21.15.116:0/387136355 conn(0x7f6d580fcdd0 msgr2=0x7f6d580ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.697+0000 7f6d5db37640 1 -- 172.21.15.116:0/387136355 shutdown_connections 2024-03-19T01:10:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.697+0000 7f6d5db37640 1 -- 172.21.15.116:0/387136355 wait complete. 2024-03-19T01:10:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d5db37640 1 Processor -- start 2024-03-19T01:10:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d5db37640 1 -- start start 2024-03-19T01:10:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d5db37640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d58101840 0x7f6d58197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d5db37640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d58102a30 0x7f6d581980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d5db37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d581033f0 0x7f6d5819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d5db37640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5810d490 con 0x7f6d581033f0 2024-03-19T01:10:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d5db37640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5810d310 con 0x7f6d58102a30 2024-03-19T01:10:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d5db37640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6d5810d610 con 0x7f6d58101840 2024-03-19T01:10:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d57fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d581033f0 0x7f6d5819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:05.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d57fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d581033f0 0x7f6d5819f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41108/0 (socket says 172.21.15.116:41108) 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d57fff640 1 -- 172.21.15.116:0/340512319 learned_addr learned my addr 172.21.15.116:0/340512319 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d577fe640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d58101840 0x7f6d58197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d4ffff640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d58102a30 0x7f6d581980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d57fff640 1 -- 172.21.15.116:0/340512319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d58101840 msgr2=0x7f6d58197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d57fff640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d58101840 0x7f6d58197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d57fff640 1 -- 172.21.15.116:0/340512319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d58102a30 msgr2=0x7f6d581980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d57fff640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d58102a30 0x7f6d581980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.698+0000 7f6d57fff640 1 -- 172.21.15.116:0/340512319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d5819f790 con 0x7f6d581033f0 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.699+0000 7f6d57fff640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d581033f0 0x7f6d5819f120 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000efd0 tx=0x7f6d5000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.699+0000 7f6d557fa640 1 -- 172.21.15.116:0/340512319 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d50003b70 con 0x7f6d581033f0 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.699+0000 7f6d557fa640 1 -- 172.21.15.116:0/340512319 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d50003d10 con 0x7f6d581033f0 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.699+0000 7f6d5db37640 1 -- 172.21.15.116:0/340512319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d5819fa80 con 0x7f6d581033f0 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.699+0000 7f6d557fa640 1 -- 172.21.15.116:0/340512319 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d500109f0 con 0x7f6d581033f0 2024-03-19T01:10:05.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.699+0000 7f6d5db37640 1 -- 172.21.15.116:0/340512319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d5819ff80 con 0x7f6d581033f0 2024-03-19T01:10:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.700+0000 7f6d5db37640 1 -- 172.21.15.116:0/340512319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d58108be0 con 0x7f6d581033f0 2024-03-19T01:10:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.701+0000 7f6d557fa640 1 -- 172.21.15.116:0/340512319 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f6d50004020 con 0x7f6d581033f0 2024-03-19T01:10:05.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.701+0000 7f6d557fa640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6d440a2800 0x7f6d440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.701+0000 7f6d577fe640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6d440a2800 0x7f6d440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.702+0000 7f6d557fa640 1 -- 172.21.15.116:0/340512319 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6d500ceae0 con 0x7f6d581033f0 2024-03-19T01:10:05.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.702+0000 7f6d577fe640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6d440a2800 0x7f6d440a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6d48000c00 tx=0x7f6d48005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:05.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.706+0000 7f6d557fa640 1 -- 172.21.15.116:0/340512319 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6d5009c020 con 0x7f6d581033f0 2024-03-19T01:10:05.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:05.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:05.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:05.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:05.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:05.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:05.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:05 smithi162 ceph-mon[67178]: Deploying daemon mds.e on smithi156 2024-03-19T01:10:05.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:05 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.116:6836/3764874477,v1:172.21.15.116:6837/3764874477] up:boot 2024-03-19T01:10:05.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:05 smithi162 ceph-mon[67178]: fsmap 4 up:standby 2024-03-19T01:10:05.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T01:10:05.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:05 smithi162 ceph-mon[67178]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:05.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:05.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:05.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:05.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:05.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:05.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:05.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:05 smithi116 ceph-mon[62701]: Deploying daemon mds.e on smithi156 2024-03-19T01:10:05.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:05 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.116:6836/3764874477,v1:172.21.15.116:6837/3764874477] up:boot 2024-03-19T01:10:05.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:05 smithi116 ceph-mon[62701]: fsmap 4 up:standby 2024-03-19T01:10:05.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-19T01:10:05.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:05 smithi116 ceph-mon[62701]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:05.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.920+0000 7f6d5db37640 1 -- 172.21.15.116:0/340512319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d5802d1f0 con 0x7f6d581033f0 2024-03-19T01:10:05.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.920+0000 7f6d557fa640 1 -- 172.21.15.116:0/340512319 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f6d5009b920 con 0x7f6d581033f0 2024-03-19T01:10:05.922 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:05.922 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-19T01:10:05.923 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 5 2024-03-19T01:10:05.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.922+0000 7f6d5db37640 1 -- 172.21.15.116:0/340512319 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6d440a2800 msgr2=0x7f6d440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:05.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.922+0000 7f6d5db37640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6d440a2800 0x7f6d440a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6d48000c00 tx=0x7f6d48005fd0 comp rx=0 tx=0).stop 2024-03-19T01:10:05.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.922+0000 7f6d5db37640 1 -- 172.21.15.116:0/340512319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d581033f0 msgr2=0x7f6d5819f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.922+0000 7f6d5db37640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d581033f0 0x7f6d5819f120 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6d5000efd0 tx=0x7f6d5000c6a0 comp rx=0 tx=0).stop 2024-03-19T01:10:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.922+0000 7f6d5db37640 1 -- 172.21.15.116:0/340512319 shutdown_connections 2024-03-19T01:10:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.922+0000 7f6d5db37640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d581033f0 0x7f6d5819f120 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.923+0000 7f6d5db37640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6d440a2800 0x7f6d440a4cc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.923+0000 7f6d5db37640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6d58102a30 0x7f6d581980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.923+0000 7f6d5db37640 1 --2- 172.21.15.116:0/340512319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6d58101840 0x7f6d58197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.923+0000 7f6d5db37640 1 -- 172.21.15.116:0/340512319 >> 172.21.15.116:0/340512319 conn(0x7f6d580fcdd0 msgr2=0x7f6d58105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.923+0000 7f6d5db37640 1 -- 172.21.15.116:0/340512319 shutdown_connections 2024-03-19T01:10:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:05.923+0000 7f6d5db37640 1 -- 172.21.15.116:0/340512319 wait complete. 2024-03-19T01:10:06.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:06 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/340512319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:06.975 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:10:06 smithi156 systemd[1]: Starting Ceph mds.e for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:10:07.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:06 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/340512319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:07.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:06 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/340512319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:07.338 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:10:07 smithi156 podman[96675]: 2024-03-19 01:10:06.981098291 +0000 UTC m=+0.017878717 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:10:07.338 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:10:07 smithi156 podman[96675]: 2024-03-19 01:10:07.084081933 +0000 UTC m=+0.120862337 container create 09e1cff74bc46521e7c80bc70e2375d5487118c817a20a9129084e5241d8685b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:10:07.338 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:10:07 smithi156 podman[96675]: 2024-03-19 01:10:07.336871313 +0000 UTC m=+0.373651725 container init 09e1cff74bc46521e7c80bc70e2375d5487118c817a20a9129084e5241d8685b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5) 2024-03-19T01:10:07.676 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:10:07 smithi156 podman[96675]: 2024-03-19 01:10:07.339756457 +0000 UTC m=+0.376536863 container start 09e1cff74bc46521e7c80bc70e2375d5487118c817a20a9129084e5241d8685b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-e, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:10:07.676 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:10:07 smithi156 ceph-mds[96694]: -- 172.21.15.156:0/3143065094 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cc6d399a40 con 0x55cc6d36c800 2024-03-19T01:10:07.676 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:10:07 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-e[96690]: starting mds.e at 2024-03-19T01:10:07.676 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:10:07 smithi156 bash[96675]: 09e1cff74bc46521e7c80bc70e2375d5487118c817a20a9129084e5241d8685b 2024-03-19T01:10:07.676 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:10:07 smithi156 systemd[1]: Started Ceph mds.e for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:10:07.936 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:07.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:07 smithi116 ceph-mon[62701]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:07.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:07 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:07.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:07 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:07.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:07 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:07.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:07 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:07.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:07 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:07.965 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:07 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:07.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:07 smithi156 ceph-mon[66799]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:07.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:07 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:07.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:07 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:07.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:07 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:07.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:07 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:07.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:07 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:07.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:07 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:08.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:07 smithi162 ceph-mon[67178]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:08.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:07 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:08.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:07 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:08.050 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:07 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:08.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:07 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:08.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:07 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:08.051 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:07 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.049+0000 7f4dce60b640 1 -- 172.21.15.116:0/1634874455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc81033f0 msgr2=0x7f4dc8109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.049+0000 7f4dce60b640 1 --2- 172.21.15.116:0/1634874455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc81033f0 0x7f4dc8109c80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0009ca0 tx=0x7f4dc002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:10:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.049+0000 7f4dce60b640 1 -- 172.21.15.116:0/1634874455 shutdown_connections 2024-03-19T01:10:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.049+0000 7f4dce60b640 1 --2- 172.21.15.116:0/1634874455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc81033f0 0x7f4dc8109c80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.049+0000 7f4dce60b640 1 --2- 172.21.15.116:0/1634874455 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4dc8102a30 0x7f4dc8102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.049+0000 7f4dce60b640 1 --2- 172.21.15.116:0/1634874455 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4dc8101840 0x7f4dc8101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.049+0000 7f4dce60b640 1 -- 172.21.15.116:0/1634874455 >> 172.21.15.116:0/1634874455 conn(0x7f4dc80fcdd0 msgr2=0x7f4dc80ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.049+0000 7f4dce60b640 1 -- 172.21.15.116:0/1634874455 shutdown_connections 2024-03-19T01:10:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.049+0000 7f4dce60b640 1 -- 172.21.15.116:0/1634874455 wait complete. 2024-03-19T01:10:08.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.050+0000 7f4dce60b640 1 Processor -- start 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.050+0000 7f4dce60b640 1 -- start start 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.050+0000 7f4dce60b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4dc8101840 0x7f4dc80755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.050+0000 7f4dce60b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4dc8102a30 0x7f4dc8073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.050+0000 7f4dce60b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc81033f0 0x7f4dc8074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.050+0000 7f4dce60b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc810d4a0 con 0x7f4dc81033f0 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.050+0000 7f4dce60b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc810d320 con 0x7f4dc8102a30 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.050+0000 7f4dce60b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc810d620 con 0x7f4dc8101840 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.050+0000 7f4dccba8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc81033f0 0x7f4dc8074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.050+0000 7f4dccba8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc81033f0 0x7f4dc8074150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35200/0 (socket says 172.21.15.116:35200) 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.050+0000 7f4dccba8640 1 -- 172.21.15.116:0/15130761 learned_addr learned my addr 172.21.15.116:0/15130761 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.050+0000 7f4dc7fff640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4dc8101840 0x7f4dc80755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.050+0000 7f4dc77fe640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4dc8102a30 0x7f4dc8073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.051+0000 7f4dccba8640 1 -- 172.21.15.116:0/15130761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4dc8101840 msgr2=0x7f4dc80755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:08.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.051+0000 7f4dccba8640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4dc8101840 0x7f4dc80755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.051+0000 7f4dccba8640 1 -- 172.21.15.116:0/15130761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4dc8102a30 msgr2=0x7f4dc8073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.051+0000 7f4dccba8640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4dc8102a30 0x7f4dc8073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.051+0000 7f4dccba8640 1 -- 172.21.15.116:0/15130761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc80749e0 con 0x7f4dc81033f0 2024-03-19T01:10:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.051+0000 7f4dc77fe640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4dc8102a30 0x7f4dc8073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:10:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.051+0000 7f4dccba8640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc81033f0 0x7f4dc8074150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4dc002c700 tx=0x7f4dc0032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.051+0000 7f4dc57fa640 1 -- 172.21.15.116:0/15130761 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0039040 con 0x7f4dc81033f0 2024-03-19T01:10:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.051+0000 7f4dc57fa640 1 -- 172.21.15.116:0/15130761 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dc0044070 con 0x7f4dc81033f0 2024-03-19T01:10:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.051+0000 7f4dc57fa640 1 -- 172.21.15.116:0/15130761 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc00042a0 con 0x7f4dc81033f0 2024-03-19T01:10:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.051+0000 7f4dce60b640 1 -- 172.21.15.116:0/15130761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc81a1b70 con 0x7f4dc81033f0 2024-03-19T01:10:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.051+0000 7f4dce60b640 1 -- 172.21.15.116:0/15130761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc81a2070 con 0x7f4dc81033f0 2024-03-19T01:10:08.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.052+0000 7f4dce60b640 1 -- 172.21.15.116:0/15130761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dc8108b00 con 0x7f4dc81033f0 2024-03-19T01:10:08.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.053+0000 7f4dc57fa640 1 -- 172.21.15.116:0/15130761 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f4dc0003970 con 0x7f4dc81033f0 2024-03-19T01:10:08.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.054+0000 7f4dc57fa640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4db00a2800 0x7f4db00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:08.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.054+0000 7f4dc7fff640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4db00a2800 0x7f4db00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:08.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.054+0000 7f4dc57fa640 1 -- 172.21.15.116:0/15130761 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4dc00f0ab0 con 0x7f4dc81033f0 2024-03-19T01:10:08.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.054+0000 7f4dc7fff640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4db00a2800 0x7f4db00a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4db8005fd0 tx=0x7f4db800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:08.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.058+0000 7f4dc57fa640 1 -- 172.21.15.116:0/15130761 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4dc00be080 con 0x7f4dc81033f0 2024-03-19T01:10:08.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.271+0000 7f4dce60b640 1 -- 172.21.15.116:0/15130761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4dc802d1f0 con 0x7f4dc81033f0 2024-03-19T01:10:08.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.271+0000 7f4dc57fa640 1 -- 172.21.15.116:0/15130761 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7f4dc00bd980 con 0x7f4dc81033f0 2024-03-19T01:10:08.273 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:08.274 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-19T01:10:08.274 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 6 2024-03-19T01:10:08.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.273+0000 7f4dce60b640 1 -- 172.21.15.116:0/15130761 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4db00a2800 msgr2=0x7f4db00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.274+0000 7f4dce60b640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4db00a2800 0x7f4db00a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4db8005fd0 tx=0x7f4db800b410 comp rx=0 tx=0).stop 2024-03-19T01:10:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.274+0000 7f4dce60b640 1 -- 172.21.15.116:0/15130761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc81033f0 msgr2=0x7f4dc8074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.274+0000 7f4dce60b640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc81033f0 0x7f4dc8074150 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4dc002c700 tx=0x7f4dc0032f20 comp rx=0 tx=0).stop 2024-03-19T01:10:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.274+0000 7f4dce60b640 1 -- 172.21.15.116:0/15130761 shutdown_connections 2024-03-19T01:10:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.274+0000 7f4dce60b640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4dc81033f0 0x7f4dc8074150 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.274+0000 7f4dce60b640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4db00a2800 0x7f4db00a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.274+0000 7f4dce60b640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4dc8102a30 0x7f4dc8073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.274+0000 7f4dce60b640 1 --2- 172.21.15.116:0/15130761 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4dc8101840 0x7f4dc80755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.274+0000 7f4dce60b640 1 -- 172.21.15.116:0/15130761 >> 172.21.15.116:0/15130761 conn(0x7f4dc80fcdd0 msgr2=0x7f4dc8105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.274+0000 7f4dce60b640 1 -- 172.21.15.116:0/15130761 shutdown_connections 2024-03-19T01:10:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:08.274+0000 7f4dce60b640 1 -- 172.21.15.116:0/15130761 wait complete. 2024-03-19T01:10:08.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:08 smithi162 ceph-mon[67178]: Deploying daemon mds.f on smithi162 2024-03-19T01:10:08.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:08 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.156:6834/1738017780,v1:172.21.15.156:6835/1738017780] up:boot 2024-03-19T01:10:08.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:08 smithi162 ceph-mon[67178]: fsmap 5 up:standby 2024-03-19T01:10:08.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:08 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T01:10:08.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:08 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/15130761' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:08.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:08 smithi156 ceph-mon[66799]: Deploying daemon mds.f on smithi162 2024-03-19T01:10:08.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:08 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.156:6834/1738017780,v1:172.21.15.156:6835/1738017780] up:boot 2024-03-19T01:10:08.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:08 smithi156 ceph-mon[66799]: fsmap 5 up:standby 2024-03-19T01:10:08.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:08 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T01:10:08.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:08 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/15130761' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:09.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:08 smithi116 ceph-mon[62701]: Deploying daemon mds.f on smithi162 2024-03-19T01:10:09.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:08 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.156:6834/1738017780,v1:172.21.15.156:6835/1738017780] up:boot 2024-03-19T01:10:09.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:08 smithi116 ceph-mon[62701]: fsmap 5 up:standby 2024-03-19T01:10:09.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T01:10:09.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:08 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/15130761' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:09.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:09 smithi162 ceph-mon[67178]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:09.899 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:10:09 smithi162 systemd[1]: Starting Ceph mds.f for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:10:09.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:09 smithi156 ceph-mon[66799]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:10.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:09 smithi116 ceph-mon[62701]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:10.184 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:10:09 smithi162 podman[97660]: 2024-03-19 01:10:09.808255579 +0000 UTC m=+0.017783743 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:10:10.185 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:10:09 smithi162 podman[97660]: 2024-03-19 01:10:09.92388173 +0000 UTC m=+0.133409884 container create 53eec699f51fb00402eaaa0aa27a19be4f6666a3d7052c249263c07ca7e6dbea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:10:10.185 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:10:10 smithi162 podman[97660]: 2024-03-19 01:10:10.184720321 +0000 UTC m=+0.394248490 container init 53eec699f51fb00402eaaa0aa27a19be4f6666a3d7052c249263c07ca7e6dbea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-19T01:10:10.287 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:10.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.402+0000 7ffaddc61640 1 -- 172.21.15.116:0/392063164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffad8101350 msgr2=0x7ffad8108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.402+0000 7ffaddc61640 1 --2- 172.21.15.116:0/392063164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffad8101350 0x7ffad8108bf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffad0009ca0 tx=0x7ffad002c140 comp rx=0 tx=0).stop 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.402+0000 7ffaddc61640 1 -- 172.21.15.116:0/392063164 shutdown_connections 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.402+0000 7ffaddc61640 1 --2- 172.21.15.116:0/392063164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffad8101350 0x7ffad8108bf0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.402+0000 7ffaddc61640 1 --2- 172.21.15.116:0/392063164 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffad8100990 0x7ffad8100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.402+0000 7ffaddc61640 1 --2- 172.21.15.116:0/392063164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffad8101dd0 0x7ffad81021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.402+0000 7ffaddc61640 1 -- 172.21.15.116:0/392063164 >> 172.21.15.116:0/392063164 conn(0x7ffad80fb440 msgr2=0x7ffad80fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.402+0000 7ffaddc61640 1 -- 172.21.15.116:0/392063164 shutdown_connections 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.402+0000 7ffaddc61640 1 -- 172.21.15.116:0/392063164 wait complete. 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.403+0000 7ffaddc61640 1 Processor -- start 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.403+0000 7ffaddc61640 1 -- start start 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.403+0000 7ffaddc61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffad8100990 0x7ffad8197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.403+0000 7ffaddc61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffad8101350 0x7ffad81980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.403+0000 7ffaddc61640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffad8101dd0 0x7ffad819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:10.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.403+0000 7ffaddc61640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffad810c3f0 con 0x7ffad8101350 2024-03-19T01:10:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.403+0000 7ffaddc61640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ffad810c270 con 0x7ffad8101dd0 2024-03-19T01:10:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.403+0000 7ffaddc61640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ffad810c570 con 0x7ffad8100990 2024-03-19T01:10:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffad8101350 0x7ffad81980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffad8101350 0x7ffad81980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35226/0 (socket says 172.21.15.116:35226) 2024-03-19T01:10:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad6ffd640 1 -- 172.21.15.116:0/1190943447 learned_addr learned my addr 172.21.15.116:0/1190943447 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad77fe640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffad8100990 0x7ffad8197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad7fff640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffad8101dd0 0x7ffad819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad6ffd640 1 -- 172.21.15.116:0/1190943447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffad8100990 msgr2=0x7ffad8197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad6ffd640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffad8100990 0x7ffad8197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad6ffd640 1 -- 172.21.15.116:0/1190943447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffad8101dd0 msgr2=0x7ffad819f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad6ffd640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffad8101dd0 0x7ffad819f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad6ffd640 1 -- 172.21.15.116:0/1190943447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad819f7b0 con 0x7ffad8101350 2024-03-19T01:10:10.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad6ffd640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffad8101350 0x7ffad81980d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00e9c0 tx=0x7ffacc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:10.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad4ff9640 1 -- 172.21.15.116:0/1190943447 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc00cd80 con 0x7ffad8101350 2024-03-19T01:10:10.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffaddc61640 1 -- 172.21.15.116:0/1190943447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad819faa0 con 0x7ffad8101350 2024-03-19T01:10:10.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad4ff9640 1 -- 172.21.15.116:0/1190943447 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffacc0035f0 con 0x7ffad8101350 2024-03-19T01:10:10.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffad4ff9640 1 -- 172.21.15.116:0/1190943447 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffacc010690 con 0x7ffad8101350 2024-03-19T01:10:10.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.404+0000 7ffaddc61640 1 -- 172.21.15.116:0/1190943447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad819ffa0 con 0x7ffad8101350 2024-03-19T01:10:10.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.405+0000 7ffaddc61640 1 -- 172.21.15.116:0/1190943447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad81a0230 con 0x7ffad8101350 2024-03-19T01:10:10.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.406+0000 7ffad4ff9640 1 -- 172.21.15.116:0/1190943447 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7ffacc0037a0 con 0x7ffad8101350 2024-03-19T01:10:10.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.407+0000 7ffad4ff9640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ffac00a2800 0x7ffac00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:10.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.407+0000 7ffad77fe640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ffac00a2800 0x7ffac00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:10.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.407+0000 7ffad4ff9640 1 -- 172.21.15.116:0/1190943447 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ffacc0ce9c0 con 0x7ffad8101350 2024-03-19T01:10:10.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.407+0000 7ffad77fe640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ffac00a2800 0x7ffac00a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009bd0 tx=0x7ffac8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:10.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.412+0000 7ffad4ff9640 1 -- 172.21.15.116:0/1190943447 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffacc09bf00 con 0x7ffad8101350 2024-03-19T01:10:10.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.625+0000 7ffaddc61640 1 -- 172.21.15.116:0/1190943447 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffad81a0400 con 0x7ffad8101350 2024-03-19T01:10:10.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.626+0000 7ffad4ff9640 1 -- 172.21.15.116:0/1190943447 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7ffacc09b800 con 0x7ffad8101350 2024-03-19T01:10:10.628 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:10.628 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-19T01:10:10.628 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 6 2024-03-19T01:10:10.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.628+0000 7ffaddc61640 1 -- 172.21.15.116:0/1190943447 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ffac00a2800 msgr2=0x7ffac00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:10.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.628+0000 7ffaddc61640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ffac00a2800 0x7ffac00a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ffac8009bd0 tx=0x7ffac8005f90 comp rx=0 tx=0).stop 2024-03-19T01:10:10.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.628+0000 7ffaddc61640 1 -- 172.21.15.116:0/1190943447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffad8101350 msgr2=0x7ffad81980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:10.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.628+0000 7ffaddc61640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffad8101350 0x7ffad81980d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00e9c0 tx=0x7ffacc00ee80 comp rx=0 tx=0).stop 2024-03-19T01:10:10.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.628+0000 7ffaddc61640 1 -- 172.21.15.116:0/1190943447 shutdown_connections 2024-03-19T01:10:10.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.628+0000 7ffaddc61640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ffac00a2800 0x7ffac00a4cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:10.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.628+0000 7ffaddc61640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffad8101dd0 0x7ffad819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:10.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.628+0000 7ffaddc61640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffad8101350 0x7ffad81980d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:10.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.628+0000 7ffaddc61640 1 --2- 172.21.15.116:0/1190943447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ffad8100990 0x7ffad8197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:10.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.628+0000 7ffaddc61640 1 -- 172.21.15.116:0/1190943447 >> 172.21.15.116:0/1190943447 conn(0x7ffad80fb440 msgr2=0x7ffad80fcec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:10.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.628+0000 7ffaddc61640 1 -- 172.21.15.116:0/1190943447 shutdown_connections 2024-03-19T01:10:10.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:10.628+0000 7ffaddc61640 1 -- 172.21.15.116:0/1190943447 wait complete. 2024-03-19T01:10:10.649 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:10:10 smithi162 podman[97660]: 2024-03-19 01:10:10.187292836 +0000 UTC m=+0.396820994 container start 53eec699f51fb00402eaaa0aa27a19be4f6666a3d7052c249263c07ca7e6dbea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-19T01:10:10.649 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:10:10 smithi162 ceph-mds[97680]: -- 172.21.15.162:0/2826837885 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557b466f1a40 con 0x557b466c4c00 2024-03-19T01:10:10.649 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:10:10 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-f[97676]: starting mds.f at 2024-03-19T01:10:10.649 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:10:10 smithi162 bash[97660]: 53eec699f51fb00402eaaa0aa27a19be4f6666a3d7052c249263c07ca7e6dbea 2024-03-19T01:10:10.649 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:10:10 smithi162 systemd[1]: Started Ceph mds.f for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:10:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:11.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:11.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:11.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:11.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:11 smithi162 ceph-mon[67178]: Deploying daemon mds.c on smithi162 2024-03-19T01:10:11.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:11 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1190943447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:11.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:11 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.162:6834/1592939032,v1:172.21.15.162:6835/1592939032] up:boot 2024-03-19T01:10:11.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:11 smithi162 ceph-mon[67178]: fsmap 6 up:standby 2024-03-19T01:10:11.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T01:10:11.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:11 smithi162 ceph-mon[67178]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:11.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:11.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:11.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:11.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:11.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:11.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:11.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:11 smithi116 ceph-mon[62701]: Deploying daemon mds.c on smithi162 2024-03-19T01:10:11.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:11 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1190943447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:11.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:11 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.162:6834/1592939032,v1:172.21.15.162:6835/1592939032] up:boot 2024-03-19T01:10:11.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:11 smithi116 ceph-mon[62701]: fsmap 6 up:standby 2024-03-19T01:10:11.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T01:10:11.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:11 smithi116 ceph-mon[62701]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:11.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:11.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:11.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:11.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:11.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:11.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:11.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:11 smithi156 ceph-mon[66799]: Deploying daemon mds.c on smithi162 2024-03-19T01:10:11.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:11 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1190943447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:11.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:11 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.162:6834/1592939032,v1:172.21.15.162:6835/1592939032] up:boot 2024-03-19T01:10:11.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:11 smithi156 ceph-mon[66799]: fsmap 6 up:standby 2024-03-19T01:10:11.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T01:10:11.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:11 smithi156 ceph-mon[66799]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:12.642 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:12.649 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:10:12 smithi162 systemd[1]: Starting Ceph mds.c for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:10:12.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.754+0000 7fa815da9640 1 -- 172.21.15.116:0/3214980094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8101022f0 msgr2=0x7fa810108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:12.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.754+0000 7fa815da9640 1 --2- 172.21.15.116:0/3214980094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8101022f0 0x7fa810108b80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa808009ca0 tx=0x7fa80802c0f0 comp rx=0 tx=0).stop 2024-03-19T01:10:12.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.754+0000 7fa815da9640 1 -- 172.21.15.116:0/3214980094 shutdown_connections 2024-03-19T01:10:12.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.754+0000 7fa815da9640 1 --2- 172.21.15.116:0/3214980094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8101022f0 0x7fa810108b80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:12.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.754+0000 7fa815da9640 1 --2- 172.21.15.116:0/3214980094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa810101930 0x7fa810101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.754+0000 7fa815da9640 1 --2- 172.21.15.116:0/3214980094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa810100740 0x7fa810100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.754+0000 7fa815da9640 1 -- 172.21.15.116:0/3214980094 >> 172.21.15.116:0/3214980094 conn(0x7fa8100fbcb0 msgr2=0x7fa8100fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.754+0000 7fa815da9640 1 -- 172.21.15.116:0/3214980094 shutdown_connections 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.754+0000 7fa815da9640 1 -- 172.21.15.116:0/3214980094 wait complete. 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa815da9640 1 Processor -- start 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa815da9640 1 -- start start 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa815da9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa810100740 0x7fa810197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa815da9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa810101930 0x7fa810198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa815da9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8101022f0 0x7fa81019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa815da9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa81010c3c0 con 0x7fa8101022f0 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa815da9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa81010c240 con 0x7fa810101930 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa815da9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa81010c540 con 0x7fa810100740 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa80ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8101022f0 0x7fa81019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:12.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa80ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8101022f0 0x7fa81019f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35242/0 (socket says 172.21.15.116:35242) 2024-03-19T01:10:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa80f7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa810100740 0x7fa810197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa80ffff640 1 -- 172.21.15.116:0/475528032 learned_addr learned my addr 172.21.15.116:0/475528032 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa807fff640 1 --2- 172.21.15.116:0/475528032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa810101930 0x7fa810198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa80ffff640 1 -- 172.21.15.116:0/475528032 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa810100740 msgr2=0x7fa810197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa80ffff640 1 --2- 172.21.15.116:0/475528032 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa810100740 0x7fa810197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa80ffff640 1 -- 172.21.15.116:0/475528032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa810101930 msgr2=0x7fa810198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa80ffff640 1 --2- 172.21.15.116:0/475528032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa810101930 0x7fa810198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa80ffff640 1 -- 172.21.15.116:0/475528032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81019f740 con 0x7fa8101022f0 2024-03-19T01:10:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa80ffff640 1 --2- 172.21.15.116:0/475528032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8101022f0 0x7fa81019f0d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa808037040 tx=0x7fa808032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.755+0000 7fa80d7fa640 1 -- 172.21.15.116:0/475528032 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808041020 con 0x7fa8101022f0 2024-03-19T01:10:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.756+0000 7fa80d7fa640 1 -- 172.21.15.116:0/475528032 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa808034d90 con 0x7fa8101022f0 2024-03-19T01:10:12.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.756+0000 7fa815da9640 1 -- 172.21.15.116:0/475528032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa81019f9d0 con 0x7fa8101022f0 2024-03-19T01:10:12.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.756+0000 7fa80d7fa640 1 -- 172.21.15.116:0/475528032 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808039040 con 0x7fa8101022f0 2024-03-19T01:10:12.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.756+0000 7fa815da9640 1 -- 172.21.15.116:0/475528032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa81019fec0 con 0x7fa8101022f0 2024-03-19T01:10:12.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.757+0000 7fa815da9640 1 -- 172.21.15.116:0/475528032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8101a00f0 con 0x7fa8101022f0 2024-03-19T01:10:12.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.758+0000 7fa80d7fa640 1 -- 172.21.15.116:0/475528032 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fa808044020 con 0x7fa8101022f0 2024-03-19T01:10:12.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.758+0000 7fa80d7fa640 1 --2- 172.21.15.116:0/475528032 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa7fc0a2800 0x7fa7fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:12.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.759+0000 7fa80f7fe640 1 --2- 172.21.15.116:0/475528032 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa7fc0a2800 0x7fa7fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:12.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.759+0000 7fa80d7fa640 1 -- 172.21.15.116:0/475528032 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa8080f04a0 con 0x7fa8101022f0 2024-03-19T01:10:12.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.759+0000 7fa80f7fe640 1 --2- 172.21.15.116:0/475528032 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa7fc0a2800 0x7fa7fc0a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa810101790 tx=0x7fa800005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:12.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.763+0000 7fa80d7fa640 1 -- 172.21.15.116:0/475528032 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa80804b020 con 0x7fa8101022f0 2024-03-19T01:10:12.931 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:10:12 smithi162 podman[97967]: 2024-03-19 01:10:12.556567383 +0000 UTC m=+0.017876582 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:10:12.931 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:10:12 smithi162 podman[97967]: 2024-03-19 01:10:12.670040523 +0000 UTC m=+0.131349738 container create 000c09a15f6a4b40c28871e894612ca1e673c2fdc4810ee6129f207f6242f87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307) 2024-03-19T01:10:12.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.973+0000 7fa815da9640 1 -- 172.21.15.116:0/475528032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8101a0480 con 0x7fa8101022f0 2024-03-19T01:10:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.974+0000 7fa80d7fa640 1 -- 172.21.15.116:0/475528032 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7fa8080bd9e0 con 0x7fa8101022f0 2024-03-19T01:10:12.976 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:12.976 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-19T01:10:12.976 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 7 2024-03-19T01:10:12.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.976+0000 7fa815da9640 1 -- 172.21.15.116:0/475528032 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa7fc0a2800 msgr2=0x7fa7fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:12.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.976+0000 7fa815da9640 1 --2- 172.21.15.116:0/475528032 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa7fc0a2800 0x7fa7fc0a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa810101790 tx=0x7fa800005f90 comp rx=0 tx=0).stop 2024-03-19T01:10:12.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.976+0000 7fa815da9640 1 -- 172.21.15.116:0/475528032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8101022f0 msgr2=0x7fa81019f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:12.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.976+0000 7fa815da9640 1 --2- 172.21.15.116:0/475528032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8101022f0 0x7fa81019f0d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa808037040 tx=0x7fa808032f20 comp rx=0 tx=0).stop 2024-03-19T01:10:12.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.976+0000 7fa815da9640 1 -- 172.21.15.116:0/475528032 shutdown_connections 2024-03-19T01:10:12.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.976+0000 7fa815da9640 1 --2- 172.21.15.116:0/475528032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8101022f0 0x7fa81019f0d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:12.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.976+0000 7fa815da9640 1 --2- 172.21.15.116:0/475528032 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa7fc0a2800 0x7fa7fc0a4cc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:12.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.976+0000 7fa815da9640 1 --2- 172.21.15.116:0/475528032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa810101930 0x7fa810198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.976+0000 7fa815da9640 1 --2- 172.21.15.116:0/475528032 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa810100740 0x7fa810197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.976+0000 7fa815da9640 1 -- 172.21.15.116:0/475528032 >> 172.21.15.116:0/475528032 conn(0x7fa8100fbcb0 msgr2=0x7fa8100fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.976+0000 7fa815da9640 1 -- 172.21.15.116:0/475528032 shutdown_connections 2024-03-19T01:10:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:12.976+0000 7fa815da9640 1 -- 172.21.15.116:0/475528032 wait complete. 2024-03-19T01:10:13.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:13 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/475528032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:13.399 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:10:12 smithi162 podman[97967]: 2024-03-19 01:10:12.931371437 +0000 UTC m=+0.392680646 container init 000c09a15f6a4b40c28871e894612ca1e673c2fdc4810ee6129f207f6242f87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-c, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-19T01:10:13.399 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:10:12 smithi162 podman[97967]: 2024-03-19 01:10:12.934035267 +0000 UTC m=+0.395344473 container start 000c09a15f6a4b40c28871e894612ca1e673c2fdc4810ee6129f207f6242f87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-19T01:10:13.399 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:10:12 smithi162 ceph-mds[97986]: -- 172.21.15.162:0/1093150892 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564e31427a40 con 0x564e313fa800 2024-03-19T01:10:13.399 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:10:12 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-c[97982]: starting mds.c at 2024-03-19T01:10:13.399 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:10:13 smithi162 bash[97967]: 000c09a15f6a4b40c28871e894612ca1e673c2fdc4810ee6129f207f6242f87b 2024-03-19T01:10:13.400 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:10:13 smithi162 systemd[1]: Started Ceph mds.c for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:10:13.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:13 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/475528032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:13.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:13 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/475528032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:14 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] up:boot 2024-03-19T01:10:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:14 smithi162 ceph-mon[67178]: fsmap 7 up:standby 2024-03-19T01:10:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T01:10:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:14 smithi162 ceph-mon[67178]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:14.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:14.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:14.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:14.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:14.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:14.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:14 smithi162 ceph-mon[67178]: Deploying daemon mds.h on smithi156 2024-03-19T01:10:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:14 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] up:boot 2024-03-19T01:10:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:14 smithi116 ceph-mon[62701]: fsmap 7 up:standby 2024-03-19T01:10:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T01:10:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:14 smithi116 ceph-mon[62701]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:14.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:14.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:14.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:14 smithi116 ceph-mon[62701]: Deploying daemon mds.h on smithi156 2024-03-19T01:10:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:14 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] up:boot 2024-03-19T01:10:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:14 smithi156 ceph-mon[66799]: fsmap 7 up:standby 2024-03-19T01:10:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T01:10:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:14 smithi156 ceph-mon[66799]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:14.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:14.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:14.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:14.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:14.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:14.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:14 smithi156 ceph-mon[66799]: Deploying daemon mds.h on smithi156 2024-03-19T01:10:14.990 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:15.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.101+0000 7f53350ac640 1 -- 172.21.15.116:0/4193182576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5330104c70 msgr2=0x7f5330107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.101+0000 7f53350ac640 1 --2- 172.21.15.116:0/4193182576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5330104c70 0x7f5330107060 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5324009900 tx=0x7f532402bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.101+0000 7f53350ac640 1 -- 172.21.15.116:0/4193182576 shutdown_connections 2024-03-19T01:10:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.101+0000 7f53350ac640 1 --2- 172.21.15.116:0/4193182576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53301075a0 0x7f53301099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.101+0000 7f53350ac640 1 --2- 172.21.15.116:0/4193182576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5330104c70 0x7f5330107060 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.101+0000 7f53350ac640 1 --2- 172.21.15.116:0/4193182576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5330102340 0x7f5330104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.101+0000 7f53350ac640 1 -- 172.21.15.116:0/4193182576 >> 172.21.15.116:0/4193182576 conn(0x7f53300fbd50 msgr2=0x7f53300fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.101+0000 7f53350ac640 1 -- 172.21.15.116:0/4193182576 shutdown_connections 2024-03-19T01:10:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.101+0000 7f53350ac640 1 -- 172.21.15.116:0/4193182576 wait complete. 2024-03-19T01:10:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.102+0000 7f53350ac640 1 Processor -- start 2024-03-19T01:10:15.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.102+0000 7f53350ac640 1 -- start start 2024-03-19T01:10:15.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.102+0000 7f53350ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5330102340 0x7f5330197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:15.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.102+0000 7f53350ac640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5330104c70 0x7f53301980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:15.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.102+0000 7f53350ac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53301075a0 0x7f533019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:15.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.102+0000 7f53350ac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f533010d4c0 con 0x7f5330102340 2024-03-19T01:10:15.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.102+0000 7f53350ac640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f533010d340 con 0x7f5330104c70 2024-03-19T01:10:15.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.102+0000 7f53350ac640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f533010d640 con 0x7f53301075a0 2024-03-19T01:10:15.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.102+0000 7f532ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5330102340 0x7f5330197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:15.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.102+0000 7f532ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5330102340 0x7f5330197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35248/0 (socket says 172.21.15.116:35248) 2024-03-19T01:10:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.102+0000 7f532ed9d640 1 -- 172.21.15.116:0/630964071 learned_addr learned my addr 172.21.15.116:0/630964071 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.102+0000 7f532e59c640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5330104c70 0x7f53301980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.102+0000 7f532f59e640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53301075a0 0x7f533019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.103+0000 7f532ed9d640 1 -- 172.21.15.116:0/630964071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53301075a0 msgr2=0x7f533019f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.103+0000 7f532ed9d640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53301075a0 0x7f533019f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.103+0000 7f532ed9d640 1 -- 172.21.15.116:0/630964071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5330104c70 msgr2=0x7f53301980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.103+0000 7f532ed9d640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5330104c70 0x7f53301980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.103+0000 7f532ed9d640 1 -- 172.21.15.116:0/630964071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533019f790 con 0x7f5330102340 2024-03-19T01:10:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.103+0000 7f532e59c640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5330104c70 0x7f53301980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:10:15.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.103+0000 7f532ed9d640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5330102340 0x7f5330197b70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f532000e9f0 tx=0x7f532000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:15.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.105+0000 7f5317fff640 1 -- 172.21.15.116:0/630964071 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532000cd80 con 0x7f5330102340 2024-03-19T01:10:15.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.105+0000 7f53350ac640 1 -- 172.21.15.116:0/630964071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f533019fa80 con 0x7f5330102340 2024-03-19T01:10:15.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.105+0000 7f5317fff640 1 -- 172.21.15.116:0/630964071 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5320010910 con 0x7f5330102340 2024-03-19T01:10:15.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.105+0000 7f5317fff640 1 -- 172.21.15.116:0/630964071 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5320018990 con 0x7f5330102340 2024-03-19T01:10:15.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.105+0000 7f53350ac640 1 -- 172.21.15.116:0/630964071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f533019ffa0 con 0x7f5330102340 2024-03-19T01:10:15.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.106+0000 7f53350ac640 1 -- 172.21.15.116:0/630964071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5330068a60 con 0x7f5330102340 2024-03-19T01:10:15.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.107+0000 7f5317fff640 1 -- 172.21.15.116:0/630964071 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f5320010ac0 con 0x7f5330102340 2024-03-19T01:10:15.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.107+0000 7f5317fff640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53180a2800 0x7f53180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:15.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.108+0000 7f532e59c640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53180a2800 0x7f53180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:15.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.108+0000 7f5317fff640 1 -- 172.21.15.116:0/630964071 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f53200ce910 con 0x7f5330102340 2024-03-19T01:10:15.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.108+0000 7f532e59c640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53180a2800 0x7f53180a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5324000c00 tx=0x7f5324005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:15.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.112+0000 7f5317fff640 1 -- 172.21.15.116:0/630964071 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f532009b800 con 0x7f5330102340 2024-03-19T01:10:15.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.326+0000 7f53350ac640 1 -- 172.21.15.116:0/630964071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f533002d2e0 con 0x7f5330102340 2024-03-19T01:10:15.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.326+0000 7f5317fff640 1 -- 172.21.15.116:0/630964071 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f533002d2e0 con 0x7f5330102340 2024-03-19T01:10:15.328 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:15.328 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-19T01:10:15.328 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 8 2024-03-19T01:10:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.328+0000 7f53350ac640 1 -- 172.21.15.116:0/630964071 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53180a2800 msgr2=0x7f53180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.328+0000 7f53350ac640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53180a2800 0x7f53180a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5324000c00 tx=0x7f5324005fd0 comp rx=0 tx=0).stop 2024-03-19T01:10:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.328+0000 7f53350ac640 1 -- 172.21.15.116:0/630964071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5330102340 msgr2=0x7f5330197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.328+0000 7f53350ac640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5330102340 0x7f5330197b70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f532000e9f0 tx=0x7f532000eeb0 comp rx=0 tx=0).stop 2024-03-19T01:10:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.328+0000 7f53350ac640 1 -- 172.21.15.116:0/630964071 shutdown_connections 2024-03-19T01:10:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.328+0000 7f53350ac640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53301075a0 0x7f533019f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.328+0000 7f53350ac640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53180a2800 0x7f53180a4cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.328+0000 7f53350ac640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5330104c70 0x7f53301980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.328+0000 7f53350ac640 1 --2- 172.21.15.116:0/630964071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5330102340 0x7f5330197b70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.328+0000 7f53350ac640 1 -- 172.21.15.116:0/630964071 >> 172.21.15.116:0/630964071 conn(0x7f53300fbd50 msgr2=0x7f5330102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.328+0000 7f53350ac640 1 -- 172.21.15.116:0/630964071 shutdown_connections 2024-03-19T01:10:15.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:15.328+0000 7f53350ac640 1 -- 172.21.15.116:0/630964071 wait complete. 2024-03-19T01:10:15.474 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:10:15 smithi156 systemd[1]: Starting Ceph mds.h for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:10:15.834 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:10:15 smithi156 podman[96996]: 2024-03-19 01:10:15.477506172 +0000 UTC m=+0.018178115 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:10:15.835 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:10:15 smithi156 podman[96996]: 2024-03-19 01:10:15.581766678 +0000 UTC m=+0.122438613 container create 5c7c19489976d25ee75480dd3049ebeaa730b27905973591990e3defc5c39144 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:10:15.835 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:10:15 smithi156 podman[96996]: 2024-03-19 01:10:15.834376415 +0000 UTC m=+0.375048370 container init 5c7c19489976d25ee75480dd3049ebeaa730b27905973591990e3defc5c39144 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-h, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-19T01:10:16.114 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:10:15 smithi156 podman[96996]: 2024-03-19 01:10:15.837388269 +0000 UTC m=+0.378060203 container start 5c7c19489976d25ee75480dd3049ebeaa730b27905973591990e3defc5c39144 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-03-19T01:10:16.114 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:10:15 smithi156 ceph-mds[97015]: -- 172.21.15.156:0/259903211 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5566807bda40 con 0x556680791000 2024-03-19T01:10:16.114 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:10:15 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-h[97011]: starting mds.h at 2024-03-19T01:10:16.114 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:10:15 smithi156 bash[96996]: 5c7c19489976d25ee75480dd3049ebeaa730b27905973591990e3defc5c39144 2024-03-19T01:10:16.114 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:10:15 smithi156 systemd[1]: Started Ceph mds.h for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:10:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:16 smithi162 ceph-mon[67178]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:16 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/630964071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:16 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:16 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:16 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:16 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:16 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:16 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:16 smithi116 ceph-mon[62701]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:16 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/630964071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:16 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:16 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:16.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:16 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:16.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:16 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:16.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:16 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:16.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:16 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:16 smithi156 ceph-mon[66799]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:16 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/630964071' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:16 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:16 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:16 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:16 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:16 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:16 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:17.342 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:17.372 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:17 smithi116 ceph-mon[62701]: Deploying daemon mds.i on smithi162 2024-03-19T01:10:17.372 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:17 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] up:boot 2024-03-19T01:10:17.372 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:17 smithi116 ceph-mon[62701]: fsmap 8 up:standby 2024-03-19T01:10:17.372 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T01:10:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:17 smithi162 ceph-mon[67178]: Deploying daemon mds.i on smithi162 2024-03-19T01:10:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:17 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] up:boot 2024-03-19T01:10:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:17 smithi162 ceph-mon[67178]: fsmap 8 up:standby 2024-03-19T01:10:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T01:10:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.454+0000 7fdf8e885640 1 -- 172.21.15.116:0/1860293210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf88100740 msgr2=0x7fdf88100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.454+0000 7fdf8e885640 1 --2- 172.21.15.116:0/1860293210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf88100740 0x7fdf88100b60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009900 tx=0x7fdf7802bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.454+0000 7fdf8e885640 1 -- 172.21.15.116:0/1860293210 shutdown_connections 2024-03-19T01:10:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.454+0000 7fdf8e885640 1 --2- 172.21.15.116:0/1860293210 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf881022f0 0x7fdf88108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.454+0000 7fdf8e885640 1 --2- 172.21.15.116:0/1860293210 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf88101930 0x7fdf88101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.454+0000 7fdf8e885640 1 --2- 172.21.15.116:0/1860293210 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf88100740 0x7fdf88100b60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.454+0000 7fdf8e885640 1 -- 172.21.15.116:0/1860293210 >> 172.21.15.116:0/1860293210 conn(0x7fdf880fbcb0 msgr2=0x7fdf880fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.454+0000 7fdf8e885640 1 -- 172.21.15.116:0/1860293210 shutdown_connections 2024-03-19T01:10:17.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.454+0000 7fdf8e885640 1 -- 172.21.15.116:0/1860293210 wait complete. 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.454+0000 7fdf8e885640 1 Processor -- start 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf8e885640 1 -- start start 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf8e885640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf88100740 0x7fdf88197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf8e885640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf88101930 0x7fdf881980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf8e885640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf881022f0 0x7fdf8819f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf8e885640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8810d4a0 con 0x7fdf88100740 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf8e885640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8810d320 con 0x7fdf881022f0 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf8e885640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8810d620 con 0x7fdf88101930 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf88100740 0x7fdf88197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf87fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf88100740 0x7fdf88197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56500/0 (socket says 172.21.15.116:56500) 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf87fff640 1 -- 172.21.15.116:0/1639022640 learned_addr learned my addr 172.21.15.116:0/1639022640 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf877fe640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf88101930 0x7fdf881980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf8ce22640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf881022f0 0x7fdf8819f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:17.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf87fff640 1 -- 172.21.15.116:0/1639022640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf88101930 msgr2=0x7fdf881980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf87fff640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf88101930 0x7fdf881980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf87fff640 1 -- 172.21.15.116:0/1639022640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf881022f0 msgr2=0x7fdf8819f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf87fff640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf881022f0 0x7fdf8819f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.455+0000 7fdf87fff640 1 -- 172.21.15.116:0/1639022640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8819f7a0 con 0x7fdf88100740 2024-03-19T01:10:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.456+0000 7fdf87fff640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf88100740 0x7fdf88197b80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdf78005ab0 tx=0x7fdf78005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.456+0000 7fdf857fa640 1 -- 172.21.15.116:0/1639022640 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78041070 con 0x7fdf88100740 2024-03-19T01:10:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.456+0000 7fdf857fa640 1 -- 172.21.15.116:0/1639022640 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf780054e0 con 0x7fdf88100740 2024-03-19T01:10:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.456+0000 7fdf8e885640 1 -- 172.21.15.116:0/1639022640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf8819fa30 con 0x7fdf88100740 2024-03-19T01:10:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.456+0000 7fdf857fa640 1 -- 172.21.15.116:0/1639022640 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78034440 con 0x7fdf88100740 2024-03-19T01:10:17.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.456+0000 7fdf8e885640 1 -- 172.21.15.116:0/1639022640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf8819ff20 con 0x7fdf88100740 2024-03-19T01:10:17.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.457+0000 7fdf8e885640 1 -- 172.21.15.116:0/1639022640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf88107ae0 con 0x7fdf88100740 2024-03-19T01:10:17.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.458+0000 7fdf857fa640 1 -- 172.21.15.116:0/1639022640 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fdf78044070 con 0x7fdf88100740 2024-03-19T01:10:17.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.458+0000 7fdf857fa640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdf700a2800 0x7fdf700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:17.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.458+0000 7fdf877fe640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdf700a2800 0x7fdf700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:17.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.458+0000 7fdf857fa640 1 -- 172.21.15.116:0/1639022640 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fdf780f02e0 con 0x7fdf88100740 2024-03-19T01:10:17.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.459+0000 7fdf877fe640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdf700a2800 0x7fdf700a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdf88199090 tx=0x7fdf7c00a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:17.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.463+0000 7fdf857fa640 1 -- 172.21.15.116:0/1639022640 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdf780bd820 con 0x7fdf88100740 2024-03-19T01:10:17.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:17 smithi156 ceph-mon[66799]: Deploying daemon mds.i on smithi162 2024-03-19T01:10:17.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:17 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] up:boot 2024-03-19T01:10:17.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:17 smithi156 ceph-mon[66799]: fsmap 8 up:standby 2024-03-19T01:10:17.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:17 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T01:10:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.673+0000 7fdf8e885640 1 -- 172.21.15.116:0/1639022640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf8802d5d0 con 0x7fdf88100740 2024-03-19T01:10:17.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.674+0000 7fdf857fa640 1 -- 172.21.15.116:0/1639022640 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7fdf8802d5d0 con 0x7fdf88100740 2024-03-19T01:10:17.675 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:17.675 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-19T01:10:17.675 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 9 2024-03-19T01:10:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.676+0000 7fdf8e885640 1 -- 172.21.15.116:0/1639022640 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdf700a2800 msgr2=0x7fdf700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.676+0000 7fdf8e885640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdf700a2800 0x7fdf700a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdf88199090 tx=0x7fdf7c00a4c0 comp rx=0 tx=0).stop 2024-03-19T01:10:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.676+0000 7fdf8e885640 1 -- 172.21.15.116:0/1639022640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf88100740 msgr2=0x7fdf88197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.676+0000 7fdf8e885640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf88100740 0x7fdf88197b80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdf78005ab0 tx=0x7fdf78005dc0 comp rx=0 tx=0).stop 2024-03-19T01:10:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.676+0000 7fdf8e885640 1 -- 172.21.15.116:0/1639022640 shutdown_connections 2024-03-19T01:10:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.676+0000 7fdf8e885640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdf700a2800 0x7fdf700a4cc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.676+0000 7fdf8e885640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf881022f0 0x7fdf8819f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.676+0000 7fdf8e885640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf88101930 0x7fdf881980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.676+0000 7fdf8e885640 1 --2- 172.21.15.116:0/1639022640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf88100740 0x7fdf88197b80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.676+0000 7fdf8e885640 1 -- 172.21.15.116:0/1639022640 >> 172.21.15.116:0/1639022640 conn(0x7fdf880fbcb0 msgr2=0x7fdf88104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.676+0000 7fdf8e885640 1 -- 172.21.15.116:0/1639022640 shutdown_connections 2024-03-19T01:10:17.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:17.676+0000 7fdf8e885640 1 -- 172.21.15.116:0/1639022640 wait complete. 2024-03-19T01:10:18.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:18 smithi162 ceph-mon[67178]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:18.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:18 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1639022640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:18.400 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:10:18 smithi162 systemd[1]: Starting Ceph mds.i for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:10:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:18 smithi116 ceph-mon[62701]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:18 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1639022640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:18.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:18 smithi156 ceph-mon[66799]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:18 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1639022640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:18.703 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:10:18 smithi162 podman[98283]: 2024-03-19 01:10:18.339225565 +0000 UTC m=+0.017520291 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:10:18.703 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:10:18 smithi162 podman[98283]: 2024-03-19 01:10:18.441322093 +0000 UTC m=+0.119616824 container create d91e846e05c30fecd815f527e477acba3aa2037e5569e7d38879bd0437729db6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-i, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:10:18.703 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:10:18 smithi162 podman[98283]: 2024-03-19 01:10:18.67771727 +0000 UTC m=+0.356011991 container init d91e846e05c30fecd815f527e477acba3aa2037e5569e7d38879bd0437729db6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-i, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-19T01:10:18.704 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:10:18 smithi162 podman[98283]: 2024-03-19 01:10:18.680503568 +0000 UTC m=+0.358798288 container start d91e846e05c30fecd815f527e477acba3aa2037e5569e7d38879bd0437729db6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-i, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:10:18.987 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:10:18 smithi162 ceph-mds[98302]: -- 172.21.15.162:0/3139462998 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555b66a89a40 con 0x555b66a5d000 2024-03-19T01:10:18.987 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:10:18 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-i[98298]: starting mds.i at 2024-03-19T01:10:18.987 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:10:18 smithi162 bash[98283]: d91e846e05c30fecd815f527e477acba3aa2037e5569e7d38879bd0437729db6 2024-03-19T01:10:18.988 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:10:18 smithi162 systemd[1]: Started Ceph mds.i for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:10:19.689 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:19.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.796+0000 7f0113073640 1 -- 172.21.15.116:0/4013129192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c101930 msgr2=0x7f010c101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:19.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.796+0000 7f0113073640 1 --2- 172.21.15.116:0/4013129192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c101930 0x7f010c101db0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f00f8009900 tx=0x7f00f802bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:19.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.796+0000 7f0113073640 1 -- 172.21.15.116:0/4013129192 shutdown_connections 2024-03-19T01:10:19.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.796+0000 7f0113073640 1 --2- 172.21.15.116:0/4013129192 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f010c1022f0 0x7f010c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:19.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.796+0000 7f0113073640 1 --2- 172.21.15.116:0/4013129192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c101930 0x7f010c101db0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:19.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.796+0000 7f0113073640 1 --2- 172.21.15.116:0/4013129192 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f010c100740 0x7f010c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.796+0000 7f0113073640 1 -- 172.21.15.116:0/4013129192 >> 172.21.15.116:0/4013129192 conn(0x7f010c0fbcb0 msgr2=0x7f010c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.796+0000 7f0113073640 1 -- 172.21.15.116:0/4013129192 shutdown_connections 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.796+0000 7f0113073640 1 -- 172.21.15.116:0/4013129192 wait complete. 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0113073640 1 Processor -- start 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0113073640 1 -- start start 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0113073640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f010c100740 0x7f010c0694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0113073640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f010c101930 0x7f010c19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0113073640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c1022f0 0x7f010c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0113073640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f010c10c4b0 con 0x7f010c1022f0 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0113073640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f010c10c330 con 0x7f010c101930 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0113073640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f010c10c630 con 0x7f010c100740 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0111610640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c1022f0 0x7f010c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0111610640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c1022f0 0x7f010c19efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56506/0 (socket says 172.21.15.116:56506) 2024-03-19T01:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0111610640 1 -- 172.21.15.116:0/1760495975 learned_addr learned my addr 172.21.15.116:0/1760495975 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0110e0f640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f010c100740 0x7f010c0694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f010bfff640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f010c101930 0x7f010c19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0111610640 1 -- 172.21.15.116:0/1760495975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f010c100740 msgr2=0x7f010c0694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0111610640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f010c100740 0x7f010c0694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0111610640 1 -- 172.21.15.116:0/1760495975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f010c101930 msgr2=0x7f010c19c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0111610640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f010c101930 0x7f010c19c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0111610640 1 -- 172.21.15.116:0/1760495975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f010c19f8d0 con 0x7f010c1022f0 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f010bfff640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f010c101930 0x7f010c19c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0111610640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c1022f0 0x7f010c19efc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f010400b970 tx=0x7f010400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0109ffb640 1 -- 172.21.15.116:0/1760495975 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0104019070 con 0x7f010c1022f0 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.797+0000 7f0109ffb640 1 -- 172.21.15.116:0/1760495975 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0104014440 con 0x7f010c1022f0 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.798+0000 7f0113073640 1 -- 172.21.15.116:0/1760495975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f010c19fb60 con 0x7f010c1022f0 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.798+0000 7f0109ffb640 1 -- 172.21.15.116:0/1760495975 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01040075d0 con 0x7f010c1022f0 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.798+0000 7f0113073640 1 -- 172.21.15.116:0/1760495975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f010c1a00b0 con 0x7f010c1022f0 2024-03-19T01:10:19.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.799+0000 7f0113073640 1 -- 172.21.15.116:0/1760495975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f010c107a60 con 0x7f010c1022f0 2024-03-19T01:10:19.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.799+0000 7f0109ffb640 1 -- 172.21.15.116:0/1760495975 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f010401c070 con 0x7f010c1022f0 2024-03-19T01:10:19.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.800+0000 7f0109ffb640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f00fc0a2800 0x7f00fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:19.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.800+0000 7f0110e0f640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f00fc0a2800 0x7f00fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.800+0000 7f0109ffb640 1 -- 172.21.15.116:0/1760495975 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f01040cdb70 con 0x7f010c1022f0 2024-03-19T01:10:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.800+0000 7f0110e0f640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f00fc0a2800 0x7f00fc0a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f010c101790 tx=0x7f0100009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:19.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:19.804+0000 7f0109ffb640 1 -- 172.21.15.116:0/1760495975 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f010409b0b0 con 0x7f010c1022f0 2024-03-19T01:10:20.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.022+0000 7f0113073640 1 -- 172.21.15.116:0/1760495975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f010c02d3a0 con 0x7f010c1022f0 2024-03-19T01:10:20.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.023+0000 7f0109ffb640 1 -- 172.21.15.116:0/1760495975 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7f010409a9b0 con 0x7f010c1022f0 2024-03-19T01:10:20.024 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:20.024 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/441486392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":441486392},{"type":"v1","addr":"172.21.15.162:6839","nonce":441486392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-19T01:10:20.024 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 10 2024-03-19T01:10:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.025+0000 7f0113073640 1 -- 172.21.15.116:0/1760495975 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f00fc0a2800 msgr2=0x7f00fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.025+0000 7f0113073640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f00fc0a2800 0x7f00fc0a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f010c101790 tx=0x7f0100009110 comp rx=0 tx=0).stop 2024-03-19T01:10:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.025+0000 7f0113073640 1 -- 172.21.15.116:0/1760495975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c1022f0 msgr2=0x7f010c19efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.025+0000 7f0113073640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c1022f0 0x7f010c19efc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f010400b970 tx=0x7f010400bd70 comp rx=0 tx=0).stop 2024-03-19T01:10:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.025+0000 7f0113073640 1 -- 172.21.15.116:0/1760495975 shutdown_connections 2024-03-19T01:10:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.025+0000 7f0113073640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f010c1022f0 0x7f010c19efc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.025+0000 7f0113073640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f00fc0a2800 0x7f00fc0a4cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.025+0000 7f0113073640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f010c101930 0x7f010c19c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:20.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.025+0000 7f0113073640 1 --2- 172.21.15.116:0/1760495975 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f010c100740 0x7f010c0694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.025+0000 7f0113073640 1 -- 172.21.15.116:0/1760495975 >> 172.21.15.116:0/1760495975 conn(0x7f010c0fbcb0 msgr2=0x7f010c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.025+0000 7f0113073640 1 -- 172.21.15.116:0/1760495975 shutdown_connections 2024-03-19T01:10:20.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:20.025+0000 7f0113073640 1 -- 172.21.15.116:0/1760495975 wait complete. 2024-03-19T01:10:20.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:20.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:20.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:20.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:20.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:20.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:20.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:19 smithi162 ceph-mon[67178]: Deploying daemon mds.g on smithi116 2024-03-19T01:10:20.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:19 smithi162 ceph-mon[67178]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:20.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:19 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] up:boot 2024-03-19T01:10:20.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:19 smithi162 ceph-mon[67178]: fsmap 9 up:standby 2024-03-19T01:10:20.150 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T01:10:20.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:20.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:20.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:20.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:20.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:20.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:20.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:19 smithi116 ceph-mon[62701]: Deploying daemon mds.g on smithi116 2024-03-19T01:10:20.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:19 smithi116 ceph-mon[62701]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:20.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:19 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] up:boot 2024-03-19T01:10:20.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:19 smithi116 ceph-mon[62701]: fsmap 9 up:standby 2024-03-19T01:10:20.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T01:10:20.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:20.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:20.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:20.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:20.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:20.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:20.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:19 smithi156 ceph-mon[66799]: Deploying daemon mds.g on smithi116 2024-03-19T01:10:20.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:19 smithi156 ceph-mon[66799]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:20.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:19 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] up:boot 2024-03-19T01:10:20.225 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:19 smithi156 ceph-mon[66799]: fsmap 9 up:standby 2024-03-19T01:10:20.225 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T01:10:21.111 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:20 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1760495975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:21.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:20 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1760495975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:21.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:20 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1760495975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:21.405 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:10:21 smithi116 podman[103961]: 2024-03-19 01:10:21.355195589 +0000 UTC m=+0.139241498 container create d1da46433e9cb2d1e57f304be45402d1a49c47822d03a8adee7962d05bbc0d14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-g, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True) 2024-03-19T01:10:21.867 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:10:21 smithi116 podman[103961]: 2024-03-19 01:10:21.599731654 +0000 UTC m=+0.383777563 container init d1da46433e9cb2d1e57f304be45402d1a49c47822d03a8adee7962d05bbc0d14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-g, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-19T01:10:21.868 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:10:21 smithi116 podman[103961]: 2024-03-19 01:10:21.602227246 +0000 UTC m=+0.386273164 container start d1da46433e9cb2d1e57f304be45402d1a49c47822d03a8adee7962d05bbc0d14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-g, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-19T01:10:21.868 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:10:21 smithi116 ceph-mds[103980]: -- 172.21.15.116:0/87826371 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559255851a40 con 0x559255825000 2024-03-19T01:10:21.868 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:10:21 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-g[103976]: starting mds.g at 2024-03-19T01:10:21.868 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:10:21 smithi116 bash[103961]: d1da46433e9cb2d1e57f304be45402d1a49c47822d03a8adee7962d05bbc0d14 2024-03-19T01:10:21.868 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:10:21 smithi116 systemd[1]: Started Ceph mds.g for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:10:22.037 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:22.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.146+0000 7f6b96b9d640 1 -- 172.21.15.116:0/3802127972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b90100570 msgr2=0x7f6b90100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:22.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.146+0000 7f6b96b9d640 1 --2- 172.21.15.116:0/3802127972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b90100570 0x7f6b90100990 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009900 tx=0x7f6b8c02bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.146+0000 7f6b96b9d640 1 -- 172.21.15.116:0/3802127972 shutdown_connections 2024-03-19T01:10:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.146+0000 7f6b96b9d640 1 --2- 172.21.15.116:0/3802127972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b90102120 0x7f6b901089b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.146+0000 7f6b96b9d640 1 --2- 172.21.15.116:0/3802127972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b90101760 0x7f6b90101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.146+0000 7f6b96b9d640 1 --2- 172.21.15.116:0/3802127972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b90100570 0x7f6b90100990 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.146+0000 7f6b96b9d640 1 -- 172.21.15.116:0/3802127972 >> 172.21.15.116:0/3802127972 conn(0x7f6b900fbb00 msgr2=0x7f6b900fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.146+0000 7f6b96b9d640 1 -- 172.21.15.116:0/3802127972 shutdown_connections 2024-03-19T01:10:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.146+0000 7f6b96b9d640 1 -- 172.21.15.116:0/3802127972 wait complete. 2024-03-19T01:10:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b96b9d640 1 Processor -- start 2024-03-19T01:10:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b96b9d640 1 -- start start 2024-03-19T01:10:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b96b9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b90100570 0x7f6b90196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b96b9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b90101760 0x7f6b90196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:22.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b96b9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b90102120 0x7f6b9019de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b96b9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9010c200 con 0x7f6b90100570 2024-03-19T01:10:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b96b9d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9010c080 con 0x7f6b90101760 2024-03-19T01:10:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b96b9d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9010c380 con 0x7f6b90102120 2024-03-19T01:10:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b95b9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b90100570 0x7f6b90196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b95b9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b90100570 0x7f6b90196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56536/0 (socket says 172.21.15.116:56536) 2024-03-19T01:10:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b95b9b640 1 -- 172.21.15.116:0/2191659576 learned_addr learned my addr 172.21.15.116:0/2191659576 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b9639c640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b90102120 0x7f6b9019de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b9539a640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b90101760 0x7f6b90196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b95b9b640 1 -- 172.21.15.116:0/2191659576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b90102120 msgr2=0x7f6b9019de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b95b9b640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b90102120 0x7f6b9019de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b95b9b640 1 -- 172.21.15.116:0/2191659576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b90101760 msgr2=0x7f6b90196dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:22.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b95b9b640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b90101760 0x7f6b90196dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b95b9b640 1 -- 172.21.15.116:0/2191659576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9019e4b0 con 0x7f6b90100570 2024-03-19T01:10:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b9639c640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b90102120 0x7f6b9019de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:10:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.147+0000 7f6b9539a640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b90101760 0x7f6b90196dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:10:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.148+0000 7f6b95b9b640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b90100570 0x7f6b90196890 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c000c00 tx=0x7f6b8c02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.148+0000 7f6b86ffd640 1 -- 172.21.15.116:0/2191659576 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c041070 con 0x7f6b90100570 2024-03-19T01:10:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.148+0000 7f6b86ffd640 1 -- 172.21.15.116:0/2191659576 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b8c004450 con 0x7f6b90100570 2024-03-19T01:10:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.148+0000 7f6b96b9d640 1 -- 172.21.15.116:0/2191659576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9019e740 con 0x7f6b90100570 2024-03-19T01:10:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.148+0000 7f6b86ffd640 1 -- 172.21.15.116:0/2191659576 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c034440 con 0x7f6b90100570 2024-03-19T01:10:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.148+0000 7f6b96b9d640 1 -- 172.21.15.116:0/2191659576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9019ec30 con 0x7f6b90100570 2024-03-19T01:10:22.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.149+0000 7f6b96b9d640 1 -- 172.21.15.116:0/2191659576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b90107910 con 0x7f6b90100570 2024-03-19T01:10:22.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:21 smithi162 ceph-mon[67178]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:22.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:22.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:22.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:22.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:22.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:22.153 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:21 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:22.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.150+0000 7f6b86ffd640 1 -- 172.21.15.116:0/2191659576 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f6b8c044070 con 0x7f6b90100570 2024-03-19T01:10:22.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.150+0000 7f6b86ffd640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b7c0a2800 0x7f6b7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:22.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.151+0000 7f6b86ffd640 1 -- 172.21.15.116:0/2191659576 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f6b8c0f03f0 con 0x7f6b90100570 2024-03-19T01:10:22.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.151+0000 7f6b9539a640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b7c0a2800 0x7f6b7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:22.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.151+0000 7f6b9539a640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b7c0a2800 0x7f6b7c0a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6b90197da0 tx=0x7f6b80005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:22.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.154+0000 7f6b86ffd640 1 -- 172.21.15.116:0/2191659576 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b8c0bd9c0 con 0x7f6b90100570 2024-03-19T01:10:22.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:21 smithi116 ceph-mon[62701]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:22.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:22.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:22.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:22.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:22.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:22.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:21 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:22.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:21 smithi156 ceph-mon[66799]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:22.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:22.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:22.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:22.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:22.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:22.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:21 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:22.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.391+0000 7f6b96b9d640 1 -- 172.21.15.116:0/2191659576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b9002d1f0 con 0x7f6b90100570 2024-03-19T01:10:22.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.392+0000 7f6b86ffd640 1 -- 172.21.15.116:0/2191659576 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7f6b8c0bd2c0 con 0x7f6b90100570 2024-03-19T01:10:22.393 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:22.393 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1242639081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1242639081},{"type":"v1","addr":"172.21.15.116:6839","nonce":1242639081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/441486392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":441486392},{"type":"v1","addr":"172.21.15.162:6839","nonce":441486392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-19T01:10:22.393 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 11 2024-03-19T01:10:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.394+0000 7f6b96b9d640 1 -- 172.21.15.116:0/2191659576 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b7c0a2800 msgr2=0x7f6b7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.394+0000 7f6b96b9d640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b7c0a2800 0x7f6b7c0a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6b90197da0 tx=0x7f6b80005f90 comp rx=0 tx=0).stop 2024-03-19T01:10:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.394+0000 7f6b96b9d640 1 -- 172.21.15.116:0/2191659576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b90100570 msgr2=0x7f6b90196890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.394+0000 7f6b96b9d640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b90100570 0x7f6b90196890 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c000c00 tx=0x7f6b8c02c640 comp rx=0 tx=0).stop 2024-03-19T01:10:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.394+0000 7f6b96b9d640 1 -- 172.21.15.116:0/2191659576 shutdown_connections 2024-03-19T01:10:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.394+0000 7f6b96b9d640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b90102120 0x7f6b9019de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.394+0000 7f6b96b9d640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b7c0a2800 0x7f6b7c0a4cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.394+0000 7f6b96b9d640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b90101760 0x7f6b90196dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:22.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.394+0000 7f6b96b9d640 1 --2- 172.21.15.116:0/2191659576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b90100570 0x7f6b90196890 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:22.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.394+0000 7f6b96b9d640 1 -- 172.21.15.116:0/2191659576 >> 172.21.15.116:0/2191659576 conn(0x7f6b900fbb00 msgr2=0x7f6b90104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:22.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.394+0000 7f6b96b9d640 1 -- 172.21.15.116:0/2191659576 shutdown_connections 2024-03-19T01:10:22.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:22.394+0000 7f6b96b9d640 1 -- 172.21.15.116:0/2191659576 wait complete. 2024-03-19T01:10:23.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:22 smithi116 ceph-mon[62701]: Deploying daemon mds.a on smithi116 2024-03-19T01:10:23.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:22 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.116:6838/1242639081,v1:172.21.15.116:6839/1242639081] up:boot 2024-03-19T01:10:23.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:22 smithi116 ceph-mon[62701]: fsmap 10 up:standby 2024-03-19T01:10:23.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:22 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T01:10:23.124 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:22 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2191659576' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:23.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:22 smithi162 ceph-mon[67178]: Deploying daemon mds.a on smithi116 2024-03-19T01:10:23.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:22 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.116:6838/1242639081,v1:172.21.15.116:6839/1242639081] up:boot 2024-03-19T01:10:23.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:22 smithi162 ceph-mon[67178]: fsmap 10 up:standby 2024-03-19T01:10:23.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:22 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T01:10:23.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:22 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2191659576' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:23.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:22 smithi156 ceph-mon[66799]: Deploying daemon mds.a on smithi116 2024-03-19T01:10:23.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:22 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.116:6838/1242639081,v1:172.21.15.116:6839/1242639081] up:boot 2024-03-19T01:10:23.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:22 smithi156 ceph-mon[66799]: fsmap 10 up:standby 2024-03-19T01:10:23.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:22 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T01:10:23.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:22 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2191659576' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:23.973 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:23 smithi116 ceph-mon[62701]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:23.974 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:10:23 smithi116 systemd[1]: Starting Ceph mds.a for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:10:24.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:23 smithi162 ceph-mon[67178]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:24.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:23 smithi156 ceph-mon[66799]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:24.273 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:10:24 smithi116 podman[104309]: 2024-03-19 01:10:24.055000456 +0000 UTC m=+0.018838541 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:10:24.274 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:10:24 smithi116 podman[104309]: 2024-03-19 01:10:24.168759881 +0000 UTC m=+0.132597965 container create ab58a3562cb42f442fb893ada37619575e3ea0a2f202d61debbc0048510da8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-19T01:10:24.407 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:24.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.515+0000 7ff5889bc640 1 -- 172.21.15.116:0/2229047647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584100740 msgr2=0x7ff584100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.515+0000 7ff5889bc640 1 --2- 172.21.15.116:0/2229047647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584100740 0x7ff584100b60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff574009900 tx=0x7ff57402bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.516+0000 7ff5889bc640 1 -- 172.21.15.116:0/2229047647 shutdown_connections 2024-03-19T01:10:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.516+0000 7ff5889bc640 1 --2- 172.21.15.116:0/2229047647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5841022f0 0x7ff584108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.516+0000 7ff5889bc640 1 --2- 172.21.15.116:0/2229047647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff584101930 0x7ff584101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.516+0000 7ff5889bc640 1 --2- 172.21.15.116:0/2229047647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584100740 0x7ff584100b60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.516+0000 7ff5889bc640 1 -- 172.21.15.116:0/2229047647 >> 172.21.15.116:0/2229047647 conn(0x7ff5840fbcb0 msgr2=0x7ff5840fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.516+0000 7ff5889bc640 1 -- 172.21.15.116:0/2229047647 shutdown_connections 2024-03-19T01:10:24.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.516+0000 7ff5889bc640 1 -- 172.21.15.116:0/2229047647 wait complete. 2024-03-19T01:10:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.516+0000 7ff5889bc640 1 Processor -- start 2024-03-19T01:10:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.516+0000 7ff5889bc640 1 -- start start 2024-03-19T01:10:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.516+0000 7ff5889bc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff584100740 0x7ff5840694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:24.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.516+0000 7ff5889bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584101930 0x7ff58419ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.516+0000 7ff5889bc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5841022f0 0x7ff5841a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.516+0000 7ff5889bc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff58410c3c0 con 0x7ff584101930 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff5889bc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff58410c240 con 0x7ff584100740 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff5889bc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff58410c540 con 0x7ff5841022f0 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff581d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584101930 0x7ff58419ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff581d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584101930 0x7ff58419ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56574/0 (socket says 172.21.15.116:56574) 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff581d9b640 1 -- 172.21.15.116:0/2647202502 learned_addr learned my addr 172.21.15.116:0/2647202502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff58259c640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff584100740 0x7ff5840694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff582d9d640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5841022f0 0x7ff5841a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff581d9b640 1 -- 172.21.15.116:0/2647202502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5841022f0 msgr2=0x7ff5841a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff581d9b640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5841022f0 0x7ff5841a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff581d9b640 1 -- 172.21.15.116:0/2647202502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff584100740 msgr2=0x7ff5840694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff581d9b640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff584100740 0x7ff5840694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:24.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff581d9b640 1 -- 172.21.15.116:0/2647202502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5841a19c0 con 0x7ff584101930 2024-03-19T01:10:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff58259c640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff584100740 0x7ff5840694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:10:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff582d9d640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5841022f0 0x7ff5841a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:10:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff581d9b640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584101930 0x7ff58419ec80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c890 tx=0x7ff57800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff5737fe640 1 -- 172.21.15.116:0/2647202502 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff578004220 con 0x7ff584101930 2024-03-19T01:10:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff5889bc640 1 -- 172.21.15.116:0/2647202502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5841a1c50 con 0x7ff584101930 2024-03-19T01:10:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff5737fe640 1 -- 172.21.15.116:0/2647202502 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5780043c0 con 0x7ff584101930 2024-03-19T01:10:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff5737fe640 1 -- 172.21.15.116:0/2647202502 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff578004a00 con 0x7ff584101930 2024-03-19T01:10:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.517+0000 7ff5889bc640 1 -- 172.21.15.116:0/2647202502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5841a21a0 con 0x7ff584101930 2024-03-19T01:10:24.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.518+0000 7ff5889bc640 1 -- 172.21.15.116:0/2647202502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff584107a60 con 0x7ff584101930 2024-03-19T01:10:24.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.520+0000 7ff5737fe640 1 -- 172.21.15.116:0/2647202502 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7ff5780079b0 con 0x7ff584101930 2024-03-19T01:10:24.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.520+0000 7ff5737fe640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff56c0a2830 0x7ff56c0a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:24.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.520+0000 7ff58259c640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff56c0a2830 0x7ff56c0a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:24.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.520+0000 7ff5737fe640 1 -- 172.21.15.116:0/2647202502 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff5780ce690 con 0x7ff584101930 2024-03-19T01:10:24.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.521+0000 7ff58259c640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff56c0a2830 0x7ff56c0a4cf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff57402c7b0 tx=0x7ff574005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:24.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.527+0000 7ff5737fe640 1 -- 172.21.15.116:0/2647202502 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff578014030 con 0x7ff584101930 2024-03-19T01:10:24.655 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:10:24 smithi116 podman[104309]: 2024-03-19 01:10:24.396777922 +0000 UTC m=+0.360616008 container init ab58a3562cb42f442fb893ada37619575e3ea0a2f202d61debbc0048510da8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-a, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:10:24.655 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:10:24 smithi116 podman[104309]: 2024-03-19 01:10:24.400201471 +0000 UTC m=+0.364039564 container start ab58a3562cb42f442fb893ada37619575e3ea0a2f202d61debbc0048510da8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-19T01:10:24.656 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:10:24 smithi116 ceph-mds[104338]: -- 172.21.15.116:0/147400429 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558e9e1d9a40 con 0x558e9e1acc00 2024-03-19T01:10:24.656 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:10:24 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-a[104332]: starting mds.a at 2024-03-19T01:10:24.656 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:10:24 smithi116 bash[104309]: ab58a3562cb42f442fb893ada37619575e3ea0a2f202d61debbc0048510da8ee 2024-03-19T01:10:24.656 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:10:24 smithi116 systemd[1]: Started Ceph mds.a for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:10:24.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.765+0000 7ff5889bc640 1 -- 172.21.15.116:0/2647202502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff58402d3a0 con 0x7ff584101930 2024-03-19T01:10:24.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.765+0000 7ff5737fe640 1 -- 172.21.15.116:0/2647202502 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7ff57809bbd0 con 0x7ff584101930 2024-03-19T01:10:24.766 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:24.766 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1242639081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1242639081},{"type":"v1","addr":"172.21.15.116:6839","nonce":1242639081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/441486392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":441486392},{"type":"v1","addr":"172.21.15.162:6839","nonce":441486392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-03-19T01:10:24.767 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 11 2024-03-19T01:10:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.767+0000 7ff5889bc640 1 -- 172.21.15.116:0/2647202502 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff56c0a2830 msgr2=0x7ff56c0a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.767+0000 7ff5889bc640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff56c0a2830 0x7ff56c0a4cf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff57402c7b0 tx=0x7ff574005fd0 comp rx=0 tx=0).stop 2024-03-19T01:10:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.767+0000 7ff5889bc640 1 -- 172.21.15.116:0/2647202502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584101930 msgr2=0x7ff58419ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.767+0000 7ff5889bc640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584101930 0x7ff58419ec80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c890 tx=0x7ff57800cd50 comp rx=0 tx=0).stop 2024-03-19T01:10:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.767+0000 7ff5889bc640 1 -- 172.21.15.116:0/2647202502 shutdown_connections 2024-03-19T01:10:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.767+0000 7ff5889bc640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff5841022f0 0x7ff5841a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.767+0000 7ff5889bc640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff584101930 0x7ff58419ec80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.767+0000 7ff5889bc640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff56c0a2830 0x7ff56c0a4cf0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.767+0000 7ff5889bc640 1 --2- 172.21.15.116:0/2647202502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff584100740 0x7ff5840694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.767+0000 7ff5889bc640 1 -- 172.21.15.116:0/2647202502 >> 172.21.15.116:0/2647202502 conn(0x7ff5840fbcb0 msgr2=0x7ff584104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.767+0000 7ff5889bc640 1 -- 172.21.15.116:0/2647202502 shutdown_connections 2024-03-19T01:10:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:24.767+0000 7ff5889bc640 1 -- 172.21.15.116:0/2647202502 wait complete. 2024-03-19T01:10:25.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:25.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:25.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:25.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:25.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:25.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:25.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:25 smithi162 ceph-mon[67178]: Deploying daemon mds.b on smithi156 2024-03-19T01:10:25.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:25 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2647202502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:25.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:25 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] up:boot 2024-03-19T01:10:25.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:25 smithi162 ceph-mon[67178]: fsmap 11 up:standby 2024-03-19T01:10:25.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:25 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T01:10:25.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:25 smithi162 ceph-mon[67178]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:25.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:25.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:25.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:25.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:25.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:25.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:25.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:25 smithi116 ceph-mon[62701]: Deploying daemon mds.b on smithi156 2024-03-19T01:10:25.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:25 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2647202502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:25.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:25 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] up:boot 2024-03-19T01:10:25.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:25 smithi116 ceph-mon[62701]: fsmap 11 up:standby 2024-03-19T01:10:25.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:25 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T01:10:25.906 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:25 smithi116 ceph-mon[62701]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-19T01:10:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-19T01:10:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:25 smithi156 ceph-mon[66799]: Deploying daemon mds.b on smithi156 2024-03-19T01:10:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:25 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2647202502' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:25 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] up:boot 2024-03-19T01:10:25.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:25 smithi156 ceph-mon[66799]: fsmap 11 up:standby 2024-03-19T01:10:25.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:25 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T01:10:25.975 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:25 smithi156 ceph-mon[66799]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:26.779 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.892+0000 7fe67fd3b640 1 -- 172.21.15.116:0/2863321121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe678102360 msgr2=0x7fe678104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.892+0000 7fe67fd3b640 1 --2- 172.21.15.116:0/2863321121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe678102360 0x7fe678104750 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe674009900 tx=0x7fe67402bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.892+0000 7fe67fd3b640 1 -- 172.21.15.116:0/2863321121 shutdown_connections 2024-03-19T01:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.892+0000 7fe67fd3b640 1 --2- 172.21.15.116:0/2863321121 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6781075c0 0x7fe6781099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.892+0000 7fe67fd3b640 1 --2- 172.21.15.116:0/2863321121 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe678104c90 0x7fe678107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67fd3b640 1 --2- 172.21.15.116:0/2863321121 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe678102360 0x7fe678104750 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:26.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67fd3b640 1 -- 172.21.15.116:0/2863321121 >> 172.21.15.116:0/2863321121 conn(0x7fe6780fbcb0 msgr2=0x7fe6780fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67fd3b640 1 -- 172.21.15.116:0/2863321121 shutdown_connections 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67fd3b640 1 -- 172.21.15.116:0/2863321121 wait complete. 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67fd3b640 1 Processor -- start 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67fd3b640 1 -- start start 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67fd3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe678102360 0x7fe678101cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67fd3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe678104c90 0x7fe678100310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67fd3b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6781075c0 0x7fe678100850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67fd3b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe67810d4e0 con 0x7fe678104c90 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67fd3b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe67810d360 con 0x7fe6781075c0 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67fd3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe67810d660 con 0x7fe678102360 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67d2d6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe678104c90 0x7fe678100310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67d2d6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe678104c90 0x7fe678100310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35856/0 (socket says 172.21.15.116:35856) 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.893+0000 7fe67d2d6640 1 -- 172.21.15.116:0/3441501917 learned_addr learned my addr 172.21.15.116:0/3441501917 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:26.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.894+0000 7fe67e2d8640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6781075c0 0x7fe678100850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.894+0000 7fe67dad7640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe678102360 0x7fe678101cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.894+0000 7fe67d2d6640 1 -- 172.21.15.116:0/3441501917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe678102360 msgr2=0x7fe678101cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.894+0000 7fe67d2d6640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe678102360 0x7fe678101cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.894+0000 7fe67d2d6640 1 -- 172.21.15.116:0/3441501917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6781075c0 msgr2=0x7fe678100850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.894+0000 7fe67d2d6640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6781075c0 0x7fe678100850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.894+0000 7fe67d2d6640 1 -- 172.21.15.116:0/3441501917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6781010e0 con 0x7fe678104c90 2024-03-19T01:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.894+0000 7fe67d2d6640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe678104c90 0x7fe678100310 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe66800c970 tx=0x7fe66800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.894+0000 7fe66effd640 1 -- 172.21.15.116:0/3441501917 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe668007bb0 con 0x7fe678104c90 2024-03-19T01:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.894+0000 7fe66effd640 1 -- 172.21.15.116:0/3441501917 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe668007d50 con 0x7fe678104c90 2024-03-19T01:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.894+0000 7fe67fd3b640 1 -- 172.21.15.116:0/3441501917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6781a1c40 con 0x7fe678104c90 2024-03-19T01:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.894+0000 7fe66effd640 1 -- 172.21.15.116:0/3441501917 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6680187a0 con 0x7fe678104c90 2024-03-19T01:10:26.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.894+0000 7fe67fd3b640 1 -- 172.21.15.116:0/3441501917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6781a2140 con 0x7fe678104c90 2024-03-19T01:10:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.895+0000 7fe67fd3b640 1 -- 172.21.15.116:0/3441501917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6780fd7a0 con 0x7fe678104c90 2024-03-19T01:10:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.896+0000 7fe66effd640 1 -- 172.21.15.116:0/3441501917 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fe66800f450 con 0x7fe678104c90 2024-03-19T01:10:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.897+0000 7fe66effd640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe6640a2800 0x7fe6640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.897+0000 7fe67dad7640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe6640a2800 0x7fe6640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:26.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.897+0000 7fe66effd640 1 -- 172.21.15.116:0/3441501917 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe6680ce980 con 0x7fe678104c90 2024-03-19T01:10:26.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.897+0000 7fe67dad7640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe6640a2800 0x7fe6640a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe674005ab0 tx=0x7fe674004f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:26.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:26.900+0000 7fe66effd640 1 -- 172.21.15.116:0/3441501917 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe668014030 con 0x7fe678104c90 2024-03-19T01:10:26.974 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:10:26 smithi156 systemd[1]: Starting Ceph mds.b for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:10:27.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.120+0000 7fe67fd3b640 1 -- 172.21.15.116:0/3441501917 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe67802d1f0 con 0x7fe678104c90 2024-03-19T01:10:27.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.121+0000 7fe66effd640 1 -- 172.21.15.116:0/3441501917 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7fe66809bec0 con 0x7fe678104c90 2024-03-19T01:10:27.122 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:27.122 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1242639081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1242639081},{"type":"v1","addr":"172.21.15.116:6839","nonce":1242639081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/441486392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":441486392},{"type":"v1","addr":"172.21.15.162:6839","nonce":441486392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24488,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/3343188504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3343188504},{"type":"v1","addr":"172.21.15.116:6841","nonce":3343188504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-03-19T01:10:27.122 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 12 2024-03-19T01:10:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.123+0000 7fe67fd3b640 1 -- 172.21.15.116:0/3441501917 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe6640a2800 msgr2=0x7fe6640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.123+0000 7fe67fd3b640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe6640a2800 0x7fe6640a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe674005ab0 tx=0x7fe674004f10 comp rx=0 tx=0).stop 2024-03-19T01:10:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.123+0000 7fe67fd3b640 1 -- 172.21.15.116:0/3441501917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe678104c90 msgr2=0x7fe678100310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.123+0000 7fe67fd3b640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe678104c90 0x7fe678100310 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe66800c970 tx=0x7fe66800ce30 comp rx=0 tx=0).stop 2024-03-19T01:10:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.123+0000 7fe67fd3b640 1 -- 172.21.15.116:0/3441501917 shutdown_connections 2024-03-19T01:10:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.123+0000 7fe67fd3b640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe6640a2800 0x7fe6640a4cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.123+0000 7fe67fd3b640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe6781075c0 0x7fe678100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.123+0000 7fe67fd3b640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe678104c90 0x7fe678100310 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.123+0000 7fe67fd3b640 1 --2- 172.21.15.116:0/3441501917 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe678102360 0x7fe678101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.123+0000 7fe67fd3b640 1 -- 172.21.15.116:0/3441501917 >> 172.21.15.116:0/3441501917 conn(0x7fe6780fbcb0 msgr2=0x7fe678102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.123+0000 7fe67fd3b640 1 -- 172.21.15.116:0/3441501917 shutdown_connections 2024-03-19T01:10:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:27.123+0000 7fe67fd3b640 1 -- 172.21.15.116:0/3441501917 wait complete. 2024-03-19T01:10:27.348 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:10:27 smithi156 podman[97307]: 2024-03-19 01:10:26.977009514 +0000 UTC m=+0.018161144 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:10:27.349 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:10:27 smithi156 podman[97307]: 2024-03-19 01:10:27.087307426 +0000 UTC m=+0.128459041 container create dfeadb187af2ffa2bbb3472d1679dbf281f0da6997143435336f69ba9cc9a961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-19T01:10:27.349 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:10:27 smithi156 podman[97307]: 2024-03-19 01:10:27.348861814 +0000 UTC m=+0.390013432 container init dfeadb187af2ffa2bbb3472d1679dbf281f0da6997143435336f69ba9cc9a961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-19T01:10:27.724 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:10:27 smithi156 podman[97307]: 2024-03-19 01:10:27.351871693 +0000 UTC m=+0.393023315 container start dfeadb187af2ffa2bbb3472d1679dbf281f0da6997143435336f69ba9cc9a961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-03-19T01:10:27.724 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:10:27 smithi156 ceph-mds[97326]: -- 172.21.15.156:0/2940547520 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557d063b5a40 con 0x557d06388c00 2024-03-19T01:10:27.724 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:10:27 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b[97322]: starting mds.b at 2024-03-19T01:10:27.724 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:10:27 smithi156 bash[97307]: dfeadb187af2ffa2bbb3472d1679dbf281f0da6997143435336f69ba9cc9a961 2024-03-19T01:10:27.724 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:10:27 smithi156 systemd[1]: Started Ceph mds.b for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:10:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:28 smithi162 ceph-mon[67178]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:28 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3441501917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:10:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:28 smithi116 ceph-mon[62701]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:28 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3441501917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:10:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:28 smithi156 ceph-mon[66799]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:28 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3441501917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:10:29.136 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:29.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.246+0000 7fd117697640 1 -- 172.21.15.116:0/1179966185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd110100730 msgr2=0x7fd110100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.246+0000 7fd117697640 1 --2- 172.21.15.116:0/1179966185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd110100730 0x7fd110100b50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd10c009900 tx=0x7fd10c02bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.246+0000 7fd117697640 1 -- 172.21.15.116:0/1179966185 shutdown_connections 2024-03-19T01:10:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.246+0000 7fd117697640 1 --2- 172.21.15.116:0/1179966185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1101022e0 0x7fd110108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.246+0000 7fd117697640 1 --2- 172.21.15.116:0/1179966185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd110101920 0x7fd110101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.246+0000 7fd117697640 1 --2- 172.21.15.116:0/1179966185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd110100730 0x7fd110100b50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.246+0000 7fd117697640 1 -- 172.21.15.116:0/1179966185 >> 172.21.15.116:0/1179966185 conn(0x7fd1100fbcc0 msgr2=0x7fd1100fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.246+0000 7fd117697640 1 -- 172.21.15.116:0/1179966185 shutdown_connections 2024-03-19T01:10:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.246+0000 7fd117697640 1 -- 172.21.15.116:0/1179966185 wait complete. 2024-03-19T01:10:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd117697640 1 Processor -- start 2024-03-19T01:10:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd117697640 1 -- start start 2024-03-19T01:10:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd117697640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd110100730 0x7fd110197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd117697640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd110101920 0x7fd110198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:29.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd117697640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1101022e0 0x7fd11019f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd117697640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd11010c3f0 con 0x7fd110101920 2024-03-19T01:10:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd117697640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd11010c270 con 0x7fd110100730 2024-03-19T01:10:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd117697640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd11010c570 con 0x7fd1101022e0 2024-03-19T01:10:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd114c32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd110101920 0x7fd110198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd115433640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd110100730 0x7fd110197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd114c32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd110101920 0x7fd110198160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35862/0 (socket says 172.21.15.116:35862) 2024-03-19T01:10:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd114c32640 1 -- 172.21.15.116:0/79733153 learned_addr learned my addr 172.21.15.116:0/79733153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd115c34640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1101022e0 0x7fd11019f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd115433640 1 -- 172.21.15.116:0/79733153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1101022e0 msgr2=0x7fd11019f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd115433640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1101022e0 0x7fd11019f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd115433640 1 -- 172.21.15.116:0/79733153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd110101920 msgr2=0x7fd110198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd115433640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd110101920 0x7fd110198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd115433640 1 -- 172.21.15.116:0/79733153 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd11019f7d0 con 0x7fd110100730 2024-03-19T01:10:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd114c32640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd110101920 0x7fd110198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:10:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.247+0000 7fd115433640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd110100730 0x7fd110197c20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd10c009230 tx=0x7fd10c005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.248+0000 7fd1067fc640 1 -- 172.21.15.116:0/79733153 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10c043020 con 0x7fd110100730 2024-03-19T01:10:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.248+0000 7fd117697640 1 -- 172.21.15.116:0/79733153 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd11019fa60 con 0x7fd110100730 2024-03-19T01:10:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.248+0000 7fd117697640 1 -- 172.21.15.116:0/79733153 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd11019ffd0 con 0x7fd110100730 2024-03-19T01:10:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.248+0000 7fd1067fc640 1 -- 172.21.15.116:0/79733153 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd10c0054e0 con 0x7fd110100730 2024-03-19T01:10:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.248+0000 7fd1067fc640 1 -- 172.21.15.116:0/79733153 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd10c003d80 con 0x7fd110100730 2024-03-19T01:10:29.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.249+0000 7fd117697640 1 -- 172.21.15.116:0/79733153 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd110068a60 con 0x7fd110100730 2024-03-19T01:10:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.250+0000 7fd1067fc640 1 -- 172.21.15.116:0/79733153 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fd10c046020 con 0x7fd110100730 2024-03-19T01:10:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.250+0000 7fd1067fc640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd0fc0a2800 0x7fd0fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.250+0000 7fd114c32640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd0fc0a2800 0x7fd0fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.251+0000 7fd1067fc640 1 -- 172.21.15.116:0/79733153 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd10c0f0160 con 0x7fd110100730 2024-03-19T01:10:29.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.251+0000 7fd114c32640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd0fc0a2800 0x7fd0fc0a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd1101990c0 tx=0x7fd10000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.254+0000 7fd1067fc640 1 -- 172.21.15.116:0/79733153 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd10c04d030 con 0x7fd110100730 2024-03-19T01:10:29.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:29 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] up:boot 2024-03-19T01:10:29.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:29 smithi162 ceph-mon[67178]: fsmap 12 up:standby 2024-03-19T01:10:29.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:29 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T01:10:29.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:29 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] up:boot 2024-03-19T01:10:29.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:29 smithi116 ceph-mon[62701]: fsmap 12 up:standby 2024-03-19T01:10:29.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:29 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T01:10:29.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:29 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] up:boot 2024-03-19T01:10:29.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:29 smithi156 ceph-mon[66799]: fsmap 12 up:standby 2024-03-19T01:10:29.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:29 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T01:10:29.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.500+0000 7fd117697640 1 -- 172.21.15.116:0/79733153 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1101a03f0 con 0x7fd110100730 2024-03-19T01:10:29.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.501+0000 7fd1067fc640 1 -- 172.21.15.116:0/79733153 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10491 (secure 0 0 0) 0x7fd10c0bd600 con 0x7fd110100730 2024-03-19T01:10:29.502 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:29.502 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6839/1242639081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1242639081},{"type":"v1","addr":"172.21.15.116:6839","nonce":1242639081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24427,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24434,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24445,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24460,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.162:6839/441486392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":441486392},{"type":"v1","addr":"172.21.15.162:6839","nonce":441486392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24488,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6841/3343188504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3343188504},{"type":"v1","addr":"172.21.15.116:6841","nonce":3343188504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/1601537427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1601537427},{"type":"v1","addr":"172.21.15.156:6839","nonce":1601537427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-19T01:10:29.502 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 13 2024-03-19T01:10:29.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.503+0000 7fd117697640 1 -- 172.21.15.116:0/79733153 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd0fc0a2800 msgr2=0x7fd0fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:29.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.503+0000 7fd117697640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd0fc0a2800 0x7fd0fc0a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd1101990c0 tx=0x7fd10000b410 comp rx=0 tx=0).stop 2024-03-19T01:10:29.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.503+0000 7fd117697640 1 -- 172.21.15.116:0/79733153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd110100730 msgr2=0x7fd110197c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:29.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.503+0000 7fd117697640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd110100730 0x7fd110197c20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd10c009230 tx=0x7fd10c005dd0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.503+0000 7fd117697640 1 -- 172.21.15.116:0/79733153 shutdown_connections 2024-03-19T01:10:29.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.503+0000 7fd117697640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1101022e0 0x7fd11019f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.503+0000 7fd117697640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd110101920 0x7fd110198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.503+0000 7fd117697640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd0fc0a2800 0x7fd0fc0a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.503+0000 7fd117697640 1 --2- 172.21.15.116:0/79733153 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd110100730 0x7fd110197c20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.503+0000 7fd117697640 1 -- 172.21.15.116:0/79733153 >> 172.21.15.116:0/79733153 conn(0x7fd1100fbcc0 msgr2=0x7fd1100fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:29.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.503+0000 7fd117697640 1 -- 172.21.15.116:0/79733153 shutdown_connections 2024-03-19T01:10:29.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.503+0000 7fd117697640 1 -- 172.21.15.116:0/79733153 wait complete. 2024-03-19T01:10:29.516 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:10:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.619+0000 7f0033722640 1 -- 172.21.15.116:0/4101701814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f002c075860 msgr2=0x7f002c073cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.619+0000 7f0033722640 1 --2- 172.21.15.116:0/4101701814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f002c075860 0x7f002c073cb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f001c009900 tx=0x7f001c02bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 -- 172.21.15.116:0/4101701814 shutdown_connections 2024-03-19T01:10:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 --2- 172.21.15.116:0/4101701814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c109720 0x7f002c10bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 --2- 172.21.15.116:0/4101701814 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f002c074280 0x7f002c074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 --2- 172.21.15.116:0/4101701814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f002c075860 0x7f002c073cb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 -- 172.21.15.116:0/4101701814 >> 172.21.15.116:0/4101701814 conn(0x7f002c0fcdd0 msgr2=0x7f002c0ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 -- 172.21.15.116:0/4101701814 shutdown_connections 2024-03-19T01:10:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 -- 172.21.15.116:0/4101701814 wait complete. 2024-03-19T01:10:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 Processor -- start 2024-03-19T01:10:29.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 -- start start 2024-03-19T01:10:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f002c074280 0x7f002c199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c075860 0x7f002c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f002c109720 0x7f002c1a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f002c10f6c0 con 0x7f002c109720 2024-03-19T01:10:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f002c10f540 con 0x7f002c075860 2024-03-19T01:10:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.620+0000 7f0033722640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f002c10f840 con 0x7f002c074280 2024-03-19T01:10:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f0031cbf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f002c109720 0x7f002c1a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f0031cbf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f002c109720 0x7f002c1a1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35884/0 (socket says 172.21.15.116:35884) 2024-03-19T01:10:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f0031cbf640 1 -- 172.21.15.116:0/818926618 learned_addr learned my addr 172.21.15.116:0/818926618 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f00314be640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f002c074280 0x7f002c199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f0030cbd640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c075860 0x7f002c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f0031cbf640 1 -- 172.21.15.116:0/818926618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f002c074280 msgr2=0x7f002c199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:29.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f0031cbf640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f002c074280 0x7f002c199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f0031cbf640 1 -- 172.21.15.116:0/818926618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c075860 msgr2=0x7f002c19a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f0031cbf640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c075860 0x7f002c19a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f0031cbf640 1 -- 172.21.15.116:0/818926618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002c1a19d0 con 0x7f002c109720 2024-03-19T01:10:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f0030cbd640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c075860 0x7f002c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:10:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f0031cbf640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f002c109720 0x7f002c1a1360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f002400efd0 tx=0x7f002400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f00227fc640 1 -- 172.21.15.116:0/818926618 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0024003b70 con 0x7f002c109720 2024-03-19T01:10:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f0033722640 1 -- 172.21.15.116:0/818926618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f002c1a1cc0 con 0x7f002c109720 2024-03-19T01:10:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f00227fc640 1 -- 172.21.15.116:0/818926618 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0024003d10 con 0x7f002c109720 2024-03-19T01:10:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f00227fc640 1 -- 172.21.15.116:0/818926618 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00240109f0 con 0x7f002c109720 2024-03-19T01:10:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.621+0000 7f0033722640 1 -- 172.21.15.116:0/818926618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f002c1a21c0 con 0x7f002c109720 2024-03-19T01:10:29.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.622+0000 7f0033722640 1 -- 172.21.15.116:0/818926618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f002c108450 con 0x7f002c109720 2024-03-19T01:10:29.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.623+0000 7f00227fc640 1 -- 172.21.15.116:0/818926618 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f0024010b90 con 0x7f002c109720 2024-03-19T01:10:29.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.624+0000 7f00227fc640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f00180a2800 0x7f00180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:29.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.624+0000 7f00314be640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f00180a2800 0x7f00180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:29.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.624+0000 7f00227fc640 1 -- 172.21.15.116:0/818926618 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f00240cee70 con 0x7f002c109720 2024-03-19T01:10:29.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.624+0000 7f00314be640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f00180a2800 0x7f00180a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f001c02c7b0 tx=0x7f001c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:29.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.628+0000 7f00227fc640 1 -- 172.21.15.116:0/818926618 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f002409c3b0 con 0x7f002c109720 2024-03-19T01:10:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.814+0000 7f0033722640 1 -- 172.21.15.116:0/818926618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f002c02d440 con 0x7f002c109720 2024-03-19T01:10:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.815+0000 7f00227fc640 1 -- 172.21.15.116:0/818926618 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18516 (secure 0 0 0) 0x7f002409bcb0 con 0x7f002c109720 2024-03-19T01:10:29.816 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:29.816 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:09:44.118744+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:10:29.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.817+0000 7f0033722640 1 -- 172.21.15.116:0/818926618 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f00180a2800 msgr2=0x7f00180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:29.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.817+0000 7f0033722640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f00180a2800 0x7f00180a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f001c02c7b0 tx=0x7f001c005fd0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.817+0000 7f0033722640 1 -- 172.21.15.116:0/818926618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f002c109720 msgr2=0x7f002c1a1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:29.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.817+0000 7f0033722640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f002c109720 0x7f002c1a1360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f002400efd0 tx=0x7f002400c6a0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.817+0000 7f0033722640 1 -- 172.21.15.116:0/818926618 shutdown_connections 2024-03-19T01:10:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.817+0000 7f0033722640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f002c109720 0x7f002c1a1360 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.817+0000 7f0033722640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f00180a2800 0x7f00180a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.817+0000 7f0033722640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c075860 0x7f002c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.817+0000 7f0033722640 1 --2- 172.21.15.116:0/818926618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f002c074280 0x7f002c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.817+0000 7f0033722640 1 -- 172.21.15.116:0/818926618 >> 172.21.15.116:0/818926618 conn(0x7f002c0fcdd0 msgr2=0x7f002c105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.817+0000 7f0033722640 1 -- 172.21.15.116:0/818926618 shutdown_connections 2024-03-19T01:10:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.817+0000 7f0033722640 1 -- 172.21.15.116:0/818926618 wait complete. 2024-03-19T01:10:29.829 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:05:53.301933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:10:29.830 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:10:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.952+0000 7f7664fc6640 1 -- 172.21.15.116:0/1693066468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76601009f0 msgr2=0x7f7660100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.952+0000 7f7664fc6640 1 --2- 172.21.15.116:0/1693066468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76601009f0 0x7f7660100e50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7654009900 tx=0x7f765402bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.952+0000 7f7664fc6640 1 -- 172.21.15.116:0/1693066468 shutdown_connections 2024-03-19T01:10:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.952+0000 7f7664fc6640 1 --2- 172.21.15.116:0/1693066468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7660101390 0x7f7660101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.952+0000 7f7664fc6640 1 --2- 172.21.15.116:0/1693066468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76601009f0 0x7f7660100e50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.952+0000 7f7664fc6640 1 --2- 172.21.15.116:0/1693066468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7660100090 0x7f76601004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.952+0000 7f7664fc6640 1 -- 172.21.15.116:0/1693066468 >> 172.21.15.116:0/1693066468 conn(0x7f76600fbcb0 msgr2=0x7f76600fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.952+0000 7f7664fc6640 1 -- 172.21.15.116:0/1693066468 shutdown_connections 2024-03-19T01:10:29.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.952+0000 7f7664fc6640 1 -- 172.21.15.116:0/1693066468 wait complete. 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f7664fc6640 1 Processor -- start 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f7664fc6640 1 -- start start 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f7664fc6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7660100090 0x7f7660199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f7664fc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76601009f0 0x7f766019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f7664fc6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7660101390 0x7f76601a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f7664fc6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f766010f690 con 0x7f76601009f0 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f7664fc6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f766010f510 con 0x7f7660101390 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f7664fc6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f766010f810 con 0x7f7660100090 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f765dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76601009f0 0x7f766019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f765dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76601009f0 0x7f766019a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35912/0 (socket says 172.21.15.116:35912) 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f765dd9b640 1 -- 172.21.15.116:0/3209830266 learned_addr learned my addr 172.21.15.116:0/3209830266 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f765ed9d640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7660101390 0x7f76601a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f765e59c640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7660100090 0x7f7660199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:29.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f765dd9b640 1 -- 172.21.15.116:0/3209830266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7660100090 msgr2=0x7f7660199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f765dd9b640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7660100090 0x7f7660199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f765dd9b640 1 -- 172.21.15.116:0/3209830266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7660101390 msgr2=0x7f76601a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f765dd9b640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7660101390 0x7f76601a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f765dd9b640 1 -- 172.21.15.116:0/3209830266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76601a1900 con 0x7f76601009f0 2024-03-19T01:10:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f765e59c640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7660100090 0x7f7660199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:10:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f765dd9b640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76601009f0 0x7f766019a230 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7654009230 tx=0x7f7654004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f764f7fe640 1 -- 172.21.15.116:0/3209830266 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654041070 con 0x7f76601009f0 2024-03-19T01:10:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f764f7fe640 1 -- 172.21.15.116:0/3209830266 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76540054e0 con 0x7f76601009f0 2024-03-19T01:10:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f7664fc6640 1 -- 172.21.15.116:0/3209830266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76601a1b90 con 0x7f76601009f0 2024-03-19T01:10:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.953+0000 7f764f7fe640 1 -- 172.21.15.116:0/3209830266 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76540346a0 con 0x7f76601009f0 2024-03-19T01:10:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.954+0000 7f7664fc6640 1 -- 172.21.15.116:0/3209830266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76601a2080 con 0x7f76601009f0 2024-03-19T01:10:29.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.954+0000 7f7664fc6640 1 -- 172.21.15.116:0/3209830266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f766010af40 con 0x7f76601009f0 2024-03-19T01:10:29.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.956+0000 7f764f7fe640 1 -- 172.21.15.116:0/3209830266 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f7654044070 con 0x7f76601009f0 2024-03-19T01:10:29.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.956+0000 7f764f7fe640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f76480a2800 0x7f76480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:29.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.956+0000 7f765e59c640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f76480a2800 0x7f76480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:29.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.957+0000 7f764f7fe640 1 -- 172.21.15.116:0/3209830266 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f76540f0710 con 0x7f76601009f0 2024-03-19T01:10:29.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.957+0000 7f765e59c640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f76480a2800 0x7f76480a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f766019b1c0 tx=0x7f7650005db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:29.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:29.960+0000 7f764f7fe640 1 -- 172.21.15.116:0/3209830266 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f76540bdc50 con 0x7f76601009f0 2024-03-19T01:10:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.145+0000 7f7664fc6640 1 -- 172.21.15.116:0/3209830266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f766002d380 con 0x7f76601009f0 2024-03-19T01:10:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.146+0000 7f764f7fe640 1 -- 172.21.15.116:0/3209830266 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f76540bd550 con 0x7f76601009f0 2024-03-19T01:10:30.146 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T01:10:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.148+0000 7f7664fc6640 1 -- 172.21.15.116:0/3209830266 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f76480a2800 msgr2=0x7f76480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.148+0000 7f7664fc6640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f76480a2800 0x7f76480a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f766019b1c0 tx=0x7f7650005db0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.148+0000 7f7664fc6640 1 -- 172.21.15.116:0/3209830266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76601009f0 msgr2=0x7f766019a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.148+0000 7f7664fc6640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76601009f0 0x7f766019a230 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7654009230 tx=0x7f7654004040 comp rx=0 tx=0).stop 2024-03-19T01:10:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.148+0000 7f7664fc6640 1 -- 172.21.15.116:0/3209830266 shutdown_connections 2024-03-19T01:10:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.148+0000 7f7664fc6640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f76480a2800 0x7f76480a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.148+0000 7f7664fc6640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7660101390 0x7f76601a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.148+0000 7f7664fc6640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76601009f0 0x7f766019a230 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.148+0000 7f7664fc6640 1 --2- 172.21.15.116:0/3209830266 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7660100090 0x7f7660199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.148+0000 7f7664fc6640 1 -- 172.21.15.116:0/3209830266 >> 172.21.15.116:0/3209830266 conn(0x7f76600fbcb0 msgr2=0x7f76600fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.148+0000 7f7664fc6640 1 -- 172.21.15.116:0/3209830266 shutdown_connections 2024-03-19T01:10:30.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.148+0000 7f7664fc6640 1 -- 172.21.15.116:0/3209830266 wait complete. 2024-03-19T01:10:30.159 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-19T01:10:30.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.262+0000 7fc0e244b640 1 -- 172.21.15.116:0/2060051769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc101930 msgr2=0x7fc0dc101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:30.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.262+0000 7fc0e244b640 1 --2- 172.21.15.116:0/2060051769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc101930 0x7fc0dc101db0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0009900 tx=0x7fc0d002bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:30.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.262+0000 7fc0e244b640 1 -- 172.21.15.116:0/2060051769 shutdown_connections 2024-03-19T01:10:30.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.262+0000 7fc0e244b640 1 --2- 172.21.15.116:0/2060051769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0dc1022f0 0x7fc0dc108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.262+0000 7fc0e244b640 1 --2- 172.21.15.116:0/2060051769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc101930 0x7fc0dc101db0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.262+0000 7fc0e244b640 1 --2- 172.21.15.116:0/2060051769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0dc100740 0x7fc0dc100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.262+0000 7fc0e244b640 1 -- 172.21.15.116:0/2060051769 >> 172.21.15.116:0/2060051769 conn(0x7fc0dc0fbcb0 msgr2=0x7fc0dc0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:30.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.262+0000 7fc0e244b640 1 -- 172.21.15.116:0/2060051769 shutdown_connections 2024-03-19T01:10:30.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.262+0000 7fc0e244b640 1 -- 172.21.15.116:0/2060051769 wait complete. 2024-03-19T01:10:30.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.263+0000 7fc0e244b640 1 Processor -- start 2024-03-19T01:10:30.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.263+0000 7fc0e244b640 1 -- start start 2024-03-19T01:10:30.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.263+0000 7fc0e244b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0dc100740 0x7fc0dc197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:30.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.263+0000 7fc0e244b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc101930 0x7fc0dc198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:30.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.263+0000 7fc0e244b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0dc1022f0 0x7fc0dc19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:30.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.263+0000 7fc0e244b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc10d4e0 con 0x7fc0dc101930 2024-03-19T01:10:30.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.263+0000 7fc0e244b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc10d360 con 0x7fc0dc1022f0 2024-03-19T01:10:30.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.263+0000 7fc0e244b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc10d660 con 0x7fc0dc100740 2024-03-19T01:10:30.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.263+0000 7fc0db7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc101930 0x7fc0dc198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:30.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.263+0000 7fc0db7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc101930 0x7fc0dc198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35932/0 (socket says 172.21.15.116:35932) 2024-03-19T01:10:30.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.263+0000 7fc0db7fe640 1 -- 172.21.15.116:0/686364406 learned_addr learned my addr 172.21.15.116:0/686364406 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:30.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.263+0000 7fc0dbfff640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0dc100740 0x7fc0dc197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:30.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.263+0000 7fc0e09e8640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0dc1022f0 0x7fc0dc19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:30.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.264+0000 7fc0db7fe640 1 -- 172.21.15.116:0/686364406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0dc100740 msgr2=0x7fc0dc197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:30.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.264+0000 7fc0db7fe640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0dc100740 0x7fc0dc197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.264+0000 7fc0db7fe640 1 -- 172.21.15.116:0/686364406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0dc1022f0 msgr2=0x7fc0dc19f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:30.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.264+0000 7fc0db7fe640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0dc1022f0 0x7fc0dc19f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.264+0000 7fc0db7fe640 1 -- 172.21.15.116:0/686364406 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0dc19f760 con 0x7fc0dc101930 2024-03-19T01:10:30.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.264+0000 7fc0db7fe640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc101930 0x7fc0dc198080 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0009230 tx=0x7fc0d0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:30.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.264+0000 7fc0d97fa640 1 -- 172.21.15.116:0/686364406 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0041070 con 0x7fc0dc101930 2024-03-19T01:10:30.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.264+0000 7fc0e244b640 1 -- 172.21.15.116:0/686364406 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0dc19f9f0 con 0x7fc0dc101930 2024-03-19T01:10:30.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.264+0000 7fc0d97fa640 1 -- 172.21.15.116:0/686364406 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0d00044d0 con 0x7fc0dc101930 2024-03-19T01:10:30.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.264+0000 7fc0d97fa640 1 -- 172.21.15.116:0/686364406 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0034440 con 0x7fc0dc101930 2024-03-19T01:10:30.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.264+0000 7fc0e244b640 1 -- 172.21.15.116:0/686364406 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0dc19fee0 con 0x7fc0dc101930 2024-03-19T01:10:30.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.265+0000 7fc0e244b640 1 -- 172.21.15.116:0/686364406 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0dc107a60 con 0x7fc0dc101930 2024-03-19T01:10:30.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.266+0000 7fc0d97fa640 1 -- 172.21.15.116:0/686364406 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fc0d0044070 con 0x7fc0dc101930 2024-03-19T01:10:30.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.267+0000 7fc0d97fa640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc0c40a2800 0x7fc0c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:30.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.267+0000 7fc0dbfff640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc0c40a2800 0x7fc0c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:30.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.267+0000 7fc0d97fa640 1 -- 172.21.15.116:0/686364406 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc0d00f0250 con 0x7fc0dc101930 2024-03-19T01:10:30.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.268+0000 7fc0dbfff640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc0c40a2800 0x7fc0c40a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc005fd0 tx=0x7fc0cc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:30.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.272+0000 7fc0d97fa640 1 -- 172.21.15.116:0/686364406 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc0d00bd790 con 0x7fc0dc101930 2024-03-19T01:10:30.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:30 smithi162 ceph-mon[67178]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:30.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:30 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:30 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:30 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:30 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:30 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:30 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:30 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/79733153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:30.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:30 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/818926618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:30.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:30 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:30.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:30 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:10:30.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:30 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:30 smithi116 ceph-mon[62701]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:30.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:30 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/79733153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:30.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:30 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/818926618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:30.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:30.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:10:30.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:30 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.455+0000 7fc0e244b640 1 -- 172.21.15.116:0/686364406 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc0dc02d3b0 con 0x7fc0dc101930 2024-03-19T01:10:30.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.456+0000 7fc0d97fa640 1 -- 172.21.15.116:0/686364406 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fc0d00bd090 con 0x7fc0dc101930 2024-03-19T01:10:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:30.457 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-03-19T01:10:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:10:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2024-03-19T01:10:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2024-03-19T01:10:30.457 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:10:30.457 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-03-19T01:10:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.458+0000 7fc0e244b640 1 -- 172.21.15.116:0/686364406 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc0c40a2800 msgr2=0x7fc0c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.458+0000 7fc0e244b640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc0c40a2800 0x7fc0c40a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc005fd0 tx=0x7fc0cc00b410 comp rx=0 tx=0).stop 2024-03-19T01:10:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.458+0000 7fc0e244b640 1 -- 172.21.15.116:0/686364406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc101930 msgr2=0x7fc0dc198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.458+0000 7fc0e244b640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc101930 0x7fc0dc198080 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc0d0009230 tx=0x7fc0d0005d50 comp rx=0 tx=0).stop 2024-03-19T01:10:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.458+0000 7fc0e244b640 1 -- 172.21.15.116:0/686364406 shutdown_connections 2024-03-19T01:10:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.458+0000 7fc0e244b640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc0c40a2800 0x7fc0c40a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.458+0000 7fc0e244b640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc0dc1022f0 0x7fc0dc19f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.458+0000 7fc0e244b640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0dc101930 0x7fc0dc198080 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.458+0000 7fc0e244b640 1 --2- 172.21.15.116:0/686364406 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc0dc100740 0x7fc0dc197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.458+0000 7fc0e244b640 1 -- 172.21.15.116:0/686364406 >> 172.21.15.116:0/686364406 conn(0x7fc0dc0fbcb0 msgr2=0x7fc0dc104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:30.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.458+0000 7fc0e244b640 1 -- 172.21.15.116:0/686364406 shutdown_connections 2024-03-19T01:10:30.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.458+0000 7fc0e244b640 1 -- 172.21.15.116:0/686364406 wait complete. 2024-03-19T01:10:30.470 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-19T01:10:30.470 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-19T01:10:30.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:30 smithi156 ceph-mon[66799]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:30.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:30 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:30 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:30 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:30 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:30 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:30 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:30 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/79733153' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:30.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:30 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/818926618' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:30.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:30 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:30.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:30 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:10:30.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:30 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:30.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.577+0000 7f2515b90640 1 -- 172.21.15.116:0/3669848340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25101022d0 msgr2=0x7f2510108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.577+0000 7f2515b90640 1 --2- 172.21.15.116:0/3669848340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25101022d0 0x7f2510108b60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2508009ca0 tx=0x7f250802c0f0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.577+0000 7f2515b90640 1 -- 172.21.15.116:0/3669848340 shutdown_connections 2024-03-19T01:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.577+0000 7f2515b90640 1 --2- 172.21.15.116:0/3669848340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25101022d0 0x7f2510108b60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.577+0000 7f2515b90640 1 --2- 172.21.15.116:0/3669848340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2510101910 0x7f2510101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.577+0000 7f2515b90640 1 --2- 172.21.15.116:0/3669848340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2510100720 0x7f2510100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.577+0000 7f2515b90640 1 -- 172.21.15.116:0/3669848340 >> 172.21.15.116:0/3669848340 conn(0x7f25100fbcb0 msgr2=0x7f25100fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.577+0000 7f2515b90640 1 -- 172.21.15.116:0/3669848340 shutdown_connections 2024-03-19T01:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.577+0000 7f2515b90640 1 -- 172.21.15.116:0/3669848340 wait complete. 2024-03-19T01:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.577+0000 7f2515b90640 1 Processor -- start 2024-03-19T01:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.577+0000 7f2515b90640 1 -- start start 2024-03-19T01:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f2515b90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2510100720 0x7f25101046b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f2515b90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2510101910 0x7f2510102cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:30.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f2515b90640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25101022d0 0x7f2510103230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f2515b90640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f251010c380 con 0x7f2510101910 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f2515b90640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f251010c200 con 0x7f25101022d0 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f2515b90640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f251010c500 con 0x7f2510100720 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2510101910 0x7f2510102cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2510101910 0x7f2510102cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35952/0 (socket says 172.21.15.116:35952) 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25101022d0 0x7f2510103230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250effd640 1 -- 172.21.15.116:0/2187272265 learned_addr learned my addr 172.21.15.116:0/2187272265 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250f7fe640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2510100720 0x7f25101046b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250effd640 1 -- 172.21.15.116:0/2187272265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2510100720 msgr2=0x7f25101046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250effd640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2510100720 0x7f25101046b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250effd640 1 -- 172.21.15.116:0/2187272265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25101022d0 msgr2=0x7f2510103230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250effd640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25101022d0 0x7f2510103230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250effd640 1 -- 172.21.15.116:0/2187272265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2510103ac0 con 0x7f2510101910 2024-03-19T01:10:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250ffff640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25101022d0 0x7f2510103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:10:30.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250effd640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2510101910 0x7f2510102cf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f250000d950 tx=0x7f250000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:30.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250cff9640 1 -- 172.21.15.116:0/2187272265 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2500017030 con 0x7f2510101910 2024-03-19T01:10:30.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f2515b90640 1 -- 172.21.15.116:0/2187272265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25101a1c40 con 0x7f2510101910 2024-03-19T01:10:30.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250cff9640 1 -- 172.21.15.116:0/2187272265 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2500004020 con 0x7f2510101910 2024-03-19T01:10:30.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f2515b90640 1 -- 172.21.15.116:0/2187272265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25101a2190 con 0x7f2510101910 2024-03-19T01:10:30.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.578+0000 7f250cff9640 1 -- 172.21.15.116:0/2187272265 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25000047a0 con 0x7f2510101910 2024-03-19T01:10:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.580+0000 7f2515b90640 1 -- 172.21.15.116:0/2187272265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25101a2420 con 0x7f2510101910 2024-03-19T01:10:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.581+0000 7f250cff9640 1 -- 172.21.15.116:0/2187272265 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f25000041d0 con 0x7f2510101910 2024-03-19T01:10:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.582+0000 7f250cff9640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f24fc0a2800 0x7f24fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.582+0000 7f250f7fe640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f24fc0a2800 0x7f24fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.582+0000 7f250cff9640 1 -- 172.21.15.116:0/2187272265 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f25000ce610 con 0x7f2510101910 2024-03-19T01:10:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.582+0000 7f250f7fe640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f24fc0a2800 0x7f24fc0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2510101770 tx=0x7f24f8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:30.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.585+0000 7f250cff9640 1 -- 172.21.15.116:0/2187272265 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f250009bb50 con 0x7f2510101910 2024-03-19T01:10:30.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:30.786+0000 7f2515b90640 1 -- 172.21.15.116:0/2187272265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f25101a25f0 con 0x7f2510101910 2024-03-19T01:10:31.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.127+0000 7f250cff9640 1 -- 172.21.15.116:0/2187272265 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f250009b450 con 0x7f2510101910 2024-03-19T01:10:31.128 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' created 2024-03-19T01:10:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.129+0000 7f2515b90640 1 -- 172.21.15.116:0/2187272265 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f24fc0a2800 msgr2=0x7f24fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.129+0000 7f2515b90640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f24fc0a2800 0x7f24fc0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2510101770 tx=0x7f24f8005f90 comp rx=0 tx=0).stop 2024-03-19T01:10:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.129+0000 7f2515b90640 1 -- 172.21.15.116:0/2187272265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2510101910 msgr2=0x7f2510102cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.129+0000 7f2515b90640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2510101910 0x7f2510102cf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f250000d950 tx=0x7f250000de10 comp rx=0 tx=0).stop 2024-03-19T01:10:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.129+0000 7f2515b90640 1 -- 172.21.15.116:0/2187272265 shutdown_connections 2024-03-19T01:10:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.129+0000 7f2515b90640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f24fc0a2800 0x7f24fc0a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.129+0000 7f2515b90640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25101022d0 0x7f2510103230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.129+0000 7f2515b90640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2510101910 0x7f2510102cf0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.129+0000 7f2515b90640 1 --2- 172.21.15.116:0/2187272265 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2510100720 0x7f25101046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.129+0000 7f2515b90640 1 -- 172.21.15.116:0/2187272265 >> 172.21.15.116:0/2187272265 conn(0x7f25100fbcb0 msgr2=0x7f25100fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.129+0000 7f2515b90640 1 -- 172.21.15.116:0/2187272265 shutdown_connections 2024-03-19T01:10:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.130+0000 7f2515b90640 1 -- 172.21.15.116:0/2187272265 wait complete. 2024-03-19T01:10:31.146 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-19T01:10:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.257+0000 7f3773a44640 1 -- 172.21.15.116:0/2277263173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c100720 msgr2=0x7f376c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.257+0000 7f3773a44640 1 --2- 172.21.15.116:0/2277263173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c100720 0x7f376c100b40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3768009ca0 tx=0x7f376802c0f0 comp rx=0 tx=0).stop 2024-03-19T01:10:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.257+0000 7f3773a44640 1 -- 172.21.15.116:0/2277263173 shutdown_connections 2024-03-19T01:10:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.257+0000 7f3773a44640 1 --2- 172.21.15.116:0/2277263173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f376c1022d0 0x7f376c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.257+0000 7f3773a44640 1 --2- 172.21.15.116:0/2277263173 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f376c101910 0x7f376c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.257+0000 7f3773a44640 1 --2- 172.21.15.116:0/2277263173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c100720 0x7f376c100b40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.257+0000 7f3773a44640 1 -- 172.21.15.116:0/2277263173 >> 172.21.15.116:0/2277263173 conn(0x7f376c0fbcb0 msgr2=0x7f376c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.257+0000 7f3773a44640 1 -- 172.21.15.116:0/2277263173 shutdown_connections 2024-03-19T01:10:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.257+0000 7f3773a44640 1 -- 172.21.15.116:0/2277263173 wait complete. 2024-03-19T01:10:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.257+0000 7f3773a44640 1 Processor -- start 2024-03-19T01:10:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.257+0000 7f3773a44640 1 -- start start 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3773a44640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f376c100720 0x7f376c197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3773a44640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f376c101910 0x7f376c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3773a44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c1022d0 0x7f376c19f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3773a44640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f376c10d490 con 0x7f376c1022d0 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3773a44640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f376c10d310 con 0x7f376c100720 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3773a44640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f376c10d610 con 0x7f376c101910 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3771fe1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c1022d0 0x7f376c19f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3771fe1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c1022d0 0x7f376c19f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35958/0 (socket says 172.21.15.116:35958) 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3770fdf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f376c101910 0x7f376c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3771fe1640 1 -- 172.21.15.116:0/576194431 learned_addr learned my addr 172.21.15.116:0/576194431 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f37717e0640 1 --2- 172.21.15.116:0/576194431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f376c100720 0x7f376c197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3771fe1640 1 -- 172.21.15.116:0/576194431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f376c101910 msgr2=0x7f376c1980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3771fe1640 1 --2- 172.21.15.116:0/576194431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f376c101910 0x7f376c1980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3771fe1640 1 -- 172.21.15.116:0/576194431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f376c100720 msgr2=0x7f376c197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:31.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3771fe1640 1 --2- 172.21.15.116:0/576194431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f376c100720 0x7f376c197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:31.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3771fe1640 1 -- 172.21.15.116:0/576194431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c19f780 con 0x7f376c1022d0 2024-03-19T01:10:31.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3771fe1640 1 --2- 172.21.15.116:0/576194431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c1022d0 0x7f376c19f110 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f376400c970 tx=0x7f376400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:31.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f37627fc640 1 -- 172.21.15.116:0/576194431 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764007bb0 con 0x7f376c1022d0 2024-03-19T01:10:31.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3773a44640 1 -- 172.21.15.116:0/576194431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376c19fa70 con 0x7f376c1022d0 2024-03-19T01:10:31.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f37627fc640 1 -- 172.21.15.116:0/576194431 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3764007d50 con 0x7f376c1022d0 2024-03-19T01:10:31.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f37627fc640 1 -- 172.21.15.116:0/576194431 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764018800 con 0x7f376c1022d0 2024-03-19T01:10:31.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.258+0000 7f3773a44640 1 -- 172.21.15.116:0/576194431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376c19ffc0 con 0x7f376c1022d0 2024-03-19T01:10:31.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.259+0000 7f3773a44640 1 -- 172.21.15.116:0/576194431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f376c107ae0 con 0x7f376c1022d0 2024-03-19T01:10:31.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.260+0000 7f37627fc640 1 -- 172.21.15.116:0/576194431 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f376400fa50 con 0x7f376c1022d0 2024-03-19T01:10:31.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.261+0000 7f37627fc640 1 --2- 172.21.15.116:0/576194431 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37580a2800 0x7f37580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:31.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.261+0000 7f37717e0640 1 --2- 172.21.15.116:0/576194431 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37580a2800 0x7f37580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:31.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.261+0000 7f37627fc640 1 -- 172.21.15.116:0/576194431 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7f37640ce900 con 0x7f376c1022d0 2024-03-19T01:10:31.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.261+0000 7f37717e0640 1 --2- 172.21.15.116:0/576194431 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37580a2800 0x7f37580a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f376802c700 tx=0x7f3768032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:31.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.265+0000 7f37627fc640 1 -- 172.21.15.116:0/576194431 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3764014030 con 0x7f376c1022d0 2024-03-19T01:10:31.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:31 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3209830266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:10:31.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:31 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:10:31.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:31 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/686364406' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-19T01:10:31.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:31 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2187272265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T01:10:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:31 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3209830266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:10:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:31 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:10:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:31 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/686364406' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-19T01:10:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:31 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2187272265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T01:10:31.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:31.445+0000 7f3773a44640 1 -- 172.21.15.116:0/576194431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f376c02d1f0 con 0x7f376c1022d0 2024-03-19T01:10:31.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:31 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3209830266' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:10:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:31 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:10:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:31 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/686364406' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-19T01:10:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:31 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2187272265' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-19T01:10:32.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.131+0000 7f37627fc640 1 -- 172.21.15.116:0/576194431 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v83) ==== 170+0+0 (secure 0 0 0) 0x7f376409bcd0 con 0x7f376c1022d0 2024-03-19T01:10:32.132 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' created 2024-03-19T01:10:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.133+0000 7f3773a44640 1 -- 172.21.15.116:0/576194431 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37580a2800 msgr2=0x7f37580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:32.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.133+0000 7f3773a44640 1 --2- 172.21.15.116:0/576194431 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37580a2800 0x7f37580a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f376802c700 tx=0x7f3768032590 comp rx=0 tx=0).stop 2024-03-19T01:10:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.133+0000 7f3773a44640 1 -- 172.21.15.116:0/576194431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c1022d0 msgr2=0x7f376c19f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.133+0000 7f3773a44640 1 --2- 172.21.15.116:0/576194431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c1022d0 0x7f376c19f110 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f376400c970 tx=0x7f376400ce30 comp rx=0 tx=0).stop 2024-03-19T01:10:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.133+0000 7f3773a44640 1 -- 172.21.15.116:0/576194431 shutdown_connections 2024-03-19T01:10:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.133+0000 7f3773a44640 1 --2- 172.21.15.116:0/576194431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f376c1022d0 0x7f376c19f110 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.133+0000 7f3773a44640 1 --2- 172.21.15.116:0/576194431 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f376c101910 0x7f376c1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.133+0000 7f3773a44640 1 --2- 172.21.15.116:0/576194431 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37580a2800 0x7f37580a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:32.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.133+0000 7f3773a44640 1 --2- 172.21.15.116:0/576194431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f376c100720 0x7f376c197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:32.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.133+0000 7f3773a44640 1 -- 172.21.15.116:0/576194431 >> 172.21.15.116:0/576194431 conn(0x7f376c0fbcb0 msgr2=0x7f376c104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:32.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.134+0000 7f3773a44640 1 -- 172.21.15.116:0/576194431 shutdown_connections 2024-03-19T01:10:32.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.134+0000 7f3773a44640 1 -- 172.21.15.116:0/576194431 wait complete. 2024-03-19T01:10:32.159 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-19T01:10:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.263+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/3295845654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a44104bf0 msgr2=0x7f2a44106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.263+0000 7f2a4ac9a640 1 --2- 172.21.15.116:0/3295845654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a44104bf0 0x7f2a44106fe0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2a30009900 tx=0x7f2a3002bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.263+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/3295845654 shutdown_connections 2024-03-19T01:10:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.263+0000 7f2a4ac9a640 1 --2- 172.21.15.116:0/3295845654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a44107520 0x7f2a44109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.263+0000 7f2a4ac9a640 1 --2- 172.21.15.116:0/3295845654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a44104bf0 0x7f2a44106fe0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.263+0000 7f2a4ac9a640 1 --2- 172.21.15.116:0/3295845654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a441022c0 0x7f2a441046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:32.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.263+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/3295845654 >> 172.21.15.116:0/3295845654 conn(0x7f2a440fbcd0 msgr2=0x7f2a440fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.263+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/3295845654 shutdown_connections 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.263+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/3295845654 wait complete. 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.263+0000 7f2a4ac9a640 1 Processor -- start 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.263+0000 7f2a4ac9a640 1 -- start start 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a4ac9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a441022c0 0x7f2a44197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a4ac9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a44104bf0 0x7f2a441980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a4ac9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a44107520 0x7f2a4419f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a4ac9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4410d4c0 con 0x7f2a44104bf0 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a4ac9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4410d340 con 0x7f2a44107520 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a4ac9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4410d640 con 0x7f2a441022c0 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a43fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a44104bf0 0x7f2a441980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a43fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a44104bf0 0x7f2a441980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35986/0 (socket says 172.21.15.116:35986) 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a43fff640 1 -- 172.21.15.116:0/1636679045 learned_addr learned my addr 172.21.15.116:0/1636679045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:32.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a48a36640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a441022c0 0x7f2a44197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a49237640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a44107520 0x7f2a4419f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a43fff640 1 -- 172.21.15.116:0/1636679045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a441022c0 msgr2=0x7f2a44197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a43fff640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a441022c0 0x7f2a44197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a43fff640 1 -- 172.21.15.116:0/1636679045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a44107520 msgr2=0x7f2a4419f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a43fff640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a44107520 0x7f2a4419f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a43fff640 1 -- 172.21.15.116:0/1636679045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4419f7b0 con 0x7f2a44104bf0 2024-03-19T01:10:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a43fff640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a44104bf0 0x7f2a441980d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2a30000c00 tx=0x7f2a3002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a41ffb640 1 -- 172.21.15.116:0/1636679045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a30041070 con 0x7f2a44104bf0 2024-03-19T01:10:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a41ffb640 1 -- 172.21.15.116:0/1636679045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a30004450 con 0x7f2a44104bf0 2024-03-19T01:10:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/1636679045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a4419fa40 con 0x7f2a44104bf0 2024-03-19T01:10:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.264+0000 7f2a41ffb640 1 -- 172.21.15.116:0/1636679045 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a30034440 con 0x7f2a44104bf0 2024-03-19T01:10:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.265+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/1636679045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a4419ff80 con 0x7f2a44104bf0 2024-03-19T01:10:32.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.265+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/1636679045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a440fd7c0 con 0x7f2a44104bf0 2024-03-19T01:10:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.267+0000 7f2a41ffb640 1 -- 172.21.15.116:0/1636679045 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f2a30044070 con 0x7f2a44104bf0 2024-03-19T01:10:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.267+0000 7f2a41ffb640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2a340a2880 0x7f2a340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.268+0000 7f2a48a36640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2a340a2880 0x7f2a340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.268+0000 7f2a41ffb640 1 -- 172.21.15.116:0/1636679045 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7f2a300f0500 con 0x7f2a44104bf0 2024-03-19T01:10:32.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.268+0000 7f2a48a36640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2a340a2880 0x7f2a340a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2a38009bd0 tx=0x7f2a38005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:32.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.272+0000 7f2a41ffb640 1 -- 172.21.15.116:0/1636679045 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2a300bd190 con 0x7f2a44104bf0 2024-03-19T01:10:32.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:32.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2187272265' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-19T01:10:32.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: osdmap e82: 12 total, 12 up, 12 in 2024-03-19T01:10:32.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/576194431' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T01:10:32.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:32.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:10:32.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:32.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2187272265' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-19T01:10:32.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: osdmap e82: 12 total, 12 up, 12 in 2024-03-19T01:10:32.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/576194431' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T01:10:32.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:32.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:10:32.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2187272265' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-19T01:10:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: osdmap e82: 12 total, 12 up, 12 in 2024-03-19T01:10:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/576194431' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-19T01:10:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:10:32.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:10:32.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:10:32.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:32.486+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/1636679045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f2a4402d1f0 con 0x7f2a44104bf0 2024-03-19T01:10:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.147+0000 7f2a41ffb640 1 -- 172.21.15.116:0/1636679045 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f2a300c1230 con 0x7f2a44104bf0 2024-03-19T01:10:33.148 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 2 and data pool 3 2024-03-19T01:10:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.150+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/1636679045 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2a340a2880 msgr2=0x7f2a340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.150+0000 7f2a4ac9a640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2a340a2880 0x7f2a340a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2a38009bd0 tx=0x7f2a38005f90 comp rx=0 tx=0).stop 2024-03-19T01:10:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.150+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/1636679045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a44104bf0 msgr2=0x7f2a441980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.150+0000 7f2a4ac9a640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a44104bf0 0x7f2a441980d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2a30000c00 tx=0x7f2a3002c640 comp rx=0 tx=0).stop 2024-03-19T01:10:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.150+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/1636679045 shutdown_connections 2024-03-19T01:10:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.150+0000 7f2a4ac9a640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2a340a2880 0x7f2a340a4d40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.150+0000 7f2a4ac9a640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a44107520 0x7f2a4419f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.150+0000 7f2a4ac9a640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a44104bf0 0x7f2a441980d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.150+0000 7f2a4ac9a640 1 --2- 172.21.15.116:0/1636679045 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2a441022c0 0x7f2a44197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.150+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/1636679045 >> 172.21.15.116:0/1636679045 conn(0x7f2a440fbcd0 msgr2=0x7f2a44102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:33.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.150+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/1636679045 shutdown_connections 2024-03-19T01:10:33.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.150+0000 7f2a4ac9a640 1 -- 172.21.15.116:0/1636679045 wait complete. 2024-03-19T01:10:33.173 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:10:33.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.282+0000 7f82c4697640 1 -- 172.21.15.116:0/890018152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc1033f0 msgr2=0x7f82bc109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.282+0000 7f82c4697640 1 --2- 172.21.15.116:0/890018152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc1033f0 0x7f82bc109c80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f82b4009ca0 tx=0x7f82b402c0f0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.282+0000 7f82c4697640 1 -- 172.21.15.116:0/890018152 shutdown_connections 2024-03-19T01:10:33.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.282+0000 7f82c4697640 1 --2- 172.21.15.116:0/890018152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc1033f0 0x7f82bc109c80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.282+0000 7f82c4697640 1 --2- 172.21.15.116:0/890018152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc102a30 0x7f82bc102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.282+0000 7f82c4697640 1 --2- 172.21.15.116:0/890018152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82bc101840 0x7f82bc101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.282+0000 7f82c4697640 1 -- 172.21.15.116:0/890018152 >> 172.21.15.116:0/890018152 conn(0x7f82bc0fcdd0 msgr2=0x7f82bc0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:33.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.282+0000 7f82c4697640 1 -- 172.21.15.116:0/890018152 shutdown_connections 2024-03-19T01:10:33.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.282+0000 7f82c4697640 1 -- 172.21.15.116:0/890018152 wait complete. 2024-03-19T01:10:33.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.282+0000 7f82c4697640 1 Processor -- start 2024-03-19T01:10:33.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.282+0000 7f82c4697640 1 -- start start 2024-03-19T01:10:33.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c4697640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82bc101840 0x7f82bc197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:33.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c4697640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc102a30 0x7f82bc198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:33.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c4697640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc1033f0 0x7f82bc19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:33.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c4697640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc10d4c0 con 0x7f82bc1033f0 2024-03-19T01:10:33.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c4697640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc10d340 con 0x7f82bc102a30 2024-03-19T01:10:33.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c4697640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc10d640 con 0x7f82bc101840 2024-03-19T01:10:33.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c2c34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc1033f0 0x7f82bc19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:33.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c2c34640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc1033f0 0x7f82bc19f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36000/0 (socket says 172.21.15.116:36000) 2024-03-19T01:10:33.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c2c34640 1 -- 172.21.15.116:0/1051026610 learned_addr learned my addr 172.21.15.116:0/1051026610 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:33.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c2433640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82bc101840 0x7f82bc197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:33.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c1c32640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc102a30 0x7f82bc198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:33.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c2c34640 1 -- 172.21.15.116:0/1051026610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82bc101840 msgr2=0x7f82bc197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c2c34640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82bc101840 0x7f82bc197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c2c34640 1 -- 172.21.15.116:0/1051026610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc102a30 msgr2=0x7f82bc198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c2c34640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc102a30 0x7f82bc198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c2c34640 1 -- 172.21.15.116:0/1051026610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82bc19f6f0 con 0x7f82bc1033f0 2024-03-19T01:10:33.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c1c32640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc102a30 0x7f82bc198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:10:33.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c2c34640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc1033f0 0x7f82bc19f080 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f82b402c700 tx=0x7f82b4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:33.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82b37fe640 1 -- 172.21.15.116:0/1051026610 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b4039040 con 0x7f82bc1033f0 2024-03-19T01:10:33.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c4697640 1 -- 172.21.15.116:0/1051026610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82bc19f980 con 0x7f82bc1033f0 2024-03-19T01:10:33.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82b37fe640 1 -- 172.21.15.116:0/1051026610 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82b4044070 con 0x7f82bc1033f0 2024-03-19T01:10:33.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82b37fe640 1 -- 172.21.15.116:0/1051026610 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b40042a0 con 0x7f82bc1033f0 2024-03-19T01:10:33.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.283+0000 7f82c4697640 1 -- 172.21.15.116:0/1051026610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82bc19fe70 con 0x7f82bc1033f0 2024-03-19T01:10:33.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.284+0000 7f82c4697640 1 -- 172.21.15.116:0/1051026610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82bc108b60 con 0x7f82bc1033f0 2024-03-19T01:10:33.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.285+0000 7f82b37fe640 1 -- 172.21.15.116:0/1051026610 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f82b4003970 con 0x7f82bc1033f0 2024-03-19T01:10:33.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.286+0000 7f82b37fe640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f82a80a2800 0x7f82a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:33.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.286+0000 7f82c2433640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f82a80a2800 0x7f82a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:33.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.286+0000 7f82b37fe640 1 -- 172.21.15.116:0/1051026610 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f82b40f0e00 con 0x7f82bc1033f0 2024-03-19T01:10:33.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.286+0000 7f82c2433640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f82a80a2800 0x7f82a80a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f82ac005fd0 tx=0x7f82ac00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:33.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.290+0000 7f82b37fe640 1 -- 172.21.15.116:0/1051026610 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f82b40be080 con 0x7f82bc1033f0 2024-03-19T01:10:33.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.467+0000 7f82c4697640 1 -- 172.21.15.116:0/1051026610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82bc062420 con 0x7f82bc1033f0 2024-03-19T01:10:33.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.468+0000 7f82b37fe640 1 -- 172.21.15.116:0/1051026610 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22292 (secure 0 0 0) 0x7f82b40bd980 con 0x7f82bc1033f0 2024-03-19T01:10:33.469 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:33.470 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":84,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:10:32.487231+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:10:33.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.470+0000 7f82c4697640 1 -- 172.21.15.116:0/1051026610 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f82a80a2800 msgr2=0x7f82a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.470+0000 7f82c4697640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f82a80a2800 0x7f82a80a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f82ac005fd0 tx=0x7f82ac00b390 comp rx=0 tx=0).stop 2024-03-19T01:10:33.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.470+0000 7f82c4697640 1 -- 172.21.15.116:0/1051026610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc1033f0 msgr2=0x7f82bc19f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.470+0000 7f82c4697640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc1033f0 0x7f82bc19f080 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f82b402c700 tx=0x7f82b4032f20 comp rx=0 tx=0).stop 2024-03-19T01:10:33.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.470+0000 7f82c4697640 1 -- 172.21.15.116:0/1051026610 shutdown_connections 2024-03-19T01:10:33.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.470+0000 7f82c4697640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82bc1033f0 0x7f82bc19f080 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.471+0000 7f82c4697640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f82a80a2800 0x7f82a80a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.471+0000 7f82c4697640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82bc102a30 0x7f82bc198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.471+0000 7f82c4697640 1 --2- 172.21.15.116:0/1051026610 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82bc101840 0x7f82bc197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.471+0000 7f82c4697640 1 -- 172.21.15.116:0/1051026610 >> 172.21.15.116:0/1051026610 conn(0x7f82bc0fcdd0 msgr2=0x7f82bc105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:33.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.471+0000 7f82c4697640 1 -- 172.21.15.116:0/1051026610 shutdown_connections 2024-03-19T01:10:33.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.471+0000 7f82c4697640 1 -- 172.21.15.116:0/1051026610 wait complete. 2024-03-19T01:10:33.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:33 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/576194431' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-19T01:10:33.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:33 smithi156 ceph-mon[66799]: osdmap e83: 12 total, 12 up, 12 in 2024-03-19T01:10:33.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:33 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1636679045' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T01:10:33.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:33 smithi156 ceph-mon[66799]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-19T01:10:33.481 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:33 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/576194431' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-19T01:10:33.481 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:33 smithi116 ceph-mon[62701]: osdmap e83: 12 total, 12 up, 12 in 2024-03-19T01:10:33.481 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:33 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1636679045' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T01:10:33.481 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:33 smithi116 ceph-mon[62701]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-19T01:10:33.483 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:10:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.583+0000 7fc6da7b9640 1 -- 172.21.15.116:0/857281864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6d4101830 msgr2=0x7fc6d4101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.583+0000 7fc6da7b9640 1 --2- 172.21.15.116:0/857281864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6d4101830 0x7fc6d4101c50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4009900 tx=0x7fc6c402bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.583+0000 7fc6da7b9640 1 -- 172.21.15.116:0/857281864 shutdown_connections 2024-03-19T01:10:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.583+0000 7fc6da7b9640 1 --2- 172.21.15.116:0/857281864 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6d41033e0 0x7fc6d4109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.583+0000 7fc6da7b9640 1 --2- 172.21.15.116:0/857281864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6d4102a20 0x7fc6d4102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.583+0000 7fc6da7b9640 1 --2- 172.21.15.116:0/857281864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6d4101830 0x7fc6d4101c50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.583+0000 7fc6da7b9640 1 -- 172.21.15.116:0/857281864 >> 172.21.15.116:0/857281864 conn(0x7fc6d40fce00 msgr2=0x7fc6d40ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.583+0000 7fc6da7b9640 1 -- 172.21.15.116:0/857281864 shutdown_connections 2024-03-19T01:10:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.583+0000 7fc6da7b9640 1 -- 172.21.15.116:0/857281864 wait complete. 2024-03-19T01:10:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.583+0000 7fc6da7b9640 1 Processor -- start 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.583+0000 7fc6da7b9640 1 -- start start 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6da7b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6d4101830 0x7fc6d4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6da7b9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6d4102a20 0x7fc6d4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6da7b9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6d41033e0 0x7fc6d419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6da7b9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410d4b0 con 0x7fc6d4101830 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6da7b9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410d330 con 0x7fc6d41033e0 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6da7b9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d410d630 con 0x7fc6d4102a20 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6d4101830 0x7fc6d4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6d4101830 0x7fc6d4197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36014/0 (socket says 172.21.15.116:36014) 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6d3fff640 1 -- 172.21.15.116:0/431856879 learned_addr learned my addr 172.21.15.116:0/431856879 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6d8d56640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6d41033e0 0x7fc6d419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6d37fe640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6d4102a20 0x7fc6d4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6d3fff640 1 -- 172.21.15.116:0/431856879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6d4102a20 msgr2=0x7fc6d4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6d3fff640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6d4102a20 0x7fc6d4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6d3fff640 1 -- 172.21.15.116:0/431856879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6d41033e0 msgr2=0x7fc6d419f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6d3fff640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6d41033e0 0x7fc6d419f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6d3fff640 1 -- 172.21.15.116:0/431856879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d419f770 con 0x7fc6d4101830 2024-03-19T01:10:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6d3fff640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6d4101830 0x7fc6d4197b50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4005ab0 tx=0x7fc6c4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6d17fa640 1 -- 172.21.15.116:0/431856879 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4041070 con 0x7fc6d4101830 2024-03-19T01:10:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.584+0000 7fc6d17fa640 1 -- 172.21.15.116:0/431856879 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6c40054e0 con 0x7fc6d4101830 2024-03-19T01:10:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.585+0000 7fc6da7b9640 1 -- 172.21.15.116:0/431856879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d419fa00 con 0x7fc6d4101830 2024-03-19T01:10:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.585+0000 7fc6d17fa640 1 -- 172.21.15.116:0/431856879 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4034440 con 0x7fc6d4101830 2024-03-19T01:10:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.585+0000 7fc6da7b9640 1 -- 172.21.15.116:0/431856879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d419fef0 con 0x7fc6d4101830 2024-03-19T01:10:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.586+0000 7fc6da7b9640 1 -- 172.21.15.116:0/431856879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d4108b50 con 0x7fc6d4101830 2024-03-19T01:10:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.586+0000 7fc6d17fa640 1 -- 172.21.15.116:0/431856879 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fc6c4044070 con 0x7fc6d4101830 2024-03-19T01:10:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.587+0000 7fc6d17fa640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6bc0a2800 0x7fc6bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.587+0000 7fc6d37fe640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6bc0a2800 0x7fc6bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.587+0000 7fc6d17fa640 1 -- 172.21.15.116:0/431856879 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7fc6c40f0550 con 0x7fc6d4101830 2024-03-19T01:10:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.588+0000 7fc6d37fe640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6bc0a2800 0x7fc6bc0a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8005d00 tx=0x7fc6c8005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:33.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.592+0000 7fc6d17fa640 1 -- 172.21.15.116:0/431856879 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc6c40bd740 con 0x7fc6d4101830 2024-03-19T01:10:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:33 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/576194431' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-19T01:10:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:33 smithi162 ceph-mon[67178]: osdmap e83: 12 total, 12 up, 12 in 2024-03-19T01:10:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:33 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1636679045' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-19T01:10:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:33 smithi162 ceph-mon[67178]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-19T01:10:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.767+0000 7fc6da7b9640 1 -- 172.21.15.116:0/431856879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6d402d400 con 0x7fc6d4101830 2024-03-19T01:10:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.769+0000 7fc6d17fa640 1 -- 172.21.15.116:0/431856879 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) ==== 74+0+22292 (secure 0 0 0) 0x7fc6c40bd040 con 0x7fc6d4101830 2024-03-19T01:10:33.770 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:33.770 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":84,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:10:32.487231+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.771+0000 7fc6da7b9640 1 -- 172.21.15.116:0/431856879 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6bc0a2800 msgr2=0x7fc6bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.771+0000 7fc6da7b9640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6bc0a2800 0x7fc6bc0a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8005d00 tx=0x7fc6c8005c50 comp rx=0 tx=0).stop 2024-03-19T01:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.771+0000 7fc6da7b9640 1 -- 172.21.15.116:0/431856879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6d4101830 msgr2=0x7fc6d4197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.771+0000 7fc6da7b9640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6d4101830 0x7fc6d4197b50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4005ab0 tx=0x7fc6c4005dc0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.771+0000 7fc6da7b9640 1 -- 172.21.15.116:0/431856879 shutdown_connections 2024-03-19T01:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.771+0000 7fc6da7b9640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6bc0a2800 0x7fc6bc0a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.771+0000 7fc6da7b9640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6d41033e0 0x7fc6d419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.771+0000 7fc6da7b9640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6d4102a20 0x7fc6d4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.771+0000 7fc6da7b9640 1 --2- 172.21.15.116:0/431856879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6d4101830 0x7fc6d4197b50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.771+0000 7fc6da7b9640 1 -- 172.21.15.116:0/431856879 >> 172.21.15.116:0/431856879 conn(0x7fc6d40fce00 msgr2=0x7fc6d4105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.771+0000 7fc6da7b9640 1 -- 172.21.15.116:0/431856879 shutdown_connections 2024-03-19T01:10:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.771+0000 7fc6da7b9640 1 -- 172.21.15.116:0/431856879 wait complete. 2024-03-19T01:10:33.783 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-19T01:10:33.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.886+0000 7f1c0b097640 1 -- 172.21.15.116:0/309067129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c041022d0 msgr2=0x7f1c04108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.886+0000 7f1c0b097640 1 --2- 172.21.15.116:0/309067129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c041022d0 0x7f1c04108b60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009ca0 tx=0x7f1bfc02c0f0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.886+0000 7f1c0b097640 1 -- 172.21.15.116:0/309067129 shutdown_connections 2024-03-19T01:10:33.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.886+0000 7f1c0b097640 1 --2- 172.21.15.116:0/309067129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c041022d0 0x7f1c04108b60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.886+0000 7f1c0b097640 1 --2- 172.21.15.116:0/309067129 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c04101910 0x7f1c04101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.886+0000 7f1c0b097640 1 --2- 172.21.15.116:0/309067129 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c04100720 0x7f1c04100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.886+0000 7f1c0b097640 1 -- 172.21.15.116:0/309067129 >> 172.21.15.116:0/309067129 conn(0x7f1c040fbcb0 msgr2=0x7f1c040fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:33.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.886+0000 7f1c0b097640 1 -- 172.21.15.116:0/309067129 shutdown_connections 2024-03-19T01:10:33.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.886+0000 7f1c0b097640 1 -- 172.21.15.116:0/309067129 wait complete. 2024-03-19T01:10:33.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.886+0000 7f1c0b097640 1 Processor -- start 2024-03-19T01:10:33.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c0b097640 1 -- start start 2024-03-19T01:10:33.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c0b097640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c04100720 0x7f1c04197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:33.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c0b097640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c04101910 0x7f1c04198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:33.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c0b097640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c041022d0 0x7f1c0419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:33.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c0b097640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0410c400 con 0x7f1c04101910 2024-03-19T01:10:33.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c0b097640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0410c280 con 0x7f1c04100720 2024-03-19T01:10:33.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c0b097640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0410c580 con 0x7f1c041022d0 2024-03-19T01:10:33.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c03fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c04101910 0x7f1c04198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:33.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c03fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c04101910 0x7f1c04198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36034/0 (socket says 172.21.15.116:36034) 2024-03-19T01:10:33.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c03fff640 1 -- 172.21.15.116:0/3790458343 learned_addr learned my addr 172.21.15.116:0/3790458343 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:33.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c08e33640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c04100720 0x7f1c04197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:33.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c09634640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c041022d0 0x7f1c0419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:33.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c03fff640 1 -- 172.21.15.116:0/3790458343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c041022d0 msgr2=0x7f1c0419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c03fff640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c041022d0 0x7f1c0419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c03fff640 1 -- 172.21.15.116:0/3790458343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c04100720 msgr2=0x7f1c04197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:33.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c03fff640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c04100720 0x7f1c04197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:33.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c03fff640 1 -- 172.21.15.116:0/3790458343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0419f6f0 con 0x7f1c04101910 2024-03-19T01:10:33.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c03fff640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c04101910 0x7f1c04198010 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1bf000b4a0 tx=0x7f1bf000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:33.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c01ffb640 1 -- 172.21.15.116:0/3790458343 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0014070 con 0x7f1c04101910 2024-03-19T01:10:33.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c01ffb640 1 -- 172.21.15.116:0/3790458343 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bf0010af0 con 0x7f1c04101910 2024-03-19T01:10:33.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c0b097640 1 -- 172.21.15.116:0/3790458343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c0419f9e0 con 0x7f1c04101910 2024-03-19T01:10:33.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.887+0000 7f1c01ffb640 1 -- 172.21.15.116:0/3790458343 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf0017d10 con 0x7f1c04101910 2024-03-19T01:10:33.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.888+0000 7f1c0b097640 1 -- 172.21.15.116:0/3790458343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c0419ff30 con 0x7f1c04101910 2024-03-19T01:10:33.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.889+0000 7f1c0b097640 1 -- 172.21.15.116:0/3790458343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c041a01c0 con 0x7f1c04101910 2024-03-19T01:10:33.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.889+0000 7f1c01ffb640 1 -- 172.21.15.116:0/3790458343 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f1bf0017070 con 0x7f1c04101910 2024-03-19T01:10:33.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.890+0000 7f1c01ffb640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1bf40a2800 0x7f1bf40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:33.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.890+0000 7f1c08e33640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1bf40a2800 0x7f1bf40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:33.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.890+0000 7f1c01ffb640 1 -- 172.21.15.116:0/3790458343 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f1bf00cdd30 con 0x7f1c04101910 2024-03-19T01:10:33.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.891+0000 7f1c08e33640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1bf40a2800 0x7f1bf40a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80058c0 tx=0x7f1bf8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:33.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:33.895+0000 7f1c01ffb640 1 -- 172.21.15.116:0/3790458343 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1bf009af20 con 0x7f1c04101910 2024-03-19T01:10:34.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.105+0000 7f1c0b097640 1 -- 172.21.15.116:0/3790458343 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f1c041a0390 con 0x7f1c04101910 2024-03-19T01:10:34.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.159+0000 7f1c01ffb640 1 -- 172.21.15.116:0/3790458343 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7f1bf009a820 con 0x7f1c04101910 2024-03-19T01:10:34.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.161+0000 7f1c0b097640 1 -- 172.21.15.116:0/3790458343 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1bf40a2800 msgr2=0x7f1bf40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:34.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.161+0000 7f1c0b097640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1bf40a2800 0x7f1bf40a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80058c0 tx=0x7f1bf8005850 comp rx=0 tx=0).stop 2024-03-19T01:10:34.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.161+0000 7f1c0b097640 1 -- 172.21.15.116:0/3790458343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c04101910 msgr2=0x7f1c04198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:34.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.161+0000 7f1c0b097640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c04101910 0x7f1c04198010 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1bf000b4a0 tx=0x7f1bf000b960 comp rx=0 tx=0).stop 2024-03-19T01:10:34.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.161+0000 7f1c0b097640 1 -- 172.21.15.116:0/3790458343 shutdown_connections 2024-03-19T01:10:34.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.161+0000 7f1c0b097640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c041022d0 0x7f1c0419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.161+0000 7f1c0b097640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c04101910 0x7f1c04198010 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.161+0000 7f1c0b097640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1bf40a2800 0x7f1bf40a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.161+0000 7f1c0b097640 1 --2- 172.21.15.116:0/3790458343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c04100720 0x7f1c04197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.161+0000 7f1c0b097640 1 -- 172.21.15.116:0/3790458343 >> 172.21.15.116:0/3790458343 conn(0x7f1c040fbcb0 msgr2=0x7f1c040fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:34.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.161+0000 7f1c0b097640 1 -- 172.21.15.116:0/3790458343 shutdown_connections 2024-03-19T01:10:34.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.161+0000 7f1c0b097640 1 -- 172.21.15.116:0/3790458343 wait complete. 2024-03-19T01:10:34.173 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-19T01:10:34.173 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-19T01:10:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.277+0000 7ff064a27640 1 -- 172.21.15.116:0/3309171618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0601022f0 msgr2=0x7ff060108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.277+0000 7ff064a27640 1 --2- 172.21.15.116:0/3309171618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0601022f0 0x7ff060108b80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff058009ca0 tx=0x7ff05802c0f0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.277+0000 7ff064a27640 1 -- 172.21.15.116:0/3309171618 shutdown_connections 2024-03-19T01:10:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.277+0000 7ff064a27640 1 --2- 172.21.15.116:0/3309171618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0601022f0 0x7ff060108b80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.277+0000 7ff064a27640 1 --2- 172.21.15.116:0/3309171618 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff060101930 0x7ff060101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.277+0000 7ff064a27640 1 --2- 172.21.15.116:0/3309171618 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff060100740 0x7ff060100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.277+0000 7ff064a27640 1 -- 172.21.15.116:0/3309171618 >> 172.21.15.116:0/3309171618 conn(0x7ff0600fbcd0 msgr2=0x7ff0600fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.277+0000 7ff064a27640 1 -- 172.21.15.116:0/3309171618 shutdown_connections 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.277+0000 7ff064a27640 1 -- 172.21.15.116:0/3309171618 wait complete. 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.278+0000 7ff064a27640 1 Processor -- start 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.278+0000 7ff064a27640 1 -- start start 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.278+0000 7ff064a27640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff060100740 0x7ff060197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.278+0000 7ff064a27640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff060101930 0x7ff060198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.278+0000 7ff064a27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0601022f0 0x7ff06019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.278+0000 7ff064a27640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff06010c3a0 con 0x7ff0601022f0 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.278+0000 7ff064a27640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff06010c220 con 0x7ff060101930 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.278+0000 7ff064a27640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff06010c520 con 0x7ff060100740 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.278+0000 7ff05ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0601022f0 0x7ff06019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:34.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.278+0000 7ff05ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0601022f0 0x7ff06019f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36052/0 (socket says 172.21.15.116:36052) 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.278+0000 7ff05ed9d640 1 -- 172.21.15.116:0/3329212872 learned_addr learned my addr 172.21.15.116:0/3329212872 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.278+0000 7ff05dd9b640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff060101930 0x7ff060198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.278+0000 7ff05e59c640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff060100740 0x7ff060197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.279+0000 7ff05ed9d640 1 -- 172.21.15.116:0/3329212872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff060100740 msgr2=0x7ff060197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.279+0000 7ff05ed9d640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff060100740 0x7ff060197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.279+0000 7ff05ed9d640 1 -- 172.21.15.116:0/3329212872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff060101930 msgr2=0x7ff060198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.279+0000 7ff05ed9d640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff060101930 0x7ff060198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.279+0000 7ff05ed9d640 1 -- 172.21.15.116:0/3329212872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff06019f720 con 0x7ff0601022f0 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.279+0000 7ff05ed9d640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0601022f0 0x7ff06019f0b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff058037040 tx=0x7ff058032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.279+0000 7ff04f7fe640 1 -- 172.21.15.116:0/3329212872 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff058041020 con 0x7ff0601022f0 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.279+0000 7ff04f7fe640 1 -- 172.21.15.116:0/3329212872 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff058034d90 con 0x7ff0601022f0 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.279+0000 7ff064a27640 1 -- 172.21.15.116:0/3329212872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff06019f9b0 con 0x7ff0601022f0 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.279+0000 7ff04f7fe640 1 -- 172.21.15.116:0/3329212872 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff058039040 con 0x7ff0601022f0 2024-03-19T01:10:34.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.279+0000 7ff064a27640 1 -- 172.21.15.116:0/3329212872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff06019fea0 con 0x7ff0601022f0 2024-03-19T01:10:34.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.280+0000 7ff064a27640 1 -- 172.21.15.116:0/3329212872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0601a0100 con 0x7ff0601022f0 2024-03-19T01:10:34.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.281+0000 7ff04f7fe640 1 -- 172.21.15.116:0/3329212872 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7ff058044020 con 0x7ff0601022f0 2024-03-19T01:10:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.282+0000 7ff04f7fe640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff0480a2800 0x7ff0480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.282+0000 7ff05e59c640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff0480a2800 0x7ff0480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.282+0000 7ff04f7fe640 1 -- 172.21.15.116:0/3329212872 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7ff0580f07d0 con 0x7ff0601022f0 2024-03-19T01:10:34.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.282+0000 7ff05e59c640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff0480a2800 0x7ff0480a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff050005fd0 tx=0x7ff05000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:34.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.286+0000 7ff04f7fe640 1 -- 172.21.15.116:0/3329212872 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff05804b020 con 0x7ff0601022f0 2024-03-19T01:10:34.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:34 smithi156 ceph-mon[66799]: pgmap v286: 97 pgs: 2 creating+peering, 4 active+clean, 91 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:34 smithi156 ceph-mon[66799]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T01:10:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:34 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1636679045' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-19T01:10:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:34 smithi156 ceph-mon[66799]: osdmap e84: 12 total, 12 up, 12 in 2024-03-19T01:10:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:34 smithi156 ceph-mon[66799]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-19T01:10:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:34 smithi156 ceph-mon[66799]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-19T01:10:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:34 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1051026610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:34 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/431856879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:34 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3790458343' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T01:10:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:34 smithi156 ceph-mon[66799]: osdmap e85: 12 total, 12 up, 12 in 2024-03-19T01:10:34.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.500+0000 7ff064a27640 1 -- 172.21.15.116:0/3329212872 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7ff0601a0490 con 0x7ff0601022f0 2024-03-19T01:10:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:34 smithi162 ceph-mon[67178]: pgmap v286: 97 pgs: 2 creating+peering, 4 active+clean, 91 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:34 smithi162 ceph-mon[67178]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T01:10:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:34 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1636679045' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-19T01:10:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:34 smithi162 ceph-mon[67178]: osdmap e84: 12 total, 12 up, 12 in 2024-03-19T01:10:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:34 smithi162 ceph-mon[67178]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-19T01:10:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:34 smithi162 ceph-mon[67178]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-19T01:10:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:34 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1051026610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:34.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:34 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/431856879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:34.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:34 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3790458343' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T01:10:34.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:34 smithi162 ceph-mon[67178]: osdmap e85: 12 total, 12 up, 12 in 2024-03-19T01:10:34.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:34 smithi116 ceph-mon[62701]: pgmap v286: 97 pgs: 2 creating+peering, 4 active+clean, 91 unknown; 577 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-19T01:10:34.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:34 smithi116 ceph-mon[62701]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-19T01:10:34.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:34 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1636679045' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-19T01:10:34.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:34 smithi116 ceph-mon[62701]: osdmap e84: 12 total, 12 up, 12 in 2024-03-19T01:10:34.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:34 smithi116 ceph-mon[62701]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-19T01:10:34.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:34 smithi116 ceph-mon[62701]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-19T01:10:34.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:34 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1051026610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:34.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:34 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/431856879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:34.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:34 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3790458343' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-19T01:10:34.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:34 smithi116 ceph-mon[62701]: osdmap e85: 12 total, 12 up, 12 in 2024-03-19T01:10:34.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.939+0000 7ff04f7fe640 1 -- 172.21.15.116:0/3329212872 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v16) ==== 105+0+0 (secure 0 0 0) 0x7ff0580bd9c0 con 0x7ff0601022f0 2024-03-19T01:10:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.942+0000 7ff064a27640 1 -- 172.21.15.116:0/3329212872 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff0480a2800 msgr2=0x7ff0480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.942+0000 7ff064a27640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff0480a2800 0x7ff0480a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff050005fd0 tx=0x7ff05000b410 comp rx=0 tx=0).stop 2024-03-19T01:10:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.942+0000 7ff064a27640 1 -- 172.21.15.116:0/3329212872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0601022f0 msgr2=0x7ff06019f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.942+0000 7ff064a27640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0601022f0 0x7ff06019f0b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff058037040 tx=0x7ff058032f20 comp rx=0 tx=0).stop 2024-03-19T01:10:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.942+0000 7ff064a27640 1 -- 172.21.15.116:0/3329212872 shutdown_connections 2024-03-19T01:10:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.942+0000 7ff064a27640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0601022f0 0x7ff06019f0b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.942+0000 7ff064a27640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff0480a2800 0x7ff0480a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.942+0000 7ff064a27640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff060101930 0x7ff060198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.942+0000 7ff064a27640 1 --2- 172.21.15.116:0/3329212872 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff060100740 0x7ff060197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.942+0000 7ff064a27640 1 -- 172.21.15.116:0/3329212872 >> 172.21.15.116:0/3329212872 conn(0x7ff0600fbcd0 msgr2=0x7ff0600fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.942+0000 7ff064a27640 1 -- 172.21.15.116:0/3329212872 shutdown_connections 2024-03-19T01:10:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:34.942+0000 7ff064a27640 1 -- 172.21.15.116:0/3329212872 wait complete. 2024-03-19T01:10:34.959 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-19T01:10:35.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.066+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1469489004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e0101920 msgr2=0x7f81e0101da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.066+0000 7f81e6fa9640 1 --2- 172.21.15.116:0/1469489004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e0101920 0x7f81e0101da0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f81d0009ca0 tx=0x7f81d002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.066+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1469489004 shutdown_connections 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.066+0000 7f81e6fa9640 1 --2- 172.21.15.116:0/1469489004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f81e01022e0 0x7f81e0108b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.066+0000 7f81e6fa9640 1 --2- 172.21.15.116:0/1469489004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e0101920 0x7f81e0101da0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.066+0000 7f81e6fa9640 1 --2- 172.21.15.116:0/1469489004 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81e0100730 0x7f81e0100b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.066+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1469489004 >> 172.21.15.116:0/1469489004 conn(0x7f81e00fbcc0 msgr2=0x7f81e00fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.066+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1469489004 shutdown_connections 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.066+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1469489004 wait complete. 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.066+0000 7f81e6fa9640 1 Processor -- start 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.066+0000 7f81e6fa9640 1 -- start start 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e6fa9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f81e0100730 0x7f81e0104790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e6fa9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81e0101920 0x7f81e0102dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e6fa9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e01022e0 0x7f81e0103310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:35.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e6fa9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f81e010c390 con 0x7f81e01022e0 2024-03-19T01:10:35.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e6fa9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f81e010c210 con 0x7f81e0101920 2024-03-19T01:10:35.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e6fa9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f81e010c510 con 0x7f81e0100730 2024-03-19T01:10:35.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e5546640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e01022e0 0x7f81e0103310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:35.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e4d45640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f81e0100730 0x7f81e0104790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:35.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81dffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81e0101920 0x7f81e0102dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:35.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e5546640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e01022e0 0x7f81e0103310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36078/0 (socket says 172.21.15.116:36078) 2024-03-19T01:10:35.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e5546640 1 -- 172.21.15.116:0/1184233861 learned_addr learned my addr 172.21.15.116:0/1184233861 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:35.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e5546640 1 -- 172.21.15.116:0/1184233861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f81e0100730 msgr2=0x7f81e0104790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:35.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e5546640 1 --2- 172.21.15.116:0/1184233861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f81e0100730 0x7f81e0104790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:35.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e5546640 1 -- 172.21.15.116:0/1184233861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81e0101920 msgr2=0x7f81e0102dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:35.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e5546640 1 --2- 172.21.15.116:0/1184233861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81e0101920 0x7f81e0102dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:35.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e5546640 1 -- 172.21.15.116:0/1184233861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e0103ba0 con 0x7f81e01022e0 2024-03-19T01:10:35.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81dffff640 1 --2- 172.21.15.116:0/1184233861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81e0101920 0x7f81e0102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:10:35.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e5546640 1 --2- 172.21.15.116:0/1184233861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e01022e0 0x7f81e0103310 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f81c800c970 tx=0x7f81c800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:35.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81ddffb640 1 -- 172.21.15.116:0/1184233861 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8007bb0 con 0x7f81e01022e0 2024-03-19T01:10:35.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1184233861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e01a1c70 con 0x7f81e01022e0 2024-03-19T01:10:35.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81ddffb640 1 -- 172.21.15.116:0/1184233861 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81c8007d50 con 0x7f81e01022e0 2024-03-19T01:10:35.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.067+0000 7f81ddffb640 1 -- 172.21.15.116:0/1184233861 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c8018800 con 0x7f81e01022e0 2024-03-19T01:10:35.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.068+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1184233861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e01a2190 con 0x7f81e01022e0 2024-03-19T01:10:35.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.069+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1184233861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e0068a60 con 0x7f81e01022e0 2024-03-19T01:10:35.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.069+0000 7f81ddffb640 1 -- 172.21.15.116:0/1184233861 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f81c800fa50 con 0x7f81e01022e0 2024-03-19T01:10:35.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.070+0000 7f81ddffb640 1 --2- 172.21.15.116:0/1184233861 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f81cc0a2800 0x7f81cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:35.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.070+0000 7f81e4d45640 1 --2- 172.21.15.116:0/1184233861 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f81cc0a2800 0x7f81cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:35.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.070+0000 7f81ddffb640 1 -- 172.21.15.116:0/1184233861 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f81c80ce920 con 0x7f81e01022e0 2024-03-19T01:10:35.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.070+0000 7f81e4d45640 1 --2- 172.21.15.116:0/1184233861 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f81cc0a2800 0x7f81cc0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f81e0101780 tx=0x7f81d8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:35.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.074+0000 7f81ddffb640 1 -- 172.21.15.116:0/1184233861 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f81c8014030 con 0x7f81e01022e0 2024-03-19T01:10:35.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.290+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1184233861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f81e01a2570 con 0x7f81e01022e0 2024-03-19T01:10:35.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:35 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3790458343' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-19T01:10:35.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:35 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] up:active 2024-03-19T01:10:35.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:35 smithi116 ceph-mon[62701]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T01:10:35.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:35 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3329212872' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T01:10:35.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:35 smithi116 ceph-mon[62701]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-19T01:10:35.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:35 smithi116 ceph-mon[62701]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T01:10:35.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:35 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3329212872' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-19T01:10:35.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:35 smithi116 ceph-mon[62701]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-19T01:10:35.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:35 smithi116 ceph-mon[62701]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-19T01:10:35.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:35 smithi116 ceph-mon[62701]: pgmap v289: 97 pgs: 14 creating+peering, 54 active+clean, 29 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-03-19T01:10:35.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:35 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3790458343' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-19T01:10:35.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:35 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] up:active 2024-03-19T01:10:35.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:35 smithi156 ceph-mon[66799]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T01:10:35.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:35 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3329212872' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T01:10:35.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:35 smithi156 ceph-mon[66799]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-19T01:10:35.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:35 smithi156 ceph-mon[66799]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T01:10:35.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:35 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3329212872' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-19T01:10:35.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:35 smithi156 ceph-mon[66799]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-19T01:10:35.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:35 smithi156 ceph-mon[66799]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-19T01:10:35.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:35 smithi156 ceph-mon[66799]: pgmap v289: 97 pgs: 14 creating+peering, 54 active+clean, 29 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-03-19T01:10:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:35 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3790458343' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-19T01:10:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:35 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] up:active 2024-03-19T01:10:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:35 smithi162 ceph-mon[67178]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-19T01:10:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:35 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3329212872' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-19T01:10:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:35 smithi162 ceph-mon[67178]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-19T01:10:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:35 smithi162 ceph-mon[67178]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-19T01:10:35.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:35 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3329212872' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-19T01:10:35.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:35 smithi162 ceph-mon[67178]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-19T01:10:35.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:35 smithi162 ceph-mon[67178]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-19T01:10:35.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:35 smithi162 ceph-mon[67178]: pgmap v289: 97 pgs: 14 creating+peering, 54 active+clean, 29 unknown; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-03-19T01:10:35.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.956+0000 7f81ddffb640 1 -- 172.21.15.116:0/1184233861 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v17) ==== 121+0+0 (secure 0 0 0) 0x7f81c809bb10 con 0x7f81e01022e0 2024-03-19T01:10:35.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.960+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1184233861 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f81cc0a2800 msgr2=0x7f81cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:35.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.960+0000 7f81e6fa9640 1 --2- 172.21.15.116:0/1184233861 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f81cc0a2800 0x7f81cc0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f81e0101780 tx=0x7f81d8005f90 comp rx=0 tx=0).stop 2024-03-19T01:10:35.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.960+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1184233861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e01022e0 msgr2=0x7f81e0103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:35.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.960+0000 7f81e6fa9640 1 --2- 172.21.15.116:0/1184233861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e01022e0 0x7f81e0103310 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f81c800c970 tx=0x7f81c800ce30 comp rx=0 tx=0).stop 2024-03-19T01:10:35.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.960+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1184233861 shutdown_connections 2024-03-19T01:10:35.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.960+0000 7f81e6fa9640 1 --2- 172.21.15.116:0/1184233861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e01022e0 0x7f81e0103310 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:35.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.960+0000 7f81e6fa9640 1 --2- 172.21.15.116:0/1184233861 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f81cc0a2800 0x7f81cc0a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:35.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.960+0000 7f81e6fa9640 1 --2- 172.21.15.116:0/1184233861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81e0101920 0x7f81e0102dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:35.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.960+0000 7f81e6fa9640 1 --2- 172.21.15.116:0/1184233861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f81e0100730 0x7f81e0104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:35.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.960+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1184233861 >> 172.21.15.116:0/1184233861 conn(0x7f81e00fbcc0 msgr2=0x7f81e0069df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:35.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.960+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1184233861 shutdown_connections 2024-03-19T01:10:35.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:35.960+0000 7f81e6fa9640 1 -- 172.21.15.116:0/1184233861 wait complete. 2024-03-19T01:10:35.979 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-19T01:10:36.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.093+0000 7fd5f46b5640 1 -- 172.21.15.116:0/3212676609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5ec1022d0 msgr2=0x7fd5ec108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:36.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.093+0000 7fd5f46b5640 1 --2- 172.21.15.116:0/3212676609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5ec1022d0 0x7fd5ec108b60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc009900 tx=0x7fd5dc02bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:36.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.093+0000 7fd5f46b5640 1 -- 172.21.15.116:0/3212676609 shutdown_connections 2024-03-19T01:10:36.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.093+0000 7fd5f46b5640 1 --2- 172.21.15.116:0/3212676609 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5ec1022d0 0x7fd5ec108b60 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:36.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.093+0000 7fd5f46b5640 1 --2- 172.21.15.116:0/3212676609 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5ec101910 0x7fd5ec101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:36.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.093+0000 7fd5f46b5640 1 --2- 172.21.15.116:0/3212676609 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5ec100720 0x7fd5ec100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.093+0000 7fd5f46b5640 1 -- 172.21.15.116:0/3212676609 >> 172.21.15.116:0/3212676609 conn(0x7fd5ec0fbcb0 msgr2=0x7fd5ec0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.093+0000 7fd5f46b5640 1 -- 172.21.15.116:0/3212676609 shutdown_connections 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.093+0000 7fd5f46b5640 1 -- 172.21.15.116:0/3212676609 wait complete. 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f46b5640 1 Processor -- start 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f46b5640 1 -- start start 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f46b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5ec100720 0x7fd5ec196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f46b5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5ec101910 0x7fd5ec196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f46b5640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5ec1022d0 0x7fd5ec19dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f46b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec10c3a0 con 0x7fd5ec100720 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f46b5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec10c220 con 0x7fd5ec101910 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f46b5640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec10c520 con 0x7fd5ec1022d0 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f2451640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5ec100720 0x7fd5ec196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f2451640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5ec100720 0x7fd5ec196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36092/0 (socket says 172.21.15.116:36092) 2024-03-19T01:10:36.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f2451640 1 -- 172.21.15.116:0/4166619171 learned_addr learned my addr 172.21.15.116:0/4166619171 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f1c50640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5ec101910 0x7fd5ec196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f2c52640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5ec1022d0 0x7fd5ec19dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f2451640 1 -- 172.21.15.116:0/4166619171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5ec1022d0 msgr2=0x7fd5ec19dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f2451640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5ec1022d0 0x7fd5ec19dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f2451640 1 -- 172.21.15.116:0/4166619171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5ec101910 msgr2=0x7fd5ec196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f2451640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5ec101910 0x7fd5ec196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f2451640 1 -- 172.21.15.116:0/4166619171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5ec19e630 con 0x7fd5ec100720 2024-03-19T01:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f2c52640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5ec1022d0 0x7fd5ec19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.094+0000 7fd5f2451640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5ec100720 0x7fd5ec196a10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009d20 tx=0x7fd5e400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.095+0000 7fd5eb7fe640 1 -- 172.21.15.116:0/4166619171 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4010630 con 0x7fd5ec100720 2024-03-19T01:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.095+0000 7fd5eb7fe640 1 -- 172.21.15.116:0/4166619171 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5e4010cb0 con 0x7fd5ec100720 2024-03-19T01:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.095+0000 7fd5f46b5640 1 -- 172.21.15.116:0/4166619171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5ec19e920 con 0x7fd5ec100720 2024-03-19T01:10:36.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.095+0000 7fd5eb7fe640 1 -- 172.21.15.116:0/4166619171 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4018de0 con 0x7fd5ec100720 2024-03-19T01:10:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.095+0000 7fd5f46b5640 1 -- 172.21.15.116:0/4166619171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5ec19eec0 con 0x7fd5ec100720 2024-03-19T01:10:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.096+0000 7fd5f46b5640 1 -- 172.21.15.116:0/4166619171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5ec107a40 con 0x7fd5ec100720 2024-03-19T01:10:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.097+0000 7fd5eb7fe640 1 -- 172.21.15.116:0/4166619171 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fd5e401f020 con 0x7fd5ec100720 2024-03-19T01:10:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.098+0000 7fd5eb7fe640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd5d40a2880 0x7fd5d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.098+0000 7fd5f1c50640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd5d40a2880 0x7fd5d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.098+0000 7fd5eb7fe640 1 -- 172.21.15.116:0/4166619171 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fd5e40cf0c0 con 0x7fd5ec100720 2024-03-19T01:10:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.098+0000 7fd5f1c50640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd5d40a2880 0x7fd5d40a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec197f20 tx=0x7fd5e000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:36.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.102+0000 7fd5eb7fe640 1 -- 172.21.15.116:0/4166619171 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd5e409c2b0 con 0x7fd5ec100720 2024-03-19T01:10:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.322+0000 7fd5f46b5640 1 -- 172.21.15.116:0/4166619171 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fd5ec02d1f0 con 0x7fd5ec100720 2024-03-19T01:10:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:36 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1184233861' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T01:10:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:36 smithi116 ceph-mon[62701]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-19T01:10:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:36 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1184233861' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-19T01:10:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:36 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] up:active 2024-03-19T01:10:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:36 smithi116 ceph-mon[62701]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-19T01:10:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:36 smithi116 ceph-mon[62701]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-19T01:10:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:36 smithi116 ceph-mon[62701]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:creating} 9 up:standby 2024-03-19T01:10:36.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:36 smithi116 ceph-mon[62701]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-19T01:10:36.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:36 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1184233861' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T01:10:36.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:36 smithi156 ceph-mon[66799]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-19T01:10:36.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:36 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1184233861' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-19T01:10:36.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:36 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] up:active 2024-03-19T01:10:36.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:36 smithi156 ceph-mon[66799]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-19T01:10:36.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:36 smithi156 ceph-mon[66799]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-19T01:10:36.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:36 smithi156 ceph-mon[66799]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:creating} 9 up:standby 2024-03-19T01:10:36.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:36 smithi156 ceph-mon[66799]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-19T01:10:36.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:36 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1184233861' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-19T01:10:36.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:36 smithi162 ceph-mon[67178]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-19T01:10:36.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:36 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1184233861' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-19T01:10:36.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:36 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] up:active 2024-03-19T01:10:36.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:36 smithi162 ceph-mon[67178]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-19T01:10:36.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:36 smithi162 ceph-mon[67178]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-19T01:10:36.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:36 smithi162 ceph-mon[67178]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:creating} 9 up:standby 2024-03-19T01:10:36.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:36 smithi162 ceph-mon[67178]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-19T01:10:36.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.963+0000 7fd5eb7fe640 1 -- 172.21.15.116:0/4166619171 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v19) ==== 115+0+0 (secure 0 0 0) 0x7fd5e409bbb0 con 0x7fd5ec100720 2024-03-19T01:10:36.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.965+0000 7fd5f46b5640 1 -- 172.21.15.116:0/4166619171 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd5d40a2880 msgr2=0x7fd5d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:36.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.965+0000 7fd5f46b5640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd5d40a2880 0x7fd5d40a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec197f20 tx=0x7fd5e000b390 comp rx=0 tx=0).stop 2024-03-19T01:10:36.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.965+0000 7fd5f46b5640 1 -- 172.21.15.116:0/4166619171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5ec100720 msgr2=0x7fd5ec196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:36.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.965+0000 7fd5f46b5640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5ec100720 0x7fd5ec196a10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009d20 tx=0x7fd5e400c6a0 comp rx=0 tx=0).stop 2024-03-19T01:10:36.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.965+0000 7fd5f46b5640 1 -- 172.21.15.116:0/4166619171 shutdown_connections 2024-03-19T01:10:36.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.965+0000 7fd5f46b5640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5ec1022d0 0x7fd5ec19dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:36.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.965+0000 7fd5f46b5640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd5d40a2880 0x7fd5d40a4d40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:36.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.965+0000 7fd5f46b5640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5ec101910 0x7fd5ec196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:36.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.965+0000 7fd5f46b5640 1 --2- 172.21.15.116:0/4166619171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5ec100720 0x7fd5ec196a10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:36.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.965+0000 7fd5f46b5640 1 -- 172.21.15.116:0/4166619171 >> 172.21.15.116:0/4166619171 conn(0x7fd5ec0fbcb0 msgr2=0x7fd5ec104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:36.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.965+0000 7fd5f46b5640 1 -- 172.21.15.116:0/4166619171 shutdown_connections 2024-03-19T01:10:36.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:36.965+0000 7fd5f46b5640 1 -- 172.21.15.116:0/4166619171 wait complete. 2024-03-19T01:10:36.980 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-19T01:10:36.981 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-19T01:10:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 -- 172.21.15.116:0/1048205526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c101910 msgr2=0x7fb81c101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 --2- 172.21.15.116:0/1048205526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c101910 0x7fb81c101d90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb80c009900 tx=0x7fb80c02bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 -- 172.21.15.116:0/1048205526 shutdown_connections 2024-03-19T01:10:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 --2- 172.21.15.116:0/1048205526 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb81c1022d0 0x7fb81c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 --2- 172.21.15.116:0/1048205526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c101910 0x7fb81c101d90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 --2- 172.21.15.116:0/1048205526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb81c100720 0x7fb81c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 -- 172.21.15.116:0/1048205526 >> 172.21.15.116:0/1048205526 conn(0x7fb81c0fbcb0 msgr2=0x7fb81c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 -- 172.21.15.116:0/1048205526 shutdown_connections 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 -- 172.21.15.116:0/1048205526 wait complete. 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 Processor -- start 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 -- start start 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c100720 0x7fb81c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb81c101910 0x7fb81c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb81c1022d0 0x7fb81c19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c10c3f0 con 0x7fb81c100720 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c10c270 con 0x7fb81c101910 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.086+0000 7fb82420b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb81c10c570 con 0x7fb81c1022d0 2024-03-19T01:10:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb821fa7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c100720 0x7fb81c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb821fa7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c100720 0x7fb81c197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53484/0 (socket says 172.21.15.116:53484) 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb821fa7640 1 -- 172.21.15.116:0/2003663056 learned_addr learned my addr 172.21.15.116:0/2003663056 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb8217a6640 1 --2- 172.21.15.116:0/2003663056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb81c101910 0x7fb81c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb821fa7640 1 -- 172.21.15.116:0/2003663056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb81c1022d0 msgr2=0x7fb81c19f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb821fa7640 1 --2- 172.21.15.116:0/2003663056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb81c1022d0 0x7fb81c19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb821fa7640 1 -- 172.21.15.116:0/2003663056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb81c101910 msgr2=0x7fb81c198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb821fa7640 1 --2- 172.21.15.116:0/2003663056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb81c101910 0x7fb81c198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb821fa7640 1 -- 172.21.15.116:0/2003663056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81c19f770 con 0x7fb81c100720 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb821fa7640 1 --2- 172.21.15.116:0/2003663056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c100720 0x7fb81c197b50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb81800e9c0 tx=0x7fb81800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb812ffd640 1 -- 172.21.15.116:0/2003663056 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81800cd80 con 0x7fb81c100720 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb82420b640 1 -- 172.21.15.116:0/2003663056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb81c19fa60 con 0x7fb81c100720 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb812ffd640 1 -- 172.21.15.116:0/2003663056 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb8180035f0 con 0x7fb81c100720 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb812ffd640 1 -- 172.21.15.116:0/2003663056 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb818010690 con 0x7fb81c100720 2024-03-19T01:10:37.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.087+0000 7fb82420b640 1 -- 172.21.15.116:0/2003663056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb81c19ffd0 con 0x7fb81c100720 2024-03-19T01:10:37.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.088+0000 7fb82420b640 1 -- 172.21.15.116:0/2003663056 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb81c1a0260 con 0x7fb81c100720 2024-03-19T01:10:37.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.089+0000 7fb812ffd640 1 -- 172.21.15.116:0/2003663056 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7fb8180037a0 con 0x7fb81c100720 2024-03-19T01:10:37.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.090+0000 7fb812ffd640 1 --2- 172.21.15.116:0/2003663056 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb8080a2800 0x7fb8080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.090+0000 7fb8217a6640 1 --2- 172.21.15.116:0/2003663056 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb8080a2800 0x7fb8080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.090+0000 7fb812ffd640 1 -- 172.21.15.116:0/2003663056 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fb8180ceda0 con 0x7fb81c100720 2024-03-19T01:10:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.090+0000 7fb8217a6640 1 --2- 172.21.15.116:0/2003663056 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb8080a2800 0x7fb8080a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb80c02c240 tx=0x7fb80c02df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:37.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.094+0000 7fb812ffd640 1 -- 172.21.15.116:0/2003663056 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb81809bf90 con 0x7fb81c100720 2024-03-19T01:10:37.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.340+0000 7fb82420b640 1 -- 172.21.15.116:0/2003663056 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fb81c1a0430 con 0x7fb8080a2800 2024-03-19T01:10:37.346 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:37 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4166619171' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T01:10:37.346 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:37 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4166619171' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-19T01:10:37.346 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:37 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] up:active 2024-03-19T01:10:37.347 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:37 smithi116 ceph-mon[62701]: daemon mds.i assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-19T01:10:37.347 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:37 smithi116 ceph-mon[62701]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 9 up:standby 2024-03-19T01:10:37.347 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:37 smithi116 ceph-mon[62701]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=h=up:active,3=i=up:creating} 8 up:standby 2024-03-19T01:10:37.347 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:37 smithi116 ceph-mon[62701]: daemon mds.i is now active in filesystem cephfs as rank 3 2024-03-19T01:10:37.347 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:37 smithi116 ceph-mon[62701]: pgmap v290: 97 pgs: 14 creating+peering, 83 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 7 op/s 2024-03-19T01:10:37.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:37 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/4166619171' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T01:10:37.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:37 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/4166619171' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-19T01:10:37.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:37 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] up:active 2024-03-19T01:10:37.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:37 smithi156 ceph-mon[66799]: daemon mds.i assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-19T01:10:37.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:37 smithi156 ceph-mon[66799]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 9 up:standby 2024-03-19T01:10:37.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:37 smithi156 ceph-mon[66799]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=h=up:active,3=i=up:creating} 8 up:standby 2024-03-19T01:10:37.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:37 smithi156 ceph-mon[66799]: daemon mds.i is now active in filesystem cephfs as rank 3 2024-03-19T01:10:37.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:37 smithi156 ceph-mon[66799]: pgmap v290: 97 pgs: 14 creating+peering, 83 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 7 op/s 2024-03-19T01:10:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:37 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/4166619171' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-19T01:10:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:37 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/4166619171' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-19T01:10:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:37 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] up:active 2024-03-19T01:10:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:37 smithi162 ceph-mon[67178]: daemon mds.i assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-19T01:10:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:37 smithi162 ceph-mon[67178]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=h=up:active} 9 up:standby 2024-03-19T01:10:37.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:37 smithi162 ceph-mon[67178]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=h=up:active,3=i=up:creating} 8 up:standby 2024-03-19T01:10:37.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:37 smithi162 ceph-mon[67178]: daemon mds.i is now active in filesystem cephfs as rank 3 2024-03-19T01:10:37.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:37 smithi162 ceph-mon[67178]: pgmap v290: 97 pgs: 14 creating+peering, 83 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 7 op/s 2024-03-19T01:10:37.655 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:10:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:10:37.344+0000 7f5c7beab700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T01:10:37.655 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:10:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:10:37.344+0000 7f5c7beab700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T01:10:37.655 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:10:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:10:37.344+0000 7f5c7beab700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T01:10:37.655 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:10:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:10:37.344+0000 7f5c7beab700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T01:10:37.655 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:10:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:10:37.344+0000 7f5c7beab700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-19T01:10:37.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.983+0000 7fb812ffd640 1 -- 172.21.15.116:0/2003663056 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb81c1a0430 con 0x7fb8080a2800 2024-03-19T01:10:37.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.986+0000 7fb82420b640 1 -- 172.21.15.116:0/2003663056 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb8080a2800 msgr2=0x7fb8080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:37.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.986+0000 7fb82420b640 1 --2- 172.21.15.116:0/2003663056 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb8080a2800 0x7fb8080a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb80c02c240 tx=0x7fb80c02df90 comp rx=0 tx=0).stop 2024-03-19T01:10:37.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.986+0000 7fb82420b640 1 -- 172.21.15.116:0/2003663056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c100720 msgr2=0x7fb81c197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:37.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.986+0000 7fb82420b640 1 --2- 172.21.15.116:0/2003663056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c100720 0x7fb81c197b50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb81800e9c0 tx=0x7fb81800ee80 comp rx=0 tx=0).stop 2024-03-19T01:10:37.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.986+0000 7fb82420b640 1 -- 172.21.15.116:0/2003663056 shutdown_connections 2024-03-19T01:10:37.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.986+0000 7fb82420b640 1 --2- 172.21.15.116:0/2003663056 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb81c1022d0 0x7fb81c19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:37.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.986+0000 7fb82420b640 1 --2- 172.21.15.116:0/2003663056 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb8080a2800 0x7fb8080a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:37.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.986+0000 7fb82420b640 1 --2- 172.21.15.116:0/2003663056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb81c101910 0x7fb81c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:37.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.986+0000 7fb82420b640 1 --2- 172.21.15.116:0/2003663056 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb81c100720 0x7fb81c197b50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:37.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.986+0000 7fb82420b640 1 -- 172.21.15.116:0/2003663056 >> 172.21.15.116:0/2003663056 conn(0x7fb81c0fbcb0 msgr2=0x7fb81c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:37.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.986+0000 7fb82420b640 1 -- 172.21.15.116:0/2003663056 shutdown_connections 2024-03-19T01:10:37.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:37.986+0000 7fb82420b640 1 -- 172.21.15.116:0/2003663056 wait complete. 2024-03-19T01:10:37.999 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-19T01:10:38.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.121+0000 7f37ccadc640 1 -- 172.21.15.116:0/3349441125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37c8104bf0 msgr2=0x7f37c8106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.121+0000 7f37ccadc640 1 --2- 172.21.15.116:0/3349441125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37c8104bf0 0x7f37c8106fe0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009900 tx=0x7f37bc02bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.121+0000 7f37ccadc640 1 -- 172.21.15.116:0/3349441125 shutdown_connections 2024-03-19T01:10:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.121+0000 7f37ccadc640 1 --2- 172.21.15.116:0/3349441125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37c8107520 0x7f37c8109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.121+0000 7f37ccadc640 1 --2- 172.21.15.116:0/3349441125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37c8104bf0 0x7f37c8106fe0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.121+0000 7f37ccadc640 1 --2- 172.21.15.116:0/3349441125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c81022c0 0x7f37c81046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.121+0000 7f37ccadc640 1 -- 172.21.15.116:0/3349441125 >> 172.21.15.116:0/3349441125 conn(0x7f37c80fbcb0 msgr2=0x7f37c80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.121+0000 7f37ccadc640 1 -- 172.21.15.116:0/3349441125 shutdown_connections 2024-03-19T01:10:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.121+0000 7f37ccadc640 1 -- 172.21.15.116:0/3349441125 wait complete. 2024-03-19T01:10:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.121+0000 7f37ccadc640 1 Processor -- start 2024-03-19T01:10:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.121+0000 7f37ccadc640 1 -- start start 2024-03-19T01:10:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37ccadc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c81022c0 0x7f37c8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:38.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37ccadc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37c8104bf0 0x7f37c81980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37ccadc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37c8107520 0x7f37c819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37ccadc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f37c810d4b0 con 0x7f37c8107520 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37ccadc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f37c810d330 con 0x7f37c81022c0 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37ccadc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f37c810d630 con 0x7f37c8104bf0 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37c8107520 0x7f37c819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c81022c0 0x7f37c8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c5d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37c8104bf0 0x7f37c81980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37c8107520 0x7f37c819f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53512/0 (socket says 172.21.15.116:53512) 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c81022c0 0x7f37c8197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:48224/0 (socket says 172.21.15.116:48224) 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c6d9d640 1 -- 172.21.15.116:0/4118248555 learned_addr learned my addr 172.21.15.116:0/4118248555 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c659c640 1 -- 172.21.15.116:0/4118248555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37c8104bf0 msgr2=0x7f37c81980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c659c640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37c8104bf0 0x7f37c81980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c659c640 1 -- 172.21.15.116:0/4118248555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37c8107520 msgr2=0x7f37c819f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c659c640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37c8107520 0x7f37c819f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c659c640 1 -- 172.21.15.116:0/4118248555 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c819f780 con 0x7f37c81022c0 2024-03-19T01:10:38.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c5d9b640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37c8104bf0 0x7f37c81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:10:38.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c6d9d640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37c8107520 0x7f37c819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:10:38.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37c659c640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c81022c0 0x7f37c8197b60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f37c000eb00 tx=0x7f37c000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:38.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37b77fe640 1 -- 172.21.15.116:0/4118248555 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c0010430 con 0x7f37c81022c0 2024-03-19T01:10:38.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37ccadc640 1 -- 172.21.15.116:0/4118248555 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37c819fa70 con 0x7f37c81022c0 2024-03-19T01:10:38.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37b77fe640 1 -- 172.21.15.116:0/4118248555 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37c0010ab0 con 0x7f37c81022c0 2024-03-19T01:10:38.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37b77fe640 1 -- 172.21.15.116:0/4118248555 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c0017bb0 con 0x7f37c81022c0 2024-03-19T01:10:38.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.122+0000 7f37ccadc640 1 -- 172.21.15.116:0/4118248555 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37c819ffe0 con 0x7f37c81022c0 2024-03-19T01:10:38.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.123+0000 7f37ccadc640 1 -- 172.21.15.116:0/4118248555 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37c80fd7a0 con 0x7f37c81022c0 2024-03-19T01:10:38.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.125+0000 7f37b77fe640 1 -- 172.21.15.116:0/4118248555 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f37c0010c60 con 0x7f37c81022c0 2024-03-19T01:10:38.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.125+0000 7f37b77fe640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37b00a2800 0x7f37b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:38.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.125+0000 7f37c5d9b640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37b00a2800 0x7f37b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:38.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.125+0000 7f37b77fe640 1 -- 172.21.15.116:0/4118248555 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f37c00cf4e0 con 0x7f37c81022c0 2024-03-19T01:10:38.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.126+0000 7f37c5d9b640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37b00a2800 0x7f37b00a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f37bc000c00 tx=0x7f37bc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:38.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.133+0000 7f37b77fe640 1 -- 172.21.15.116:0/4118248555 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f37c0053740 con 0x7f37c81022c0 2024-03-19T01:10:38.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.347+0000 7f37ccadc640 1 -- 172.21.15.116:0/4118248555 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f37c8061c60 con 0x7f37b00a2800 2024-03-19T01:10:38.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.367+0000 7f37b77fe640 1 -- 172.21.15.116:0/4118248555 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f37c8061c60 con 0x7f37b00a2800 2024-03-19T01:10:38.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.369+0000 7f37ccadc640 1 -- 172.21.15.116:0/4118248555 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37b00a2800 msgr2=0x7f37b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:38.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.369+0000 7f37ccadc640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37b00a2800 0x7f37b00a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f37bc000c00 tx=0x7f37bc005fd0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.369+0000 7f37ccadc640 1 -- 172.21.15.116:0/4118248555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c81022c0 msgr2=0x7f37c8197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:38.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.369+0000 7f37ccadc640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c81022c0 0x7f37c8197b60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f37c000eb00 tx=0x7f37c000c2d0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.369+0000 7f37ccadc640 1 -- 172.21.15.116:0/4118248555 shutdown_connections 2024-03-19T01:10:38.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.369+0000 7f37ccadc640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37c8107520 0x7f37c819f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.369+0000 7f37ccadc640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37c8104bf0 0x7f37c81980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.369+0000 7f37ccadc640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37b00a2800 0x7f37b00a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.369+0000 7f37ccadc640 1 --2- 172.21.15.116:0/4118248555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37c81022c0 0x7f37c8197b60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.369+0000 7f37ccadc640 1 -- 172.21.15.116:0/4118248555 >> 172.21.15.116:0/4118248555 conn(0x7f37c80fbcb0 msgr2=0x7f37c8102ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.369+0000 7f37ccadc640 1 -- 172.21.15.116:0/4118248555 shutdown_connections 2024-03-19T01:10:38.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.369+0000 7f37ccadc640 1 -- 172.21.15.116:0/4118248555 wait complete. 2024-03-19T01:10:38.381 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-19T01:10:38.474 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:10:38 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b[97322]: 2024-03-19T01:10:38.113+0000 7f9afdc5d700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-19T01:10:38.474 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:10:38 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b[97322]: 2024-03-19T01:10:38.113+0000 7f9afdc5d700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-19T01:10:38.474 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:10:38 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b[97322]: 2024-03-19T01:10:38.113+0000 7f9afdc5d700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-19T01:10:38.474 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:10:38 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b[97322]: 2024-03-19T01:10:38.113+0000 7f9afdc5d700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-19T01:10:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.507+0000 7f7d17322640 1 -- 172.21.15.116:0/1576057305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d10101d40 msgr2=0x7f7d10102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.507+0000 7f7d17322640 1 --2- 172.21.15.116:0/1576057305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d10101d40 0x7f7d10102160 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7d00009930 tx=0x7f7d0002bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.507+0000 7f7d17322640 1 -- 172.21.15.116:0/1576057305 shutdown_connections 2024-03-19T01:10:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.507+0000 7f7d17322640 1 --2- 172.21.15.116:0/1576057305 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d101013a0 0x7f7d10108c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.507+0000 7f7d17322640 1 --2- 172.21.15.116:0/1576057305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d101009e0 0x7f7d10100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.507+0000 7f7d17322640 1 --2- 172.21.15.116:0/1576057305 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d10101d40 0x7f7d10102160 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.507+0000 7f7d17322640 1 -- 172.21.15.116:0/1576057305 >> 172.21.15.116:0/1576057305 conn(0x7f7d100fb410 msgr2=0x7f7d100fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.508+0000 7f7d17322640 1 -- 172.21.15.116:0/1576057305 shutdown_connections 2024-03-19T01:10:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.508+0000 7f7d17322640 1 -- 172.21.15.116:0/1576057305 wait complete. 2024-03-19T01:10:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.508+0000 7f7d17322640 1 Processor -- start 2024-03-19T01:10:38.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.508+0000 7f7d17322640 1 -- start start 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.508+0000 7f7d17322640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d101009e0 0x7f7d10196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.508+0000 7f7d17322640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d101013a0 0x7f7d10196fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d17322640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d10101d40 0x7f7d1019e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d17322640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1010c3d0 con 0x7f7d101013a0 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d17322640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1010c250 con 0x7f7d101009e0 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d17322640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1010c550 con 0x7f7d10101d40 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d148bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d101013a0 0x7f7d10196fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d150be640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d101009e0 0x7f7d10196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d148bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d101013a0 0x7f7d10196fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53536/0 (socket says 172.21.15.116:53536) 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d150be640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d101009e0 0x7f7d10196a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:48242/0 (socket says 172.21.15.116:48242) 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d148bd640 1 -- 172.21.15.116:0/2022742462 learned_addr learned my addr 172.21.15.116:0/2022742462 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d158bf640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d10101d40 0x7f7d1019e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d150be640 1 -- 172.21.15.116:0/2022742462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d10101d40 msgr2=0x7f7d1019e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d150be640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d10101d40 0x7f7d1019e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d150be640 1 -- 172.21.15.116:0/2022742462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d101013a0 msgr2=0x7f7d10196fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d150be640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d101013a0 0x7f7d10196fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d150be640 1 -- 172.21.15.116:0/2022742462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1019e6b0 con 0x7f7d101009e0 2024-03-19T01:10:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d148bd640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d101013a0 0x7f7d10196fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:10:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d150be640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d101009e0 0x7f7d10196a90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d00036040 tx=0x7f7d0002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d067fc640 1 -- 172.21.15.116:0/2022742462 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d00041070 con 0x7f7d101009e0 2024-03-19T01:10:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d17322640 1 -- 172.21.15.116:0/2022742462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d1019e940 con 0x7f7d101009e0 2024-03-19T01:10:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d067fc640 1 -- 172.21.15.116:0/2022742462 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d000054e0 con 0x7f7d101009e0 2024-03-19T01:10:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d067fc640 1 -- 172.21.15.116:0/2022742462 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d00003e00 con 0x7f7d101009e0 2024-03-19T01:10:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.509+0000 7f7d17322640 1 -- 172.21.15.116:0/2022742462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d1019ee50 con 0x7f7d101009e0 2024-03-19T01:10:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.510+0000 7f7d17322640 1 -- 172.21.15.116:0/2022742462 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d10107bf0 con 0x7f7d101009e0 2024-03-19T01:10:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.511+0000 7f7d067fc640 1 -- 172.21.15.116:0/2022742462 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f7d00044070 con 0x7f7d101009e0 2024-03-19T01:10:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.512+0000 7f7d067fc640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7cfc0a2800 0x7f7cfc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.512+0000 7f7d148bd640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7cfc0a2800 0x7f7cfc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.512+0000 7f7d067fc640 1 -- 172.21.15.116:0/2022742462 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7d000f08d0 con 0x7f7d101009e0 2024-03-19T01:10:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.512+0000 7f7d148bd640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7cfc0a2800 0x7f7cfc0a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7d10197fa0 tx=0x7f7d0c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.520+0000 7f7d067fc640 1 -- 172.21.15.116:0/2022742462 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7d000bd450 con 0x7f7d101009e0 2024-03-19T01:10:38.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.728+0000 7f7d17322640 1 -- 172.21.15.116:0/2022742462 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f7d10061cf0 con 0x7f7cfc0a2800 2024-03-19T01:10:38.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.732+0000 7f7d067fc640 1 -- 172.21.15.116:0/2022742462 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f7d10061cf0 con 0x7f7cfc0a2800 2024-03-19T01:10:38.733 INFO:teuthology.orchestra.run.smithi116.stdout:/volumes/qa/sv_0/f9aa7170-5394-45d1-a036-f4f64e9b943f 2024-03-19T01:10:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.734+0000 7f7d17322640 1 -- 172.21.15.116:0/2022742462 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7cfc0a2800 msgr2=0x7f7cfc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.734+0000 7f7d17322640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7cfc0a2800 0x7f7cfc0a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7d10197fa0 tx=0x7f7d0c005f90 comp rx=0 tx=0).stop 2024-03-19T01:10:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.734+0000 7f7d17322640 1 -- 172.21.15.116:0/2022742462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d101009e0 msgr2=0x7f7d10196a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.734+0000 7f7d17322640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d101009e0 0x7f7d10196a90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7d00036040 tx=0x7f7d0002c650 comp rx=0 tx=0).stop 2024-03-19T01:10:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.734+0000 7f7d17322640 1 -- 172.21.15.116:0/2022742462 shutdown_connections 2024-03-19T01:10:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.734+0000 7f7d17322640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d10101d40 0x7f7d1019e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.734+0000 7f7d17322640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d101013a0 0x7f7d10196fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.734+0000 7f7d17322640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7cfc0a2800 0x7f7cfc0a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.734+0000 7f7d17322640 1 --2- 172.21.15.116:0/2022742462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d101009e0 0x7f7d10196a90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.734+0000 7f7d17322640 1 -- 172.21.15.116:0/2022742462 >> 172.21.15.116:0/2022742462 conn(0x7f7d100fb410 msgr2=0x7f7d10104dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:38.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.734+0000 7f7d17322640 1 -- 172.21.15.116:0/2022742462 shutdown_connections 2024-03-19T01:10:38.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.734+0000 7f7d17322640 1 -- 172.21.15.116:0/2022742462 wait complete. 2024-03-19T01:10:38.747 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-19T01:10:38.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.879+0000 7f51a5c4b640 1 -- 172.21.15.116:0/3773358657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a0101930 msgr2=0x7f51a0101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:38.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.879+0000 7f51a5c4b640 1 --2- 172.21.15.116:0/3773358657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a0101930 0x7f51a0101db0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5188009900 tx=0x7f518802bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:38.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 -- 172.21.15.116:0/3773358657 shutdown_connections 2024-03-19T01:10:38.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 --2- 172.21.15.116:0/3773358657 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a01022f0 0x7f51a0108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 --2- 172.21.15.116:0/3773358657 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a0101930 0x7f51a0101db0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 --2- 172.21.15.116:0/3773358657 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51a0100740 0x7f51a0100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 -- 172.21.15.116:0/3773358657 >> 172.21.15.116:0/3773358657 conn(0x7f51a00fbcb0 msgr2=0x7f51a00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:38.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 -- 172.21.15.116:0/3773358657 shutdown_connections 2024-03-19T01:10:38.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 -- 172.21.15.116:0/3773358657 wait complete. 2024-03-19T01:10:38.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 Processor -- start 2024-03-19T01:10:38.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 -- start start 2024-03-19T01:10:38.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a0100740 0x7f51a0197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:38.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a0101930 0x7f51a01980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51a01022f0 0x7f51a019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f51a010c3c0 con 0x7f51a0100740 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f51a010c240 con 0x7f51a0101930 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.880+0000 7f51a5c4b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f51a010c540 con 0x7f51a01022f0 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f519f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a0100740 0x7f51a0197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f519f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a0100740 0x7f51a0197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53546/0 (socket says 172.21.15.116:53546) 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f519f7fe640 1 -- 172.21.15.116:0/2843513297 learned_addr learned my addr 172.21.15.116:0/2843513297 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f5197fff640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a0101930 0x7f51a01980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f519ffff640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51a01022f0 0x7f51a019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f519f7fe640 1 -- 172.21.15.116:0/2843513297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51a01022f0 msgr2=0x7f51a019f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f519f7fe640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51a01022f0 0x7f51a019f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f519f7fe640 1 -- 172.21.15.116:0/2843513297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a0101930 msgr2=0x7f51a01980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f519f7fe640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a0101930 0x7f51a01980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f519f7fe640 1 -- 172.21.15.116:0/2843513297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a019f7c0 con 0x7f51a0100740 2024-03-19T01:10:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f519f7fe640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a0100740 0x7f51a0197ba0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f519000e9c0 tx=0x7f519000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f519d7fa640 1 -- 172.21.15.116:0/2843513297 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519000ccc0 con 0x7f51a0100740 2024-03-19T01:10:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f519d7fa640 1 -- 172.21.15.116:0/2843513297 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51900035f0 con 0x7f51a0100740 2024-03-19T01:10:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f51a5c4b640 1 -- 172.21.15.116:0/2843513297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51a019fab0 con 0x7f51a0100740 2024-03-19T01:10:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f519d7fa640 1 -- 172.21.15.116:0/2843513297 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f519000ce60 con 0x7f51a0100740 2024-03-19T01:10:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.881+0000 7f51a5c4b640 1 -- 172.21.15.116:0/2843513297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51a019ff80 con 0x7f51a0100740 2024-03-19T01:10:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.882+0000 7f51a5c4b640 1 -- 172.21.15.116:0/2843513297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51a01a01b0 con 0x7f51a0100740 2024-03-19T01:10:38.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.883+0000 7f519d7fa640 1 -- 172.21.15.116:0/2843513297 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f51900037a0 con 0x7f51a0100740 2024-03-19T01:10:38.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.884+0000 7f519d7fa640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f518c0a2800 0x7f518c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:38.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.884+0000 7f5197fff640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f518c0a2800 0x7f518c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:38.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.884+0000 7f519d7fa640 1 -- 172.21.15.116:0/2843513297 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f51900ced40 con 0x7f51a0100740 2024-03-19T01:10:38.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.884+0000 7f5197fff640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f518c0a2800 0x7f518c0a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f518802c7b0 tx=0x7f5188005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:38.890+0000 7f519d7fa640 1 -- 172.21.15.116:0/2843513297 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f519009bf30 con 0x7f51a0100740 2024-03-19T01:10:39.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:38 smithi116 ceph-mon[62701]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:39.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:38 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] up:active 2024-03-19T01:10:39.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:38 smithi116 ceph-mon[62701]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-19T01:10:39.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:38 smithi116 ceph-mon[62701]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T01:10:39.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:38 smithi116 ceph-mon[62701]: Cluster is now healthy 2024-03-19T01:10:39.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:38 smithi116 ceph-mon[62701]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=h=up:active,3=i=up:active} 8 up:standby 2024-03-19T01:10:39.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:38 smithi116 ceph-mon[62701]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-19T01:10:39.156 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:38 smithi116 ceph-mon[62701]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-03-19T01:10:39.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.181+0000 7f51a5c4b640 1 -- 172.21.15.116:0/2843513297 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f51a00fdee0 con 0x7f518c0a2800 2024-03-19T01:10:39.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.204+0000 7f519d7fa640 1 -- 172.21.15.116:0/2843513297 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f51a00fdee0 con 0x7f518c0a2800 2024-03-19T01:10:39.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.206+0000 7f51a5c4b640 1 -- 172.21.15.116:0/2843513297 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f518c0a2800 msgr2=0x7f518c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:39.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.206+0000 7f51a5c4b640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f518c0a2800 0x7f518c0a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f518802c7b0 tx=0x7f5188005fd0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.206+0000 7f51a5c4b640 1 -- 172.21.15.116:0/2843513297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a0100740 msgr2=0x7f51a0197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:39.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.206+0000 7f51a5c4b640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a0100740 0x7f51a0197ba0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f519000e9c0 tx=0x7f519000edc0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.207+0000 7f51a5c4b640 1 -- 172.21.15.116:0/2843513297 shutdown_connections 2024-03-19T01:10:39.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.207+0000 7f51a5c4b640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f51a01022f0 0x7f51a019f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.207+0000 7f51a5c4b640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f518c0a2800 0x7f518c0a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.207+0000 7f51a5c4b640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51a0101930 0x7f51a01980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.207+0000 7f51a5c4b640 1 --2- 172.21.15.116:0/2843513297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51a0100740 0x7f51a0197ba0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.207+0000 7f51a5c4b640 1 -- 172.21.15.116:0/2843513297 >> 172.21.15.116:0/2843513297 conn(0x7f51a00fbcb0 msgr2=0x7f51a00fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:39.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.207+0000 7f51a5c4b640 1 -- 172.21.15.116:0/2843513297 shutdown_connections 2024-03-19T01:10:39.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.207+0000 7f51a5c4b640 1 -- 172.21.15.116:0/2843513297 wait complete. 2024-03-19T01:10:39.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-19T01:10:39.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:38 smithi156 ceph-mon[66799]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:38 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] up:active 2024-03-19T01:10:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:38 smithi156 ceph-mon[66799]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-19T01:10:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:38 smithi156 ceph-mon[66799]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T01:10:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:38 smithi156 ceph-mon[66799]: Cluster is now healthy 2024-03-19T01:10:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:38 smithi156 ceph-mon[66799]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=h=up:active,3=i=up:active} 8 up:standby 2024-03-19T01:10:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:38 smithi156 ceph-mon[66799]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-19T01:10:39.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:38 smithi156 ceph-mon[66799]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-03-19T01:10:39.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.325+0000 7f39b0436640 1 -- 172.21.15.116:0/610803115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a8100720 msgr2=0x7f39a8100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.325+0000 7f39b0436640 1 --2- 172.21.15.116:0/610803115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a8100720 0x7f39a8100b40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009860 tx=0x7f39a402bb70 comp rx=0 tx=0).stop 2024-03-19T01:10:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.325+0000 7f39b0436640 1 -- 172.21.15.116:0/610803115 shutdown_connections 2024-03-19T01:10:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.325+0000 7f39b0436640 1 --2- 172.21.15.116:0/610803115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39a81022d0 0x7f39a8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.325+0000 7f39b0436640 1 --2- 172.21.15.116:0/610803115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39a8101910 0x7f39a8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.325+0000 7f39b0436640 1 --2- 172.21.15.116:0/610803115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a8100720 0x7f39a8100b40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.325+0000 7f39b0436640 1 -- 172.21.15.116:0/610803115 >> 172.21.15.116:0/610803115 conn(0x7f39a80fbcb0 msgr2=0x7f39a80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.326+0000 7f39b0436640 1 -- 172.21.15.116:0/610803115 shutdown_connections 2024-03-19T01:10:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.326+0000 7f39b0436640 1 -- 172.21.15.116:0/610803115 wait complete. 2024-03-19T01:10:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.326+0000 7f39b0436640 1 Processor -- start 2024-03-19T01:10:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.326+0000 7f39b0436640 1 -- start start 2024-03-19T01:10:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.326+0000 7f39b0436640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a8100720 0x7f39a8196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.326+0000 7f39b0436640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39a8101910 0x7f39a8196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:39.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.326+0000 7f39b0436640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39a81022d0 0x7f39a819e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.326+0000 7f39b0436640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f39a810c370 con 0x7f39a8100720 2024-03-19T01:10:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.326+0000 7f39b0436640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f39a810c1f0 con 0x7f39a8101910 2024-03-19T01:10:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.326+0000 7f39b0436640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f39a810c4f0 con 0x7f39a81022d0 2024-03-19T01:10:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f39ae1d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a8100720 0x7f39a8196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f39ae1d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a8100720 0x7f39a8196a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53576/0 (socket says 172.21.15.116:53576) 2024-03-19T01:10:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f39ae1d2640 1 -- 172.21.15.116:0/1566113276 learned_addr learned my addr 172.21.15.116:0/1566113276 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f39ad9d1640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39a8101910 0x7f39a8196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f39ae9d3640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39a81022d0 0x7f39a819e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f39ae1d2640 1 -- 172.21.15.116:0/1566113276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39a81022d0 msgr2=0x7f39a819e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f39ae1d2640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39a81022d0 0x7f39a819e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f39ae1d2640 1 -- 172.21.15.116:0/1566113276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39a8101910 msgr2=0x7f39a8196fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f39ae1d2640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39a8101910 0x7f39a8196fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f39ae1d2640 1 -- 172.21.15.116:0/1566113276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a819e690 con 0x7f39a8100720 2024-03-19T01:10:39.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f39ae1d2640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a8100720 0x7f39a8196a70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f39a402c0b0 tx=0x7f39a4032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:39.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f399f7fe640 1 -- 172.21.15.116:0/1566113276 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4044020 con 0x7f39a8100720 2024-03-19T01:10:39.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f39b0436640 1 -- 172.21.15.116:0/1566113276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a8068980 con 0x7f39a8100720 2024-03-19T01:10:39.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f399f7fe640 1 -- 172.21.15.116:0/1566113276 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39a402c9f0 con 0x7f39a8100720 2024-03-19T01:10:39.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f399f7fe640 1 -- 172.21.15.116:0/1566113276 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4039040 con 0x7f39a8100720 2024-03-19T01:10:39.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.327+0000 7f39b0436640 1 -- 172.21.15.116:0/1566113276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a8068e70 con 0x7f39a8100720 2024-03-19T01:10:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.328+0000 7f39b0436640 1 -- 172.21.15.116:0/1566113276 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39a8068610 con 0x7f39a8100720 2024-03-19T01:10:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.329+0000 7f399f7fe640 1 -- 172.21.15.116:0/1566113276 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f39a4047020 con 0x7f39a8100720 2024-03-19T01:10:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.330+0000 7f399f7fe640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f39940a2800 0x7f39940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.330+0000 7f39ad9d1640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f39940a2800 0x7f39940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.330+0000 7f399f7fe640 1 -- 172.21.15.116:0/1566113276 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f39a40f2ac0 con 0x7f39a8100720 2024-03-19T01:10:39.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.330+0000 7f39ad9d1640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f39940a2800 0x7f39940a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f39a8197f80 tx=0x7f3998009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:39.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.334+0000 7f399f7fe640 1 -- 172.21.15.116:0/1566113276 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f39a404e020 con 0x7f39a8100720 2024-03-19T01:10:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:38 smithi162 ceph-mon[67178]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:38 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] up:active 2024-03-19T01:10:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:38 smithi162 ceph-mon[67178]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-19T01:10:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:38 smithi162 ceph-mon[67178]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-19T01:10:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:38 smithi162 ceph-mon[67178]: Cluster is now healthy 2024-03-19T01:10:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:38 smithi162 ceph-mon[67178]: fsmap cephfs:4 {0=b=up:active,1=a=up:active,2=h=up:active,3=i=up:active} 8 up:standby 2024-03-19T01:10:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:38 smithi162 ceph-mon[67178]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-03-19T01:10:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:38 smithi162 ceph-mon[67178]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-03-19T01:10:39.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.543+0000 7f39b0436640 1 -- 172.21.15.116:0/1566113276 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f39a80690d0 con 0x7f39940a2800 2024-03-19T01:10:39.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.547+0000 7f399f7fe640 1 -- 172.21.15.116:0/1566113276 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f39a80690d0 con 0x7f39940a2800 2024-03-19T01:10:39.548 INFO:teuthology.orchestra.run.smithi116.stdout:/volumes/qa/sv_1/26db1d39-573c-4af5-8f4a-46f4ed020a58 2024-03-19T01:10:39.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.549+0000 7f39b0436640 1 -- 172.21.15.116:0/1566113276 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f39940a2800 msgr2=0x7f39940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:39.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.549+0000 7f39b0436640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f39940a2800 0x7f39940a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f39a8197f80 tx=0x7f3998009260 comp rx=0 tx=0).stop 2024-03-19T01:10:39.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.549+0000 7f39b0436640 1 -- 172.21.15.116:0/1566113276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a8100720 msgr2=0x7f39a8196a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:39.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.549+0000 7f39b0436640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a8100720 0x7f39a8196a70 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f39a402c0b0 tx=0x7f39a4032430 comp rx=0 tx=0).stop 2024-03-19T01:10:39.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.549+0000 7f39b0436640 1 -- 172.21.15.116:0/1566113276 shutdown_connections 2024-03-19T01:10:39.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.549+0000 7f39b0436640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f39a81022d0 0x7f39a819e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.549+0000 7f39b0436640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f39940a2800 0x7f39940a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.549+0000 7f39b0436640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f39a8101910 0x7f39a8196fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.549+0000 7f39b0436640 1 --2- 172.21.15.116:0/1566113276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39a8100720 0x7f39a8196a70 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.549+0000 7f39b0436640 1 -- 172.21.15.116:0/1566113276 >> 172.21.15.116:0/1566113276 conn(0x7f39a80fbcb0 msgr2=0x7f39a8104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:39.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.549+0000 7f39b0436640 1 -- 172.21.15.116:0/1566113276 shutdown_connections 2024-03-19T01:10:39.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.549+0000 7f39b0436640 1 -- 172.21.15.116:0/1566113276 wait complete. 2024-03-19T01:10:39.561 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:10:39.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.692+0000 7f4cc2e36640 1 -- 172.21.15.116:0/1230189269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cbc104cd0 msgr2=0x7f4cbc1070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.692+0000 7f4cc2e36640 1 --2- 172.21.15.116:0/1230189269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cbc104cd0 0x7f4cbc1070c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4ca8009900 tx=0x7f4ca802bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.692+0000 7f4cc2e36640 1 -- 172.21.15.116:0/1230189269 shutdown_connections 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.692+0000 7f4cc2e36640 1 --2- 172.21.15.116:0/1230189269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cbc107600 0x7f4cbc1099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.692+0000 7f4cc2e36640 1 --2- 172.21.15.116:0/1230189269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cbc104cd0 0x7f4cbc1070c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.692+0000 7f4cc2e36640 1 --2- 172.21.15.116:0/1230189269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc1023a0 0x7f4cbc104790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.692+0000 7f4cc2e36640 1 -- 172.21.15.116:0/1230189269 >> 172.21.15.116:0/1230189269 conn(0x7f4cbc0fbcd0 msgr2=0x7f4cbc0fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.692+0000 7f4cc2e36640 1 -- 172.21.15.116:0/1230189269 shutdown_connections 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.692+0000 7f4cc2e36640 1 -- 172.21.15.116:0/1230189269 wait complete. 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.693+0000 7f4cc2e36640 1 Processor -- start 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.693+0000 7f4cc2e36640 1 -- start start 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.693+0000 7f4cc2e36640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cbc1023a0 0x7f4cbc197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.693+0000 7f4cc2e36640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cbc104cd0 0x7f4cbc198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.693+0000 7f4cc2e36640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc107600 0x7f4cbc19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:39.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.693+0000 7f4cc2e36640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc10d4c0 con 0x7f4cbc107600 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.693+0000 7f4cc2e36640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc10d340 con 0x7f4cbc1023a0 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.693+0000 7f4cc2e36640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4cbc10d640 con 0x7f4cbc104cd0 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.693+0000 7f4cc13d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc107600 0x7f4cbc19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.693+0000 7f4cc13d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc107600 0x7f4cbc19f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53592/0 (socket says 172.21.15.116:53592) 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.693+0000 7f4cc13d3640 1 -- 172.21.15.116:0/446702545 learned_addr learned my addr 172.21.15.116:0/446702545 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.693+0000 7f4cc0bd2640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cbc1023a0 0x7f4cbc197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.693+0000 7f4cbbfff640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cbc104cd0 0x7f4cbc198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.694+0000 7f4cc13d3640 1 -- 172.21.15.116:0/446702545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cbc104cd0 msgr2=0x7f4cbc198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.694+0000 7f4cc13d3640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cbc104cd0 0x7f4cbc198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.694+0000 7f4cc13d3640 1 -- 172.21.15.116:0/446702545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cbc1023a0 msgr2=0x7f4cbc197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.694+0000 7f4cc13d3640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cbc1023a0 0x7f4cbc197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.694+0000 7f4cc13d3640 1 -- 172.21.15.116:0/446702545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cbc19f810 con 0x7f4cbc107600 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.694+0000 7f4cc13d3640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc107600 0x7f4cbc19f1a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400efd0 tx=0x7f4cb400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.694+0000 7f4cb9ffb640 1 -- 172.21.15.116:0/446702545 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4003b70 con 0x7f4cbc107600 2024-03-19T01:10:39.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.694+0000 7f4cb9ffb640 1 -- 172.21.15.116:0/446702545 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cb4003d10 con 0x7f4cbc107600 2024-03-19T01:10:39.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.694+0000 7f4cc2e36640 1 -- 172.21.15.116:0/446702545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cbc19fb00 con 0x7f4cbc107600 2024-03-19T01:10:39.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.694+0000 7f4cb9ffb640 1 -- 172.21.15.116:0/446702545 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb40109f0 con 0x7f4cbc107600 2024-03-19T01:10:39.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.694+0000 7f4cc2e36640 1 -- 172.21.15.116:0/446702545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cbc19ffd0 con 0x7f4cbc107600 2024-03-19T01:10:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.695+0000 7f4cc2e36640 1 -- 172.21.15.116:0/446702545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cbc0fd7c0 con 0x7f4cbc107600 2024-03-19T01:10:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.697+0000 7f4cb9ffb640 1 -- 172.21.15.116:0/446702545 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 16) ==== 137291+0+0 (secure 0 0 0) 0x7f4cb4004020 con 0x7f4cbc107600 2024-03-19T01:10:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.697+0000 7f4cb9ffb640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4cac0a2800 0x7f4cac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.697+0000 7f4cb9ffb640 1 -- 172.21.15.116:0/446702545 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f4cb40ced80 con 0x7f4cbc107600 2024-03-19T01:10:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.697+0000 7f4cc0bd2640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4cac0a2800 0x7f4cac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.698+0000 7f4cc0bd2640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4cac0a2800 0x7f4cac0a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0009bd0 tx=0x7f4cb0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:39.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.701+0000 7f4cb9ffb640 1 -- 172.21.15.116:0/446702545 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4cb409bf70 con 0x7f4cbc107600 2024-03-19T01:10:39.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.920+0000 7f4cc2e36640 1 -- 172.21.15.116:0/446702545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cbc02d1f0 con 0x7f4cbc107600 2024-03-19T01:10:39.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.921+0000 7f4cb9ffb640 1 -- 172.21.15.116:0/446702545 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11938 (secure 0 0 0) 0x7f4cb409b870 con 0x7f4cbc107600 2024-03-19T01:10:39.921 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:39.922 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T01:10:32.487607+0000","modified":"2024-03-19T01:10:38.982004+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24488,"mds_2":24473,"mds_3":24460,"mds_4":24445},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1242639081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1242639081},{"type":"v1","addr":"172.21.15.116:6839","nonce":1242639081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24394":{"gid":24394,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":3,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.162:6839/441486392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":441486392},{"type":"v1","addr":"172.21.15.162:6839","nonce":441486392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"a","rank":1,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6841/3343188504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3343188504},{"type":"v1","addr":"172.21.15.116:6841","nonce":3343188504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1601537427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1601537427},{"type":"v1","addr":"172.21.15.156:6839","nonce":1601537427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24473,24445,24500,24460,24488]},"id":1}]} 2024-03-19T01:10:39.922 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2024-03-19T01:10:39.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.923+0000 7f4cc2e36640 1 -- 172.21.15.116:0/446702545 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4cac0a2800 msgr2=0x7f4cac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:39.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.923+0000 7f4cc2e36640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4cac0a2800 0x7f4cac0a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4cb0009bd0 tx=0x7f4cb0005f90 comp rx=0 tx=0).stop 2024-03-19T01:10:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.923+0000 7f4cc2e36640 1 -- 172.21.15.116:0/446702545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc107600 msgr2=0x7f4cbc19f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.923+0000 7f4cc2e36640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc107600 0x7f4cbc19f1a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4cb400efd0 tx=0x7f4cb400c6a0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.923+0000 7f4cc2e36640 1 -- 172.21.15.116:0/446702545 shutdown_connections 2024-03-19T01:10:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.923+0000 7f4cc2e36640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cbc107600 0x7f4cbc19f1a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.923+0000 7f4cc2e36640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4cbc104cd0 0x7f4cbc198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.923+0000 7f4cc2e36640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4cac0a2800 0x7f4cac0a4cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.923+0000 7f4cc2e36640 1 --2- 172.21.15.116:0/446702545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4cbc1023a0 0x7f4cbc197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.924+0000 7f4cc2e36640 1 -- 172.21.15.116:0/446702545 >> 172.21.15.116:0/446702545 conn(0x7f4cbc0fbcd0 msgr2=0x7f4cbc102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.924+0000 7f4cc2e36640 1 -- 172.21.15.116:0/446702545 shutdown_connections 2024-03-19T01:10:39.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:39.924+0000 7f4cc2e36640 1 -- 172.21.15.116:0/446702545 wait complete. 2024-03-19T01:10:39.935 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:10:40.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.042+0000 7f8aaf427640 1 -- 172.21.15.116:0/2866040510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aa8100740 msgr2=0x7f8aa8100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.042+0000 7f8aaf427640 1 --2- 172.21.15.116:0/2866040510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aa8100740 0x7f8aa8100b60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009ca0 tx=0x7f8aa002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.042+0000 7f8aaf427640 1 -- 172.21.15.116:0/2866040510 shutdown_connections 2024-03-19T01:10:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.042+0000 7f8aaf427640 1 --2- 172.21.15.116:0/2866040510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa81022f0 0x7f8aa8108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.042+0000 7f8aaf427640 1 --2- 172.21.15.116:0/2866040510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8aa8101930 0x7f8aa8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.042+0000 7f8aaf427640 1 --2- 172.21.15.116:0/2866040510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aa8100740 0x7f8aa8100b60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.042+0000 7f8aaf427640 1 -- 172.21.15.116:0/2866040510 >> 172.21.15.116:0/2866040510 conn(0x7f8aa80fbcb0 msgr2=0x7f8aa80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aaf427640 1 -- 172.21.15.116:0/2866040510 shutdown_connections 2024-03-19T01:10:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aaf427640 1 -- 172.21.15.116:0/2866040510 wait complete. 2024-03-19T01:10:40.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aaf427640 1 Processor -- start 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aaf427640 1 -- start start 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aaf427640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa8100740 0x7f8aa80694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aaf427640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aa8101930 0x7f8aa819ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aaf427640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8aa81022f0 0x7f8aa81a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aaf427640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa810c3c0 con 0x7f8aa8101930 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aaf427640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa810c240 con 0x7f8aa81022f0 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aaf427640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa810c540 con 0x7f8aa8100740 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aac9c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aa8101930 0x7f8aa819ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aac9c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aa8101930 0x7f8aa819ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53600/0 (socket says 172.21.15.116:53600) 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aac9c2640 1 -- 172.21.15.116:0/3684548049 learned_addr learned my addr 172.21.15.116:0/3684548049 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aad9c4640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8aa81022f0 0x7f8aa81a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.043+0000 7f8aad1c3640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa8100740 0x7f8aa80694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.044+0000 7f8aac9c2640 1 -- 172.21.15.116:0/3684548049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa8100740 msgr2=0x7f8aa80694f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.044+0000 7f8aac9c2640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa8100740 0x7f8aa80694f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.044+0000 7f8aac9c2640 1 -- 172.21.15.116:0/3684548049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8aa81022f0 msgr2=0x7f8aa81a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.044+0000 7f8aac9c2640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8aa81022f0 0x7f8aa81a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.044+0000 7f8aac9c2640 1 -- 172.21.15.116:0/3684548049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa81a19c0 con 0x7f8aa8101930 2024-03-19T01:10:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.044+0000 7f8aac9c2640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aa8101930 0x7f8aa819ec80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800ea60 tx=0x7f8a9800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.044+0000 7f8a9e7fc640 1 -- 172.21.15.116:0/3684548049 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9800cde0 con 0x7f8aa8101930 2024-03-19T01:10:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.044+0000 7f8a9e7fc640 1 -- 172.21.15.116:0/3684548049 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a980035f0 con 0x7f8aa8101930 2024-03-19T01:10:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.044+0000 7f8aaf427640 1 -- 172.21.15.116:0/3684548049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa81a1c50 con 0x7f8aa8101930 2024-03-19T01:10:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.044+0000 7f8a9e7fc640 1 -- 172.21.15.116:0/3684548049 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a98010670 con 0x7f8aa8101930 2024-03-19T01:10:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.044+0000 7f8aaf427640 1 -- 172.21.15.116:0/3684548049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aa81a21a0 con 0x7f8aa8101930 2024-03-19T01:10:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.045+0000 7f8aaf427640 1 -- 172.21.15.116:0/3684548049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aa8107a60 con 0x7f8aa8101930 2024-03-19T01:10:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.047+0000 7f8a9e7fc640 1 -- 172.21.15.116:0/3684548049 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f8a980037a0 con 0x7f8aa8101930 2024-03-19T01:10:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.047+0000 7f8a9e7fc640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8a940a2940 0x7f8a940a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.047+0000 7f8aad1c3640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8a940a2940 0x7f8a940a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.047+0000 7f8a9e7fc640 1 -- 172.21.15.116:0/3684548049 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f8a980cecf0 con 0x7f8aa8101930 2024-03-19T01:10:40.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.048+0000 7f8aad1c3640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8a940a2940 0x7f8a940a4e00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8aa002c700 tx=0x7f8aa0032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:40.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.054+0000 7f8a9e7fc640 1 -- 172.21.15.116:0/3684548049 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8a9809be70 con 0x7f8aa8101930 2024-03-19T01:10:40.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.246+0000 7f8aaf427640 1 -- 172.21.15.116:0/3684548049 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8aa802d3f0 con 0x7f8aa8101930 2024-03-19T01:10:40.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.247+0000 7f8a9e7fc640 1 -- 172.21.15.116:0/3684548049 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22279 (secure 0 0 0) 0x7f8a9809b770 con 0x7f8aa8101930 2024-03-19T01:10:40.248 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:40.248 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":85,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:10:34.147770+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:10:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.249+0000 7f8aaf427640 1 -- 172.21.15.116:0/3684548049 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8a940a2940 msgr2=0x7f8a940a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.249+0000 7f8aaf427640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8a940a2940 0x7f8a940a4e00 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8aa002c700 tx=0x7f8aa0032590 comp rx=0 tx=0).stop 2024-03-19T01:10:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.249+0000 7f8aaf427640 1 -- 172.21.15.116:0/3684548049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aa8101930 msgr2=0x7f8aa819ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.249+0000 7f8aaf427640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aa8101930 0x7f8aa819ec80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800ea60 tx=0x7f8a9800ef20 comp rx=0 tx=0).stop 2024-03-19T01:10:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.249+0000 7f8aaf427640 1 -- 172.21.15.116:0/3684548049 shutdown_connections 2024-03-19T01:10:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.249+0000 7f8aaf427640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8a940a2940 0x7f8a940a4e00 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.249+0000 7f8aaf427640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8aa81022f0 0x7f8aa81a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.249+0000 7f8aaf427640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8aa8101930 0x7f8aa819ec80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.249+0000 7f8aaf427640 1 --2- 172.21.15.116:0/3684548049 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8aa8100740 0x7f8aa80694f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.249+0000 7f8aaf427640 1 -- 172.21.15.116:0/3684548049 >> 172.21.15.116:0/3684548049 conn(0x7f8aa80fbcb0 msgr2=0x7f8aa8104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.250+0000 7f8aaf427640 1 -- 172.21.15.116:0/3684548049 shutdown_connections 2024-03-19T01:10:40.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.250+0000 7f8aaf427640 1 -- 172.21.15.116:0/3684548049 wait complete. 2024-03-19T01:10:40.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:39 smithi116 ceph-mon[62701]: from='client.24581 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:40.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:39 smithi116 ceph-mon[62701]: from='client.24587 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:40.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:39 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] up:active 2024-03-19T01:10:40.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:39 smithi116 ceph-mon[62701]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-19T01:10:40.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:39 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:10:40.260 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:39 smithi116 ceph-mon[62701]: pgmap v291: 97 pgs: 97 active+clean; 590 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s wr, 22 op/s 2024-03-19T01:10:40.261 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:39 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/446702545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:40.262 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-19T01:10:40.262 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:10:40.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.364+0000 7f38b6831640 1 -- 172.21.15.116:0/3569692261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38b0100740 msgr2=0x7f38b0100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.364+0000 7f38b6831640 1 --2- 172.21.15.116:0/3569692261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38b0100740 0x7f38b0100b60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f38a0009900 tx=0x7f38a002bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.364+0000 7f38b6831640 1 -- 172.21.15.116:0/3569692261 shutdown_connections 2024-03-19T01:10:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.364+0000 7f38b6831640 1 --2- 172.21.15.116:0/3569692261 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38b01022f0 0x7f38b0108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.364+0000 7f38b6831640 1 --2- 172.21.15.116:0/3569692261 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b0101930 0x7f38b0101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.364+0000 7f38b6831640 1 --2- 172.21.15.116:0/3569692261 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38b0100740 0x7f38b0100b60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.364+0000 7f38b6831640 1 -- 172.21.15.116:0/3569692261 >> 172.21.15.116:0/3569692261 conn(0x7f38b00fbcb0 msgr2=0x7f38b00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.364+0000 7f38b6831640 1 -- 172.21.15.116:0/3569692261 shutdown_connections 2024-03-19T01:10:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.364+0000 7f38b6831640 1 -- 172.21.15.116:0/3569692261 wait complete. 2024-03-19T01:10:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.365+0000 7f38b6831640 1 Processor -- start 2024-03-19T01:10:40.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.365+0000 7f38b6831640 1 -- start start 2024-03-19T01:10:40.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.365+0000 7f38b6831640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b0100740 0x7f38b0196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:40.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.365+0000 7f38b6831640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38b0101930 0x7f38b0196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:40.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.365+0000 7f38b6831640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38b01022f0 0x7f38b019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:40.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.365+0000 7f38b6831640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f38b010c410 con 0x7f38b01022f0 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.365+0000 7f38b6831640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f38b010c290 con 0x7f38b0100740 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.365+0000 7f38b6831640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f38b010c590 con 0x7f38b0101930 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38b4dce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38b01022f0 0x7f38b019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38b4dce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38b01022f0 0x7f38b019e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53610/0 (socket says 172.21.15.116:53610) 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38affff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b0100740 0x7f38b0196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38b4dce640 1 -- 172.21.15.116:0/4220189912 learned_addr learned my addr 172.21.15.116:0/4220189912 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38af7fe640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38b0101930 0x7f38b0196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38b4dce640 1 -- 172.21.15.116:0/4220189912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38b0101930 msgr2=0x7f38b0196f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38b4dce640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38b0101930 0x7f38b0196f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38b4dce640 1 -- 172.21.15.116:0/4220189912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b0100740 msgr2=0x7f38b0196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38b4dce640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b0100740 0x7f38b0196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38b4dce640 1 -- 172.21.15.116:0/4220189912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b019e670 con 0x7f38b01022f0 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38affff640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b0100740 0x7f38b0196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:10:40.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38b4dce640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38b01022f0 0x7f38b019e000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f38a800eb10 tx=0x7f38a800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:40.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38ad7fa640 1 -- 172.21.15.116:0/4220189912 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a8004b20 con 0x7f38b01022f0 2024-03-19T01:10:40.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38ad7fa640 1 -- 172.21.15.116:0/4220189912 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38a8004cc0 con 0x7f38b01022f0 2024-03-19T01:10:40.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38ad7fa640 1 -- 172.21.15.116:0/4220189912 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a8010760 con 0x7f38b01022f0 2024-03-19T01:10:40.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38b6831640 1 -- 172.21.15.116:0/4220189912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b019e960 con 0x7f38b01022f0 2024-03-19T01:10:40.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.366+0000 7f38b6831640 1 -- 172.21.15.116:0/4220189912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b019ee60 con 0x7f38b01022f0 2024-03-19T01:10:40.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.367+0000 7f38b6831640 1 -- 172.21.15.116:0/4220189912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b0107a60 con 0x7f38b01022f0 2024-03-19T01:10:40.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.369+0000 7f38ad7fa640 1 -- 172.21.15.116:0/4220189912 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f38a8005000 con 0x7f38b01022f0 2024-03-19T01:10:40.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.369+0000 7f38ad7fa640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f38980a2910 0x7f38980a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:40.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.369+0000 7f38affff640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f38980a2910 0x7f38980a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:40.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.369+0000 7f38ad7fa640 1 -- 172.21.15.116:0/4220189912 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f38a80cf410 con 0x7f38b01022f0 2024-03-19T01:10:40.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.370+0000 7f38affff640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f38980a2910 0x7f38980a4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f38a002c170 tx=0x7f38a002df90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:40.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.373+0000 7f38ad7fa640 1 -- 172.21.15.116:0/4220189912 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f38a809c500 con 0x7f38b01022f0 2024-03-19T01:10:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:39 smithi162 ceph-mon[67178]: from='client.24581 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:39 smithi162 ceph-mon[67178]: from='client.24587 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:39 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] up:active 2024-03-19T01:10:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:39 smithi162 ceph-mon[67178]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-19T01:10:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:39 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:10:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:39 smithi162 ceph-mon[67178]: pgmap v291: 97 pgs: 97 active+clean; 590 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s wr, 22 op/s 2024-03-19T01:10:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:39 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/446702545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:40.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:39 smithi156 ceph-mon[66799]: from='client.24581 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:40.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:39 smithi156 ceph-mon[66799]: from='client.24587 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:40.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:39 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] up:active 2024-03-19T01:10:40.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:39 smithi156 ceph-mon[66799]: fsmap cephfs:5 7 up:standby 5 up:active 2024-03-19T01:10:40.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:39 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:10:40.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:39 smithi156 ceph-mon[66799]: pgmap v291: 97 pgs: 97 active+clean; 590 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s wr, 22 op/s 2024-03-19T01:10:40.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:39 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/446702545' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:10:40.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.571+0000 7f38b6831640 1 -- 172.21.15.116:0/4220189912 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f38b0061c60 con 0x7f38980a2910 2024-03-19T01:10:40.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.579+0000 7f38ad7fa640 1 -- 172.21.15.116:0/4220189912 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258909 (secure 0 0 0) 0x7f38b0061c60 con 0x7f38980a2910 2024-03-19T01:10:40.583 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:40.584 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-19T01:10:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.583+0000 7f38b6831640 1 -- 172.21.15.116:0/4220189912 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f38980a2910 msgr2=0x7f38980a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.583+0000 7f38b6831640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f38980a2910 0x7f38980a4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f38a002c170 tx=0x7f38a002df90 comp rx=0 tx=0).stop 2024-03-19T01:10:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.583+0000 7f38b6831640 1 -- 172.21.15.116:0/4220189912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38b01022f0 msgr2=0x7f38b019e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.583+0000 7f38b6831640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38b01022f0 0x7f38b019e000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f38a800eb10 tx=0x7f38a800c2d0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.583+0000 7f38b6831640 1 -- 172.21.15.116:0/4220189912 shutdown_connections 2024-03-19T01:10:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.583+0000 7f38b6831640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38b01022f0 0x7f38b019e000 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.583+0000 7f38b6831640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f38b0101930 0x7f38b0196f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.583+0000 7f38b6831640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f38980a2910 0x7f38980a4dd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.583+0000 7f38b6831640 1 --2- 172.21.15.116:0/4220189912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f38b0100740 0x7f38b0196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.583+0000 7f38b6831640 1 -- 172.21.15.116:0/4220189912 >> 172.21.15.116:0/4220189912 conn(0x7f38b00fbcb0 msgr2=0x7f38b0104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:40.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.583+0000 7f38b6831640 1 -- 172.21.15.116:0/4220189912 shutdown_connections 2024-03-19T01:10:40.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.583+0000 7f38b6831640 1 -- 172.21.15.116:0/4220189912 wait complete. 2024-03-19T01:10:40.595 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-03-19T01:10:39.057145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":603819,"num_objects":75,"num_object_clones":0,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":339,"num_write_kb":2989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":236,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381044,"kb_used_data":11040,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560780,"statfs":{"total":1151940427776,"available":1151550238720,"internally_reserved":0,"allocated":11304960,"data_stored":7015933,"data_compressed":108585,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13451,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":158,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.922742"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.177163+0000","last_change":"2024-03-19T01:10:33.176468+0000","last_active":"2024-03-19T01:10:33.177163+0000","last_peered":"2024-03-19T01:10:33.177163+0000","last_clean":"2024-03-19T01:10:33.177163+0000","last_became_active":"2024-03-19T01:10:33.175995+0000","last_became_peered":"2024-03-19T01:10:33.175995+0000","last_unstale":"2024-03-19T01:10:33.177163+0000","last_undegraded":"2024-03-19T01:10:33.177163+0000","last_fullsized":"2024-03-19T01:10:33.177163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:02:35.880821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948055+0000","last_change":"2024-03-19T01:10:33.171253+0000","last_active":"2024-03-19T01:10:34.948055+0000","last_peered":"2024-03-19T01:10:34.948055+0000","last_clean":"2024-03-19T01:10:34.948055+0000","last_became_active":"2024-03-19T01:10:33.170330+0000","last_became_peered":"2024-03-19T01:10:33.170330+0000","last_unstale":"2024-03-19T01:10:34.948055+0000","last_undegraded":"2024-03-19T01:10:34.948055+0000","last_fullsized":"2024-03-19T01:10:34.948055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:38.905220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947516+0000","last_change":"2024-03-19T01:10:33.178006+0000","last_active":"2024-03-19T01:10:34.947516+0000","last_peered":"2024-03-19T01:10:34.947516+0000","last_clean":"2024-03-19T01:10:34.947516+0000","last_became_active":"2024-03-19T01:10:33.177662+0000","last_became_peered":"2024-03-19T01:10:33.177662+0000","last_unstale":"2024-03-19T01:10:34.947516+0000","last_undegraded":"2024-03-19T01:10:34.947516+0000","last_fullsized":"2024-03-19T01:10:34.947516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:08.980220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.946846+0000","last_change":"2024-03-19T01:10:33.178805+0000","last_active":"2024-03-19T01:10:34.946846+0000","last_peered":"2024-03-19T01:10:34.946846+0000","last_clean":"2024-03-19T01:10:34.946846+0000","last_became_active":"2024-03-19T01:10:33.178385+0000","last_became_peered":"2024-03-19T01:10:33.178385+0000","last_unstale":"2024-03-19T01:10:34.946846+0000","last_undegraded":"2024-03-19T01:10:34.946846+0000","last_fullsized":"2024-03-19T01:10:34.946846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:06.454345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.679378+0000","last_change":"2024-03-19T01:10:33.183110+0000","last_active":"2024-03-19T01:10:34.679378+0000","last_peered":"2024-03-19T01:10:34.679378+0000","last_clean":"2024-03-19T01:10:34.679378+0000","last_became_active":"2024-03-19T01:10:33.180036+0000","last_became_peered":"2024-03-19T01:10:33.180036+0000","last_unstale":"2024-03-19T01:10:34.679378+0000","last_undegraded":"2024-03-19T01:10:34.679378+0000","last_fullsized":"2024-03-19T01:10:34.679378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:57:51.397156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.946000+0000","last_change":"2024-03-19T01:10:33.164798+0000","last_active":"2024-03-19T01:10:34.946000+0000","last_peered":"2024-03-19T01:10:34.946000+0000","last_clean":"2024-03-19T01:10:34.946000+0000","last_became_active":"2024-03-19T01:10:33.164430+0000","last_became_peered":"2024-03-19T01:10:33.164430+0000","last_unstale":"2024-03-19T01:10:34.946000+0000","last_undegraded":"2024-03-19T01:10:34.946000+0000","last_fullsized":"2024-03-19T01:10:34.946000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:22:39.679057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948918+0000","last_change":"2024-03-19T01:10:33.172946+0000","last_active":"2024-03-19T01:10:34.948918+0000","last_peered":"2024-03-19T01:10:34.948918+0000","last_clean":"2024-03-19T01:10:34.948918+0000","last_became_active":"2024-03-19T01:10:33.172617+0000","last_became_peered":"2024-03-19T01:10:33.172617+0000","last_unstale":"2024-03-19T01:10:34.948918+0000","last_undegraded":"2024-03-19T01:10:34.948918+0000","last_fullsized":"2024-03-19T01:10:34.948918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:18:20.311445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.946954+0000","last_change":"2024-03-19T01:10:33.178162+0000","last_active":"2024-03-19T01:10:34.946954+0000","last_peered":"2024-03-19T01:10:34.946954+0000","last_clean":"2024-03-19T01:10:34.946954+0000","last_became_active":"2024-03-19T01:10:33.177752+0000","last_became_peered":"2024-03-19T01:10:33.177752+0000","last_unstale":"2024-03-19T01:10:34.946954+0000","last_undegraded":"2024-03-19T01:10:34.946954+0000","last_fullsized":"2024-03-19T01:10:34.946954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:01:35.681694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.153185+0000","last_change":"2024-03-19T01:10:33.175619+0000","last_active":"2024-03-19T01:10:34.153185+0000","last_peered":"2024-03-19T01:10:34.153185+0000","last_clean":"2024-03-19T01:10:34.153185+0000","last_became_active":"2024-03-19T01:10:33.175017+0000","last_became_peered":"2024-03-19T01:10:33.175017+0000","last_unstale":"2024-03-19T01:10:34.153185+0000","last_undegraded":"2024-03-19T01:10:34.153185+0000","last_fullsized":"2024-03-19T01:10:34.153185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:49:52.019461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.946388+0000","last_change":"2024-03-19T01:10:33.173035+0000","last_active":"2024-03-19T01:10:34.946388+0000","last_peered":"2024-03-19T01:10:34.946388+0000","last_clean":"2024-03-19T01:10:34.946388+0000","last_became_active":"2024-03-19T01:10:33.172717+0000","last_became_peered":"2024-03-19T01:10:33.172717+0000","last_unstale":"2024-03-19T01:10:34.946388+0000","last_undegraded":"2024-03-19T01:10:34.946388+0000","last_fullsized":"2024-03-19T01:10:34.946388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:38.078147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949793+0000","last_change":"2024-03-19T01:10:33.182679+0000","last_active":"2024-03-19T01:10:34.949793+0000","last_peered":"2024-03-19T01:10:34.949793+0000","last_clean":"2024-03-19T01:10:34.949793+0000","last_became_active":"2024-03-19T01:10:33.182384+0000","last_became_peered":"2024-03-19T01:10:33.182384+0000","last_unstale":"2024-03-19T01:10:34.949793+0000","last_undegraded":"2024-03-19T01:10:34.949793+0000","last_fullsized":"2024-03-19T01:10:34.949793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:37:26.542393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.946688+0000","last_change":"2024-03-19T01:10:33.178818+0000","last_active":"2024-03-19T01:10:34.946688+0000","last_peered":"2024-03-19T01:10:34.946688+0000","last_clean":"2024-03-19T01:10:34.946688+0000","last_became_active":"2024-03-19T01:10:33.178286+0000","last_became_peered":"2024-03-19T01:10:33.178286+0000","last_unstale":"2024-03-19T01:10:34.946688+0000","last_undegraded":"2024-03-19T01:10:34.946688+0000","last_fullsized":"2024-03-19T01:10:34.946688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:05:19.473738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151459+0000","last_change":"2024-03-19T01:10:33.157042+0000","last_active":"2024-03-19T01:10:34.151459+0000","last_peered":"2024-03-19T01:10:34.151459+0000","last_clean":"2024-03-19T01:10:34.151459+0000","last_became_active":"2024-03-19T01:10:33.156576+0000","last_became_peered":"2024-03-19T01:10:33.156576+0000","last_unstale":"2024-03-19T01:10:34.151459+0000","last_undegraded":"2024-03-19T01:10:34.151459+0000","last_fullsized":"2024-03-19T01:10:34.151459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:51.625850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947761+0000","last_change":"2024-03-19T01:10:33.177981+0000","last_active":"2024-03-19T01:10:34.947761+0000","last_peered":"2024-03-19T01:10:34.947761+0000","last_clean":"2024-03-19T01:10:34.947761+0000","last_became_active":"2024-03-19T01:10:33.177634+0000","last_became_peered":"2024-03-19T01:10:33.177634+0000","last_unstale":"2024-03-19T01:10:34.947761+0000","last_undegraded":"2024-03-19T01:10:34.947761+0000","last_fullsized":"2024-03-19T01:10:34.947761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:49.646938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947160+0000","last_change":"2024-03-19T01:10:33.168982+0000","last_active":"2024-03-19T01:10:34.947160+0000","last_peered":"2024-03-19T01:10:34.947160+0000","last_clean":"2024-03-19T01:10:34.947160+0000","last_became_active":"2024-03-19T01:10:33.168569+0000","last_became_peered":"2024-03-19T01:10:33.168569+0000","last_unstale":"2024-03-19T01:10:34.947160+0000","last_undegraded":"2024-03-19T01:10:34.947160+0000","last_fullsized":"2024-03-19T01:10:34.947160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:43.389585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152470+0000","last_change":"2024-03-19T01:10:33.184056+0000","last_active":"2024-03-19T01:10:34.152470+0000","last_peered":"2024-03-19T01:10:34.152470+0000","last_clean":"2024-03-19T01:10:34.152470+0000","last_became_active":"2024-03-19T01:10:33.183712+0000","last_became_peered":"2024-03-19T01:10:33.183712+0000","last_unstale":"2024-03-19T01:10:34.152470+0000","last_undegraded":"2024-03-19T01:10:34.152470+0000","last_fullsized":"2024-03-19T01:10:34.152470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:27.268859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.180748+0000","last_change":"2024-03-19T01:10:33.180114+0000","last_active":"2024-03-19T01:10:33.180748+0000","last_peered":"2024-03-19T01:10:33.180748+0000","last_clean":"2024-03-19T01:10:33.180748+0000","last_became_active":"2024-03-19T01:10:33.179796+0000","last_became_peered":"2024-03-19T01:10:33.179796+0000","last_unstale":"2024-03-19T01:10:33.180748+0000","last_undegraded":"2024-03-19T01:10:33.180748+0000","last_fullsized":"2024-03-19T01:10:33.180748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:10:30.508075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947631+0000","last_change":"2024-03-19T01:10:33.179888+0000","last_active":"2024-03-19T01:10:34.947631+0000","last_peered":"2024-03-19T01:10:34.947631+0000","last_clean":"2024-03-19T01:10:34.947631+0000","last_became_active":"2024-03-19T01:10:33.179562+0000","last_became_peered":"2024-03-19T01:10:33.179562+0000","last_unstale":"2024-03-19T01:10:34.947631+0000","last_undegraded":"2024-03-19T01:10:34.947631+0000","last_fullsized":"2024-03-19T01:10:34.947631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:53.346043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152674+0000","last_change":"2024-03-19T01:10:33.177619+0000","last_active":"2024-03-19T01:10:34.152674+0000","last_peered":"2024-03-19T01:10:34.152674+0000","last_clean":"2024-03-19T01:10:34.152674+0000","last_became_active":"2024-03-19T01:10:33.176002+0000","last_became_peered":"2024-03-19T01:10:33.176002+0000","last_unstale":"2024-03-19T01:10:34.152674+0000","last_undegraded":"2024-03-19T01:10:34.152674+0000","last_fullsized":"2024-03-19T01:10:34.152674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:50.637964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'11","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.981539+0000","last_change":"2024-03-19T01:10:32.151875+0000","last_active":"2024-03-19T01:10:36.981539+0000","last_peered":"2024-03-19T01:10:36.981539+0000","last_clean":"2024-03-19T01:10:36.981539+0000","last_became_active":"2024-03-19T01:10:32.151448+0000","last_became_peered":"2024-03-19T01:10:32.151448+0000","last_unstale":"2024-03-19T01:10:36.981539+0000","last_undegraded":"2024-03-19T01:10:36.981539+0000","last_fullsized":"2024-03-19T01:10:36.981539+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:17.930078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947663+0000","last_change":"2024-03-19T01:10:33.184258+0000","last_active":"2024-03-19T01:10:34.947663+0000","last_peered":"2024-03-19T01:10:34.947663+0000","last_clean":"2024-03-19T01:10:34.947663+0000","last_became_active":"2024-03-19T01:10:33.183899+0000","last_became_peered":"2024-03-19T01:10:33.183899+0000","last_unstale":"2024-03-19T01:10:34.947663+0000","last_undegraded":"2024-03-19T01:10:34.947663+0000","last_fullsized":"2024-03-19T01:10:34.947663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:47.220818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948067+0000","last_change":"2024-03-19T01:10:32.151998+0000","last_active":"2024-03-19T01:10:34.948067+0000","last_peered":"2024-03-19T01:10:34.948067+0000","last_clean":"2024-03-19T01:10:34.948067+0000","last_became_active":"2024-03-19T01:10:32.151644+0000","last_became_peered":"2024-03-19T01:10:32.151644+0000","last_unstale":"2024-03-19T01:10:34.948067+0000","last_undegraded":"2024-03-19T01:10:34.948067+0000","last_fullsized":"2024-03-19T01:10:34.948067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:20:28.660705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948099+0000","last_change":"2024-03-19T01:10:33.176759+0000","last_active":"2024-03-19T01:10:34.948099+0000","last_peered":"2024-03-19T01:10:34.948099+0000","last_clean":"2024-03-19T01:10:34.948099+0000","last_became_active":"2024-03-19T01:10:33.176409+0000","last_became_peered":"2024-03-19T01:10:33.176409+0000","last_unstale":"2024-03-19T01:10:34.948099+0000","last_undegraded":"2024-03-19T01:10:34.948099+0000","last_fullsized":"2024-03-19T01:10:34.948099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:21:08.513411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.679443+0000","last_change":"2024-03-19T01:10:32.149984+0000","last_active":"2024-03-19T01:10:34.679443+0000","last_peered":"2024-03-19T01:10:34.679443+0000","last_clean":"2024-03-19T01:10:34.679443+0000","last_became_active":"2024-03-19T01:10:32.149292+0000","last_became_peered":"2024-03-19T01:10:32.149292+0000","last_unstale":"2024-03-19T01:10:34.679443+0000","last_undegraded":"2024-03-19T01:10:34.679443+0000","last_fullsized":"2024-03-19T01:10:34.679443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:29:23.226555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152831+0000","last_change":"2024-03-19T01:10:33.174475+0000","last_active":"2024-03-19T01:10:34.152831+0000","last_peered":"2024-03-19T01:10:34.152831+0000","last_clean":"2024-03-19T01:10:34.152831+0000","last_became_active":"2024-03-19T01:10:33.174154+0000","last_became_peered":"2024-03-19T01:10:33.174154+0000","last_unstale":"2024-03-19T01:10:34.152831+0000","last_undegraded":"2024-03-19T01:10:34.152831+0000","last_fullsized":"2024-03-19T01:10:34.152831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:17.874609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.679352+0000","last_change":"2024-03-19T01:10:32.151717+0000","last_active":"2024-03-19T01:10:34.679352+0000","last_peered":"2024-03-19T01:10:34.679352+0000","last_clean":"2024-03-19T01:10:34.679352+0000","last_became_active":"2024-03-19T01:10:32.151356+0000","last_became_peered":"2024-03-19T01:10:32.151356+0000","last_unstale":"2024-03-19T01:10:34.679352+0000","last_undegraded":"2024-03-19T01:10:34.679352+0000","last_fullsized":"2024-03-19T01:10:34.679352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:02.614226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.709298+0000","last_change":"2024-03-19T01:10:33.177202+0000","last_active":"2024-03-19T01:10:34.709298+0000","last_peered":"2024-03-19T01:10:34.709298+0000","last_clean":"2024-03-19T01:10:34.709298+0000","last_became_active":"2024-03-19T01:10:33.172004+0000","last_became_peered":"2024-03-19T01:10:33.172004+0000","last_unstale":"2024-03-19T01:10:34.709298+0000","last_undegraded":"2024-03-19T01:10:34.709298+0000","last_fullsized":"2024-03-19T01:10:34.709298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:11:36.610997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.951792+0000","last_change":"2024-03-19T01:10:32.151138+0000","last_active":"2024-03-19T01:10:34.951792+0000","last_peered":"2024-03-19T01:10:34.951792+0000","last_clean":"2024-03-19T01:10:34.951792+0000","last_became_active":"2024-03-19T01:10:32.150297+0000","last_became_peered":"2024-03-19T01:10:32.150297+0000","last_unstale":"2024-03-19T01:10:34.951792+0000","last_undegraded":"2024-03-19T01:10:34.951792+0000","last_fullsized":"2024-03-19T01:10:34.951792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:32:37.104986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.679207+0000","last_change":"2024-03-19T01:10:33.178878+0000","last_active":"2024-03-19T01:10:34.679207+0000","last_peered":"2024-03-19T01:10:34.679207+0000","last_clean":"2024-03-19T01:10:34.679207+0000","last_became_active":"2024-03-19T01:10:33.178526+0000","last_became_peered":"2024-03-19T01:10:33.178526+0000","last_unstale":"2024-03-19T01:10:34.679207+0000","last_undegraded":"2024-03-19T01:10:34.679207+0000","last_fullsized":"2024-03-19T01:10:34.679207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:40.074065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.980389+0000","last_change":"2024-03-19T01:10:32.153255+0000","last_active":"2024-03-19T01:10:36.980389+0000","last_peered":"2024-03-19T01:10:36.980389+0000","last_clean":"2024-03-19T01:10:36.980389+0000","last_became_active":"2024-03-19T01:10:32.152976+0000","last_became_peered":"2024-03-19T01:10:32.152976+0000","last_unstale":"2024-03-19T01:10:36.980389+0000","last_undegraded":"2024-03-19T01:10:36.980389+0000","last_fullsized":"2024-03-19T01:10:36.980389+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:51:00.313688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.709146+0000","last_change":"2024-03-19T01:10:33.173710+0000","last_active":"2024-03-19T01:10:34.709146+0000","last_peered":"2024-03-19T01:10:34.709146+0000","last_clean":"2024-03-19T01:10:34.709146+0000","last_became_active":"2024-03-19T01:10:33.173325+0000","last_became_peered":"2024-03-19T01:10:33.173325+0000","last_unstale":"2024-03-19T01:10:34.709146+0000","last_undegraded":"2024-03-19T01:10:34.709146+0000","last_fullsized":"2024-03-19T01:10:34.709146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:21.755723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151265+0000","last_change":"2024-03-19T01:10:32.144286+0000","last_active":"2024-03-19T01:10:34.151265+0000","last_peered":"2024-03-19T01:10:34.151265+0000","last_clean":"2024-03-19T01:10:34.151265+0000","last_became_active":"2024-03-19T01:10:32.143959+0000","last_became_peered":"2024-03-19T01:10:32.143959+0000","last_unstale":"2024-03-19T01:10:34.151265+0000","last_undegraded":"2024-03-19T01:10:34.151265+0000","last_fullsized":"2024-03-19T01:10:34.151265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:46:15.750018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947048+0000","last_change":"2024-03-19T01:10:33.178122+0000","last_active":"2024-03-19T01:10:34.947048+0000","last_peered":"2024-03-19T01:10:34.947048+0000","last_clean":"2024-03-19T01:10:34.947048+0000","last_became_active":"2024-03-19T01:10:33.175491+0000","last_became_peered":"2024-03-19T01:10:33.175491+0000","last_unstale":"2024-03-19T01:10:34.947048+0000","last_undegraded":"2024-03-19T01:10:34.947048+0000","last_fullsized":"2024-03-19T01:10:34.947048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:34.418463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.980427+0000","last_change":"2024-03-19T01:10:32.147561+0000","last_active":"2024-03-19T01:10:36.980427+0000","last_peered":"2024-03-19T01:10:36.980427+0000","last_clean":"2024-03-19T01:10:36.980427+0000","last_became_active":"2024-03-19T01:10:32.146825+0000","last_became_peered":"2024-03-19T01:10:32.146825+0000","last_unstale":"2024-03-19T01:10:36.980427+0000","last_undegraded":"2024-03-19T01:10:36.980427+0000","last_fullsized":"2024-03-19T01:10:36.980427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:20:09.833960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948195+0000","last_change":"2024-03-19T01:10:33.170492+0000","last_active":"2024-03-19T01:10:34.948195+0000","last_peered":"2024-03-19T01:10:34.948195+0000","last_clean":"2024-03-19T01:10:34.948195+0000","last_became_active":"2024-03-19T01:10:33.170013+0000","last_became_peered":"2024-03-19T01:10:33.170013+0000","last_unstale":"2024-03-19T01:10:34.948195+0000","last_undegraded":"2024-03-19T01:10:34.948195+0000","last_fullsized":"2024-03-19T01:10:34.948195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:13.046663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.950824+0000","last_change":"2024-03-19T01:10:32.147060+0000","last_active":"2024-03-19T01:10:34.950824+0000","last_peered":"2024-03-19T01:10:34.950824+0000","last_clean":"2024-03-19T01:10:34.950824+0000","last_became_active":"2024-03-19T01:10:32.146721+0000","last_became_peered":"2024-03-19T01:10:32.146721+0000","last_unstale":"2024-03-19T01:10:34.950824+0000","last_undegraded":"2024-03-19T01:10:34.950824+0000","last_fullsized":"2024-03-19T01:10:34.950824+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:13:16.930746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152960+0000","last_change":"2024-03-19T01:10:33.174431+0000","last_active":"2024-03-19T01:10:34.152960+0000","last_peered":"2024-03-19T01:10:34.152960+0000","last_clean":"2024-03-19T01:10:34.152960+0000","last_became_active":"2024-03-19T01:10:33.174040+0000","last_became_peered":"2024-03-19T01:10:33.174040+0000","last_unstale":"2024-03-19T01:10:34.152960+0000","last_undegraded":"2024-03-19T01:10:34.152960+0000","last_fullsized":"2024-03-19T01:10:34.152960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:53.850400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.951917+0000","last_change":"2024-03-19T01:10:32.149194+0000","last_active":"2024-03-19T01:10:34.951917+0000","last_peered":"2024-03-19T01:10:34.951917+0000","last_clean":"2024-03-19T01:10:34.951917+0000","last_became_active":"2024-03-19T01:10:32.148835+0000","last_became_peered":"2024-03-19T01:10:32.148835+0000","last_unstale":"2024-03-19T01:10:34.951917+0000","last_undegraded":"2024-03-19T01:10:34.951917+0000","last_fullsized":"2024-03-19T01:10:34.951917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:55.697445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948238+0000","last_change":"2024-03-19T01:10:33.180203+0000","last_active":"2024-03-19T01:10:34.948238+0000","last_peered":"2024-03-19T01:10:34.948238+0000","last_clean":"2024-03-19T01:10:34.948238+0000","last_became_active":"2024-03-19T01:10:33.179812+0000","last_became_peered":"2024-03-19T01:10:33.179812+0000","last_unstale":"2024-03-19T01:10:34.948238+0000","last_undegraded":"2024-03-19T01:10:34.948238+0000","last_fullsized":"2024-03-19T01:10:34.948238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:38.371706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.950503+0000","last_change":"2024-03-19T01:10:32.153852+0000","last_active":"2024-03-19T01:10:34.950503+0000","last_peered":"2024-03-19T01:10:34.950503+0000","last_clean":"2024-03-19T01:10:34.950503+0000","last_became_active":"2024-03-19T01:10:32.153537+0000","last_became_peered":"2024-03-19T01:10:32.153537+0000","last_unstale":"2024-03-19T01:10:34.950503+0000","last_undegraded":"2024-03-19T01:10:34.950503+0000","last_fullsized":"2024-03-19T01:10:34.950503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:48.544344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152843+0000","last_change":"2024-03-19T01:10:33.175629+0000","last_active":"2024-03-19T01:10:34.152843+0000","last_peered":"2024-03-19T01:10:34.152843+0000","last_clean":"2024-03-19T01:10:34.152843+0000","last_became_active":"2024-03-19T01:10:33.175278+0000","last_became_peered":"2024-03-19T01:10:33.175278+0000","last_unstale":"2024-03-19T01:10:34.152843+0000","last_undegraded":"2024-03-19T01:10:34.152843+0000","last_fullsized":"2024-03-19T01:10:34.152843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:42:31.534216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'13","reported_seq":28,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:37.988224+0000","last_change":"2024-03-19T01:10:32.148290+0000","last_active":"2024-03-19T01:10:37.988224+0000","last_peered":"2024-03-19T01:10:37.988224+0000","last_clean":"2024-03-19T01:10:37.988224+0000","last_became_active":"2024-03-19T01:10:32.147940+0000","last_became_peered":"2024-03-19T01:10:32.147940+0000","last_unstale":"2024-03-19T01:10:37.988224+0000","last_undegraded":"2024-03-19T01:10:37.988224+0000","last_fullsized":"2024-03-19T01:10:37.988224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:40:58.125706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8212,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.679313+0000","last_change":"2024-03-19T01:10:33.174723+0000","last_active":"2024-03-19T01:10:34.679313+0000","last_peered":"2024-03-19T01:10:34.679313+0000","last_clean":"2024-03-19T01:10:34.679313+0000","last_became_active":"2024-03-19T01:10:33.174369+0000","last_became_peered":"2024-03-19T01:10:33.174369+0000","last_unstale":"2024-03-19T01:10:34.679313+0000","last_undegraded":"2024-03-19T01:10:34.679313+0000","last_fullsized":"2024-03-19T01:10:34.679313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:33.641050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:37.987483+0000","last_change":"2024-03-19T01:10:32.146760+0000","last_active":"2024-03-19T01:10:37.987483+0000","last_peered":"2024-03-19T01:10:37.987483+0000","last_clean":"2024-03-19T01:10:37.987483+0000","last_became_active":"2024-03-19T01:10:32.145610+0000","last_became_peered":"2024-03-19T01:10:32.145610+0000","last_unstale":"2024-03-19T01:10:37.987483+0000","last_undegraded":"2024-03-19T01:10:37.987483+0000","last_fullsized":"2024-03-19T01:10:37.987483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:10:10.959609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151336+0000","last_change":"2024-03-19T01:10:33.176948+0000","last_active":"2024-03-19T01:10:34.151336+0000","last_peered":"2024-03-19T01:10:34.151336+0000","last_clean":"2024-03-19T01:10:34.151336+0000","last_became_active":"2024-03-19T01:10:33.176604+0000","last_became_peered":"2024-03-19T01:10:33.176604+0000","last_unstale":"2024-03-19T01:10:34.151336+0000","last_undegraded":"2024-03-19T01:10:34.151336+0000","last_fullsized":"2024-03-19T01:10:34.151336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:45:45.032639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.980809+0000","last_change":"2024-03-19T01:10:32.152397+0000","last_active":"2024-03-19T01:10:36.980809+0000","last_peered":"2024-03-19T01:10:36.980809+0000","last_clean":"2024-03-19T01:10:36.980809+0000","last_became_active":"2024-03-19T01:10:32.152026+0000","last_became_peered":"2024-03-19T01:10:32.152026+0000","last_unstale":"2024-03-19T01:10:36.980809+0000","last_undegraded":"2024-03-19T01:10:36.980809+0000","last_fullsized":"2024-03-19T01:10:36.980809+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:47.789504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949920+0000","last_change":"2024-03-19T01:10:33.168529+0000","last_active":"2024-03-19T01:10:34.949920+0000","last_peered":"2024-03-19T01:10:34.949920+0000","last_clean":"2024-03-19T01:10:34.949920+0000","last_became_active":"2024-03-19T01:10:33.168198+0000","last_became_peered":"2024-03-19T01:10:33.168198+0000","last_unstale":"2024-03-19T01:10:34.949920+0000","last_undegraded":"2024-03-19T01:10:34.949920+0000","last_fullsized":"2024-03-19T01:10:34.949920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:36.030791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.979607+0000","last_change":"2024-03-19T01:10:32.145565+0000","last_active":"2024-03-19T01:10:36.979607+0000","last_peered":"2024-03-19T01:10:36.979607+0000","last_clean":"2024-03-19T01:10:36.979607+0000","last_became_active":"2024-03-19T01:10:32.145234+0000","last_became_peered":"2024-03-19T01:10:32.145234+0000","last_unstale":"2024-03-19T01:10:36.979607+0000","last_undegraded":"2024-03-19T01:10:36.979607+0000","last_fullsized":"2024-03-19T01:10:36.979607+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:35.563136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949518+0000","last_change":"2024-03-19T01:10:33.182001+0000","last_active":"2024-03-19T01:10:34.949518+0000","last_peered":"2024-03-19T01:10:34.949518+0000","last_clean":"2024-03-19T01:10:34.949518+0000","last_became_active":"2024-03-19T01:10:33.181638+0000","last_became_peered":"2024-03-19T01:10:33.181638+0000","last_unstale":"2024-03-19T01:10:34.949518+0000","last_undegraded":"2024-03-19T01:10:34.949518+0000","last_fullsized":"2024-03-19T01:10:34.949518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:59:23.155190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'10","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.951212+0000","last_change":"2024-03-19T01:10:32.154159+0000","last_active":"2024-03-19T01:10:34.951212+0000","last_peered":"2024-03-19T01:10:34.951212+0000","last_clean":"2024-03-19T01:10:34.951212+0000","last_became_active":"2024-03-19T01:10:32.153853+0000","last_became_peered":"2024-03-19T01:10:32.153853+0000","last_unstale":"2024-03-19T01:10:34.951212+0000","last_undegraded":"2024-03-19T01:10:34.951212+0000","last_fullsized":"2024-03-19T01:10:34.951212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:41:13.727588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.709146+0000","last_change":"2024-03-19T01:10:33.176659+0000","last_active":"2024-03-19T01:10:34.709146+0000","last_peered":"2024-03-19T01:10:34.709146+0000","last_clean":"2024-03-19T01:10:34.709146+0000","last_became_active":"2024-03-19T01:10:33.176319+0000","last_became_peered":"2024-03-19T01:10:33.176319+0000","last_unstale":"2024-03-19T01:10:34.709146+0000","last_undegraded":"2024-03-19T01:10:34.709146+0000","last_fullsized":"2024-03-19T01:10:34.709146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:00.311883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.679367+0000","last_change":"2024-03-19T01:10:32.144855+0000","last_active":"2024-03-19T01:10:34.679367+0000","last_peered":"2024-03-19T01:10:34.679367+0000","last_clean":"2024-03-19T01:10:34.679367+0000","last_became_active":"2024-03-19T01:10:32.144540+0000","last_became_peered":"2024-03-19T01:10:32.144540+0000","last_unstale":"2024-03-19T01:10:34.679367+0000","last_undegraded":"2024-03-19T01:10:34.679367+0000","last_fullsized":"2024-03-19T01:10:34.679367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:22.399954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.174270+0000","last_change":"2024-03-19T01:10:33.172739+0000","last_active":"2024-03-19T01:10:33.174270+0000","last_peered":"2024-03-19T01:10:33.174270+0000","last_clean":"2024-03-19T01:10:33.174270+0000","last_became_active":"2024-03-19T01:10:33.172391+0000","last_became_peered":"2024-03-19T01:10:33.172391+0000","last_unstale":"2024-03-19T01:10:33.174270+0000","last_undegraded":"2024-03-19T01:10:33.174270+0000","last_fullsized":"2024-03-19T01:10:33.174270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:15:47.775133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947437+0000","last_change":"2024-03-19T01:10:32.144491+0000","last_active":"2024-03-19T01:10:34.947437+0000","last_peered":"2024-03-19T01:10:34.947437+0000","last_clean":"2024-03-19T01:10:34.947437+0000","last_became_active":"2024-03-19T01:10:32.144170+0000","last_became_peered":"2024-03-19T01:10:32.144170+0000","last_unstale":"2024-03-19T01:10:34.947437+0000","last_undegraded":"2024-03-19T01:10:34.947437+0000","last_fullsized":"2024-03-19T01:10:34.947437+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:07.998818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947282+0000","last_change":"2024-03-19T01:10:33.175479+0000","last_active":"2024-03-19T01:10:34.947282+0000","last_peered":"2024-03-19T01:10:34.947282+0000","last_clean":"2024-03-19T01:10:34.947282+0000","last_became_active":"2024-03-19T01:10:33.172530+0000","last_became_peered":"2024-03-19T01:10:33.172530+0000","last_unstale":"2024-03-19T01:10:34.947282+0000","last_undegraded":"2024-03-19T01:10:34.947282+0000","last_fullsized":"2024-03-19T01:10:34.947282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:16.058826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949212+0000","last_change":"2024-03-19T01:10:32.143629+0000","last_active":"2024-03-19T01:10:34.949212+0000","last_peered":"2024-03-19T01:10:34.949212+0000","last_clean":"2024-03-19T01:10:34.949212+0000","last_became_active":"2024-03-19T01:10:32.143321+0000","last_became_peered":"2024-03-19T01:10:32.143321+0000","last_unstale":"2024-03-19T01:10:34.949212+0000","last_undegraded":"2024-03-19T01:10:34.949212+0000","last_fullsized":"2024-03-19T01:10:34.949212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:47.404245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.946621+0000","last_change":"2024-03-19T01:10:33.171284+0000","last_active":"2024-03-19T01:10:34.946621+0000","last_peered":"2024-03-19T01:10:34.946621+0000","last_clean":"2024-03-19T01:10:34.946621+0000","last_became_active":"2024-03-19T01:10:33.168717+0000","last_became_peered":"2024-03-19T01:10:33.168717+0000","last_unstale":"2024-03-19T01:10:34.946621+0000","last_undegraded":"2024-03-19T01:10:34.946621+0000","last_fullsized":"2024-03-19T01:10:34.946621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:52.335303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.709185+0000","last_change":"2024-03-19T01:10:33.177261+0000","last_active":"2024-03-19T01:10:34.709185+0000","last_peered":"2024-03-19T01:10:34.709185+0000","last_clean":"2024-03-19T01:10:34.709185+0000","last_became_active":"2024-03-19T01:10:33.171836+0000","last_became_peered":"2024-03-19T01:10:33.171836+0000","last_unstale":"2024-03-19T01:10:34.709185+0000","last_undegraded":"2024-03-19T01:10:34.709185+0000","last_fullsized":"2024-03-19T01:10:34.709185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:49:14.577925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151163+0000","last_change":"2024-03-19T01:10:32.141587+0000","last_active":"2024-03-19T01:10:34.151163+0000","last_peered":"2024-03-19T01:10:34.151163+0000","last_clean":"2024-03-19T01:10:34.151163+0000","last_became_active":"2024-03-19T01:10:32.141236+0000","last_became_peered":"2024-03-19T01:10:32.141236+0000","last_unstale":"2024-03-19T01:10:34.151163+0000","last_undegraded":"2024-03-19T01:10:34.151163+0000","last_fullsized":"2024-03-19T01:10:34.151163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:07:40.685399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948028+0000","last_change":"2024-03-19T01:10:33.174761+0000","last_active":"2024-03-19T01:10:34.948028+0000","last_peered":"2024-03-19T01:10:34.948028+0000","last_clean":"2024-03-19T01:10:34.948028+0000","last_became_active":"2024-03-19T01:10:33.174431+0000","last_became_peered":"2024-03-19T01:10:33.174431+0000","last_unstale":"2024-03-19T01:10:34.948028+0000","last_undegraded":"2024-03-19T01:10:34.948028+0000","last_fullsized":"2024-03-19T01:10:34.948028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:21.613055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.978591+0000","last_change":"2024-03-19T01:10:32.145816+0000","last_active":"2024-03-19T01:10:36.978591+0000","last_peered":"2024-03-19T01:10:36.978591+0000","last_clean":"2024-03-19T01:10:36.978591+0000","last_became_active":"2024-03-19T01:10:32.145055+0000","last_became_peered":"2024-03-19T01:10:32.145055+0000","last_unstale":"2024-03-19T01:10:36.978591+0000","last_undegraded":"2024-03-19T01:10:36.978591+0000","last_fullsized":"2024-03-19T01:10:36.978591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:56.428854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":338,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949358+0000","last_change":"2024-03-19T01:09:20.932686+0000","last_active":"2024-03-19T01:10:34.949358+0000","last_peered":"2024-03-19T01:10:34.949358+0000","last_clean":"2024-03-19T01:10:34.949358+0000","last_became_active":"2024-03-19T01:09:20.932170+0000","last_became_peered":"2024-03-19T01:09:20.932170+0000","last_unstale":"2024-03-19T01:10:34.949358+0000","last_undegraded":"2024-03-19T01:10:34.949358+0000","last_fullsized":"2024-03-19T01:10:34.949358+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:05:53.937682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:05:53.937682+0000","last_clean_scrub_stamp":"2024-03-19T01:05:53.937682+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:41.523200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151263+0000","last_change":"2024-03-19T01:10:33.173230+0000","last_active":"2024-03-19T01:10:34.151263+0000","last_peered":"2024-03-19T01:10:34.151263+0000","last_clean":"2024-03-19T01:10:34.151263+0000","last_became_active":"2024-03-19T01:10:33.172858+0000","last_became_peered":"2024-03-19T01:10:33.172858+0000","last_unstale":"2024-03-19T01:10:34.151263+0000","last_undegraded":"2024-03-19T01:10:34.151263+0000","last_fullsized":"2024-03-19T01:10:34.151263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:43.582211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.952180+0000","last_change":"2024-03-19T01:10:32.147419+0000","last_active":"2024-03-19T01:10:34.952180+0000","last_peered":"2024-03-19T01:10:34.952180+0000","last_clean":"2024-03-19T01:10:34.952180+0000","last_became_active":"2024-03-19T01:10:32.147111+0000","last_became_peered":"2024-03-19T01:10:32.147111+0000","last_unstale":"2024-03-19T01:10:34.952180+0000","last_undegraded":"2024-03-19T01:10:34.952180+0000","last_fullsized":"2024-03-19T01:10:34.952180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:45:34.989763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.950168+0000","last_change":"2024-03-19T01:10:32.154233+0000","last_active":"2024-03-19T01:10:34.950168+0000","last_peered":"2024-03-19T01:10:34.950168+0000","last_clean":"2024-03-19T01:10:34.950168+0000","last_became_active":"2024-03-19T01:10:32.153939+0000","last_became_peered":"2024-03-19T01:10:32.153939+0000","last_unstale":"2024-03-19T01:10:34.950168+0000","last_undegraded":"2024-03-19T01:10:34.950168+0000","last_fullsized":"2024-03-19T01:10:34.950168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:45.377161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151482+0000","last_change":"2024-03-19T01:10:33.168395+0000","last_active":"2024-03-19T01:10:34.151482+0000","last_peered":"2024-03-19T01:10:34.151482+0000","last_clean":"2024-03-19T01:10:34.151482+0000","last_became_active":"2024-03-19T01:10:33.168078+0000","last_became_peered":"2024-03-19T01:10:33.168078+0000","last_unstale":"2024-03-19T01:10:34.151482+0000","last_undegraded":"2024-03-19T01:10:34.151482+0000","last_fullsized":"2024-03-19T01:10:34.151482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:54:52.597591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.981779+0000","last_change":"2024-03-19T01:10:32.152489+0000","last_active":"2024-03-19T01:10:36.981779+0000","last_peered":"2024-03-19T01:10:36.981779+0000","last_clean":"2024-03-19T01:10:36.981779+0000","last_became_active":"2024-03-19T01:10:32.152126+0000","last_became_peered":"2024-03-19T01:10:32.152126+0000","last_unstale":"2024-03-19T01:10:36.981779+0000","last_undegraded":"2024-03-19T01:10:36.981779+0000","last_fullsized":"2024-03-19T01:10:36.981779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:26.729261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948530+0000","last_change":"2024-03-19T01:10:33.174635+0000","last_active":"2024-03-19T01:10:34.948530+0000","last_peered":"2024-03-19T01:10:34.948530+0000","last_clean":"2024-03-19T01:10:34.948530+0000","last_became_active":"2024-03-19T01:10:33.173970+0000","last_became_peered":"2024-03-19T01:10:33.173970+0000","last_unstale":"2024-03-19T01:10:34.948530+0000","last_undegraded":"2024-03-19T01:10:34.948530+0000","last_fullsized":"2024-03-19T01:10:34.948530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:16.174169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:37.987363+0000","last_change":"2024-03-19T01:10:32.148225+0000","last_active":"2024-03-19T01:10:37.987363+0000","last_peered":"2024-03-19T01:10:37.987363+0000","last_clean":"2024-03-19T01:10:37.987363+0000","last_became_active":"2024-03-19T01:10:32.147889+0000","last_became_peered":"2024-03-19T01:10:32.147889+0000","last_unstale":"2024-03-19T01:10:37.987363+0000","last_undegraded":"2024-03-19T01:10:37.987363+0000","last_fullsized":"2024-03-19T01:10:37.987363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:56.037018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947675+0000","last_change":"2024-03-19T01:10:33.172925+0000","last_active":"2024-03-19T01:10:34.947675+0000","last_peered":"2024-03-19T01:10:34.947675+0000","last_clean":"2024-03-19T01:10:34.947675+0000","last_became_active":"2024-03-19T01:10:33.172572+0000","last_became_peered":"2024-03-19T01:10:33.172572+0000","last_unstale":"2024-03-19T01:10:34.947675+0000","last_undegraded":"2024-03-19T01:10:34.947675+0000","last_fullsized":"2024-03-19T01:10:34.947675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:02.853337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948702+0000","last_change":"2024-03-19T01:10:32.146245+0000","last_active":"2024-03-19T01:10:34.948702+0000","last_peered":"2024-03-19T01:10:34.948702+0000","last_clean":"2024-03-19T01:10:34.948702+0000","last_became_active":"2024-03-19T01:10:32.145877+0000","last_became_peered":"2024-03-19T01:10:32.145877+0000","last_unstale":"2024-03-19T01:10:34.948702+0000","last_undegraded":"2024-03-19T01:10:34.948702+0000","last_fullsized":"2024-03-19T01:10:34.948702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:45.323155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.678959+0000","last_change":"2024-03-19T01:10:33.178538+0000","last_active":"2024-03-19T01:10:34.678959+0000","last_peered":"2024-03-19T01:10:34.678959+0000","last_clean":"2024-03-19T01:10:34.678959+0000","last_became_active":"2024-03-19T01:10:33.178170+0000","last_became_peered":"2024-03-19T01:10:33.178170+0000","last_unstale":"2024-03-19T01:10:34.678959+0000","last_undegraded":"2024-03-19T01:10:34.678959+0000","last_fullsized":"2024-03-19T01:10:34.678959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:41.913715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:37.986468+0000","last_change":"2024-03-19T01:10:32.147697+0000","last_active":"2024-03-19T01:10:37.986468+0000","last_peered":"2024-03-19T01:10:37.986468+0000","last_clean":"2024-03-19T01:10:37.986468+0000","last_became_active":"2024-03-19T01:10:32.147399+0000","last_became_peered":"2024-03-19T01:10:32.147399+0000","last_unstale":"2024-03-19T01:10:37.986468+0000","last_undegraded":"2024-03-19T01:10:37.986468+0000","last_fullsized":"2024-03-19T01:10:37.986468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:33:38.961646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.176745+0000","last_change":"2024-03-19T01:10:33.175990+0000","last_active":"2024-03-19T01:10:33.176745+0000","last_peered":"2024-03-19T01:10:33.176745+0000","last_clean":"2024-03-19T01:10:33.176745+0000","last_became_active":"2024-03-19T01:10:33.175539+0000","last_became_peered":"2024-03-19T01:10:33.175539+0000","last_unstale":"2024-03-19T01:10:33.176745+0000","last_undegraded":"2024-03-19T01:10:33.176745+0000","last_fullsized":"2024-03-19T01:10:33.176745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:11:42.608955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.979950+0000","last_change":"2024-03-19T01:10:32.151834+0000","last_active":"2024-03-19T01:10:36.979950+0000","last_peered":"2024-03-19T01:10:36.979950+0000","last_clean":"2024-03-19T01:10:36.979950+0000","last_became_active":"2024-03-19T01:10:32.151514+0000","last_became_peered":"2024-03-19T01:10:32.151514+0000","last_unstale":"2024-03-19T01:10:36.979950+0000","last_undegraded":"2024-03-19T01:10:36.979950+0000","last_fullsized":"2024-03-19T01:10:36.979950+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:40:16.839334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947540+0000","last_change":"2024-03-19T01:10:33.185206+0000","last_active":"2024-03-19T01:10:34.947540+0000","last_peered":"2024-03-19T01:10:34.947540+0000","last_clean":"2024-03-19T01:10:34.947540+0000","last_became_active":"2024-03-19T01:10:33.184901+0000","last_became_peered":"2024-03-19T01:10:33.184901+0000","last_unstale":"2024-03-19T01:10:34.947540+0000","last_undegraded":"2024-03-19T01:10:34.947540+0000","last_fullsized":"2024-03-19T01:10:34.947540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:54:47.266255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.951381+0000","last_change":"2024-03-19T01:10:32.146654+0000","last_active":"2024-03-19T01:10:34.951381+0000","last_peered":"2024-03-19T01:10:34.951381+0000","last_clean":"2024-03-19T01:10:34.951381+0000","last_became_active":"2024-03-19T01:10:32.146178+0000","last_became_peered":"2024-03-19T01:10:32.146178+0000","last_unstale":"2024-03-19T01:10:34.951381+0000","last_undegraded":"2024-03-19T01:10:34.951381+0000","last_fullsized":"2024-03-19T01:10:34.951381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:27.257000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949740+0000","last_change":"2024-03-19T01:10:33.181232+0000","last_active":"2024-03-19T01:10:34.949740+0000","last_peered":"2024-03-19T01:10:34.949740+0000","last_clean":"2024-03-19T01:10:34.949740+0000","last_became_active":"2024-03-19T01:10:33.180719+0000","last_became_peered":"2024-03-19T01:10:33.180719+0000","last_unstale":"2024-03-19T01:10:34.949740+0000","last_undegraded":"2024-03-19T01:10:34.949740+0000","last_fullsized":"2024-03-19T01:10:34.949740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:43.201965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.979620+0000","last_change":"2024-03-19T01:10:32.144273+0000","last_active":"2024-03-19T01:10:36.979620+0000","last_peered":"2024-03-19T01:10:36.979620+0000","last_clean":"2024-03-19T01:10:36.979620+0000","last_became_active":"2024-03-19T01:10:32.143514+0000","last_became_peered":"2024-03-19T01:10:32.143514+0000","last_unstale":"2024-03-19T01:10:36.979620+0000","last_undegraded":"2024-03-19T01:10:36.979620+0000","last_fullsized":"2024-03-19T01:10:36.979620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:57.705653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949691+0000","last_change":"2024-03-19T01:10:33.178129+0000","last_active":"2024-03-19T01:10:34.949691+0000","last_peered":"2024-03-19T01:10:34.949691+0000","last_clean":"2024-03-19T01:10:34.949691+0000","last_became_active":"2024-03-19T01:10:33.177802+0000","last_became_peered":"2024-03-19T01:10:33.177802+0000","last_unstale":"2024-03-19T01:10:34.949691+0000","last_undegraded":"2024-03-19T01:10:34.949691+0000","last_fullsized":"2024-03-19T01:10:34.949691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:46:42.698865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.979709+0000","last_change":"2024-03-19T01:10:32.149613+0000","last_active":"2024-03-19T01:10:36.979709+0000","last_peered":"2024-03-19T01:10:36.979709+0000","last_clean":"2024-03-19T01:10:36.979709+0000","last_became_active":"2024-03-19T01:10:32.147651+0000","last_became_peered":"2024-03-19T01:10:32.147651+0000","last_unstale":"2024-03-19T01:10:36.979709+0000","last_undegraded":"2024-03-19T01:10:36.979709+0000","last_fullsized":"2024-03-19T01:10:36.979709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:06:59.222944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949176+0000","last_change":"2024-03-19T01:10:33.173682+0000","last_active":"2024-03-19T01:10:34.949176+0000","last_peered":"2024-03-19T01:10:34.949176+0000","last_clean":"2024-03-19T01:10:34.949176+0000","last_became_active":"2024-03-19T01:10:33.173374+0000","last_became_peered":"2024-03-19T01:10:33.173374+0000","last_unstale":"2024-03-19T01:10:34.949176+0000","last_undegraded":"2024-03-19T01:10:34.949176+0000","last_fullsized":"2024-03-19T01:10:34.949176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:59:47.396410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151491+0000","last_change":"2024-03-19T01:10:32.149253+0000","last_active":"2024-03-19T01:10:34.151491+0000","last_peered":"2024-03-19T01:10:34.151491+0000","last_clean":"2024-03-19T01:10:34.151491+0000","last_became_active":"2024-03-19T01:10:32.148520+0000","last_became_peered":"2024-03-19T01:10:32.148520+0000","last_unstale":"2024-03-19T01:10:34.151491+0000","last_undegraded":"2024-03-19T01:10:34.151491+0000","last_fullsized":"2024-03-19T01:10:34.151491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:10:21.211510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.950410+0000","last_change":"2024-03-19T01:10:33.182684+0000","last_active":"2024-03-19T01:10:34.950410+0000","last_peered":"2024-03-19T01:10:34.950410+0000","last_clean":"2024-03-19T01:10:34.950410+0000","last_became_active":"2024-03-19T01:10:33.182269+0000","last_became_peered":"2024-03-19T01:10:33.182269+0000","last_unstale":"2024-03-19T01:10:34.950410+0000","last_undegraded":"2024-03-19T01:10:34.950410+0000","last_fullsized":"2024-03-19T01:10:34.950410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:49:33.318462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.174629+0000","last_change":"2024-03-19T01:10:33.172795+0000","last_active":"2024-03-19T01:10:33.174629+0000","last_peered":"2024-03-19T01:10:33.174629+0000","last_clean":"2024-03-19T01:10:33.174629+0000","last_became_active":"2024-03-19T01:10:33.172483+0000","last_became_peered":"2024-03-19T01:10:33.172483+0000","last_unstale":"2024-03-19T01:10:33.174629+0000","last_undegraded":"2024-03-19T01:10:33.174629+0000","last_fullsized":"2024-03-19T01:10:33.174629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:41.767064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152455+0000","last_change":"2024-03-19T01:10:33.169130+0000","last_active":"2024-03-19T01:10:34.152455+0000","last_peered":"2024-03-19T01:10:34.152455+0000","last_clean":"2024-03-19T01:10:34.152455+0000","last_became_active":"2024-03-19T01:10:33.168763+0000","last_became_peered":"2024-03-19T01:10:33.168763+0000","last_unstale":"2024-03-19T01:10:34.152455+0000","last_undegraded":"2024-03-19T01:10:34.152455+0000","last_fullsized":"2024-03-19T01:10:34.152455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:37:23.644401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152366+0000","last_change":"2024-03-19T01:10:33.178847+0000","last_active":"2024-03-19T01:10:34.152366+0000","last_peered":"2024-03-19T01:10:34.152366+0000","last_clean":"2024-03-19T01:10:34.152366+0000","last_became_active":"2024-03-19T01:10:33.178533+0000","last_became_peered":"2024-03-19T01:10:33.178533+0000","last_unstale":"2024-03-19T01:10:34.152366+0000","last_undegraded":"2024-03-19T01:10:34.152366+0000","last_fullsized":"2024-03-19T01:10:34.152366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:52:15.401797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948621+0000","last_change":"2024-03-19T01:10:33.169936+0000","last_active":"2024-03-19T01:10:34.948621+0000","last_peered":"2024-03-19T01:10:34.948621+0000","last_clean":"2024-03-19T01:10:34.948621+0000","last_became_active":"2024-03-19T01:10:33.169594+0000","last_became_peered":"2024-03-19T01:10:33.169594+0000","last_unstale":"2024-03-19T01:10:34.948621+0000","last_undegraded":"2024-03-19T01:10:34.948621+0000","last_fullsized":"2024-03-19T01:10:34.948621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:41.932575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151417+0000","last_change":"2024-03-19T01:10:33.163701+0000","last_active":"2024-03-19T01:10:34.151417+0000","last_peered":"2024-03-19T01:10:34.151417+0000","last_clean":"2024-03-19T01:10:34.151417+0000","last_became_active":"2024-03-19T01:10:33.163374+0000","last_became_peered":"2024-03-19T01:10:33.163374+0000","last_unstale":"2024-03-19T01:10:34.151417+0000","last_undegraded":"2024-03-19T01:10:34.151417+0000","last_fullsized":"2024-03-19T01:10:34.151417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:34:26.277015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.179821+0000","last_change":"2024-03-19T01:10:33.178699+0000","last_active":"2024-03-19T01:10:33.179821+0000","last_peered":"2024-03-19T01:10:33.179821+0000","last_clean":"2024-03-19T01:10:33.179821+0000","last_became_active":"2024-03-19T01:10:33.178380+0000","last_became_peered":"2024-03-19T01:10:33.178380+0000","last_unstale":"2024-03-19T01:10:33.179821+0000","last_undegraded":"2024-03-19T01:10:33.179821+0000","last_fullsized":"2024-03-19T01:10:33.179821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:52:23.012924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152858+0000","last_change":"2024-03-19T01:10:33.179908+0000","last_active":"2024-03-19T01:10:34.152858+0000","last_peered":"2024-03-19T01:10:34.152858+0000","last_clean":"2024-03-19T01:10:34.152858+0000","last_became_active":"2024-03-19T01:10:33.179578+0000","last_became_peered":"2024-03-19T01:10:33.179578+0000","last_unstale":"2024-03-19T01:10:34.152858+0000","last_undegraded":"2024-03-19T01:10:34.152858+0000","last_fullsized":"2024-03-19T01:10:34.152858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:30:53.918078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948348+0000","last_change":"2024-03-19T01:10:33.179788+0000","last_active":"2024-03-19T01:10:34.948348+0000","last_peered":"2024-03-19T01:10:34.948348+0000","last_clean":"2024-03-19T01:10:34.948348+0000","last_became_active":"2024-03-19T01:10:33.179473+0000","last_became_peered":"2024-03-19T01:10:33.179473+0000","last_unstale":"2024-03-19T01:10:34.948348+0000","last_undegraded":"2024-03-19T01:10:34.948348+0000","last_fullsized":"2024-03-19T01:10:34.948348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:36.284768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.708993+0000","last_change":"2024-03-19T01:10:33.172698+0000","last_active":"2024-03-19T01:10:34.708993+0000","last_peered":"2024-03-19T01:10:34.708993+0000","last_clean":"2024-03-19T01:10:34.708993+0000","last_became_active":"2024-03-19T01:10:33.172369+0000","last_became_peered":"2024-03-19T01:10:33.172369+0000","last_unstale":"2024-03-19T01:10:34.708993+0000","last_undegraded":"2024-03-19T01:10:34.708993+0000","last_fullsized":"2024-03-19T01:10:34.708993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:09:18.961724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947996+0000","last_change":"2024-03-19T01:10:33.184350+0000","last_active":"2024-03-19T01:10:34.947996+0000","last_peered":"2024-03-19T01:10:34.947996+0000","last_clean":"2024-03-19T01:10:34.947996+0000","last_became_active":"2024-03-19T01:10:33.180533+0000","last_became_peered":"2024-03-19T01:10:33.180533+0000","last_unstale":"2024-03-19T01:10:34.947996+0000","last_undegraded":"2024-03-19T01:10:34.947996+0000","last_fullsized":"2024-03-19T01:10:34.947996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:28.003560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152649+0000","last_change":"2024-03-19T01:10:33.180396+0000","last_active":"2024-03-19T01:10:34.152649+0000","last_peered":"2024-03-19T01:10:34.152649+0000","last_clean":"2024-03-19T01:10:34.152649+0000","last_became_active":"2024-03-19T01:10:33.180062+0000","last_became_peered":"2024-03-19T01:10:33.180062+0000","last_unstale":"2024-03-19T01:10:34.152649+0000","last_undegraded":"2024-03-19T01:10:34.152649+0000","last_fullsized":"2024-03-19T01:10:34.152649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:15.429623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151523+0000","last_change":"2024-03-19T01:10:33.167261+0000","last_active":"2024-03-19T01:10:34.151523+0000","last_peered":"2024-03-19T01:10:34.151523+0000","last_clean":"2024-03-19T01:10:34.151523+0000","last_became_active":"2024-03-19T01:10:33.166943+0000","last_became_peered":"2024-03-19T01:10:33.166943+0000","last_unstale":"2024-03-19T01:10:34.151523+0000","last_undegraded":"2024-03-19T01:10:34.151523+0000","last_fullsized":"2024-03-19T01:10:34.151523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:01:44.451161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.176417+0000","last_change":"2024-03-19T01:10:33.175750+0000","last_active":"2024-03-19T01:10:33.176417+0000","last_peered":"2024-03-19T01:10:33.176417+0000","last_clean":"2024-03-19T01:10:33.176417+0000","last_became_active":"2024-03-19T01:10:33.175411+0000","last_became_peered":"2024-03-19T01:10:33.175411+0000","last_unstale":"2024-03-19T01:10:33.176417+0000","last_undegraded":"2024-03-19T01:10:33.176417+0000","last_fullsized":"2024-03-19T01:10:33.176417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:33:43.086238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284643,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":13451,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":158,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":64410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":901120,"data_stored":429951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1015808,"data_stored":1032203,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808856,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":439339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070493,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":909312,"data_stored":430555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332130,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":954368,"data_stored":1020901,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":901120,"data_stored":429951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":441106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":925696,"data_stored":439215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":442496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":438929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":439578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1031709,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34556,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34556,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34556,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":22647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":22867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":10339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:10:40.600 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-19T01:10:40.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.737+0000 7fd340737640 1 -- 172.21.15.116:0/1638902225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd338075860 msgr2=0x7fd338073cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.737+0000 7fd340737640 1 --2- 172.21.15.116:0/1638902225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd338075860 0x7fd338073cb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd334009930 tx=0x7fd33402bc30 comp rx=0 tx=0).stop 2024-03-19T01:10:40.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.737+0000 7fd340737640 1 -- 172.21.15.116:0/1638902225 shutdown_connections 2024-03-19T01:10:40.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.737+0000 7fd340737640 1 --2- 172.21.15.116:0/1638902225 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33810de30 0x7fd338110220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.737+0000 7fd340737640 1 --2- 172.21.15.116:0/1638902225 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd338074280 0x7fd338074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.737+0000 7fd340737640 1 --2- 172.21.15.116:0/1638902225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd338075860 0x7fd338073cb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.737+0000 7fd340737640 1 -- 172.21.15.116:0/1638902225 >> 172.21.15.116:0/1638902225 conn(0x7fd3380fcdd0 msgr2=0x7fd3380ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:40.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.737+0000 7fd340737640 1 -- 172.21.15.116:0/1638902225 shutdown_connections 2024-03-19T01:10:40.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.737+0000 7fd340737640 1 -- 172.21.15.116:0/1638902225 wait complete. 2024-03-19T01:10:40.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.737+0000 7fd340737640 1 Processor -- start 2024-03-19T01:10:40.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.737+0000 7fd340737640 1 -- start start 2024-03-19T01:10:40.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd340737640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd338074280 0x7fd33819e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:40.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd340737640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd338075860 0x7fd33819e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:40.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd340737640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33810de30 0x7fd3381a57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd340737640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd338113ad0 con 0x7fd338074280 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd340737640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd338113950 con 0x7fd33810de30 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd340737640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd338113c50 con 0x7fd338075860 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd33e4d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd338074280 0x7fd33819e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd33e4d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd338074280 0x7fd33819e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53628/0 (socket says 172.21.15.116:53628) 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd33e4d3640 1 -- 172.21.15.116:0/3125927239 learned_addr learned my addr 172.21.15.116:0/3125927239 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd33dcd2640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd338075860 0x7fd33819e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd33ecd4640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33810de30 0x7fd3381a57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd33e4d3640 1 -- 172.21.15.116:0/3125927239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd338075860 msgr2=0x7fd33819e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd33e4d3640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd338075860 0x7fd33819e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd33e4d3640 1 -- 172.21.15.116:0/3125927239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33810de30 msgr2=0x7fd3381a57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd33e4d3640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33810de30 0x7fd3381a57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd33e4d3640 1 -- 172.21.15.116:0/3125927239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3381a5e10 con 0x7fd338074280 2024-03-19T01:10:40.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd33e4d3640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd338074280 0x7fd33819e1f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd334000c00 tx=0x7fd33402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:40.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd32f7fe640 1 -- 172.21.15.116:0/3125927239 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd334041070 con 0x7fd338074280 2024-03-19T01:10:40.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd32f7fe640 1 -- 172.21.15.116:0/3125927239 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd334004450 con 0x7fd338074280 2024-03-19T01:10:40.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd340737640 1 -- 172.21.15.116:0/3125927239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3381a60a0 con 0x7fd338074280 2024-03-19T01:10:40.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd32f7fe640 1 -- 172.21.15.116:0/3125927239 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd334003df0 con 0x7fd338074280 2024-03-19T01:10:40.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.738+0000 7fd340737640 1 -- 172.21.15.116:0/3125927239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3381a65e0 con 0x7fd338074280 2024-03-19T01:10:40.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.740+0000 7fd340737640 1 -- 172.21.15.116:0/3125927239 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd338104000 con 0x7fd338074280 2024-03-19T01:10:40.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.740+0000 7fd32f7fe640 1 -- 172.21.15.116:0/3125927239 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fd334044070 con 0x7fd338074280 2024-03-19T01:10:40.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.741+0000 7fd32f7fe640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd3240a2910 0x7fd3240a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:40.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.741+0000 7fd33dcd2640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd3240a2910 0x7fd3240a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:40.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.741+0000 7fd32f7fe640 1 -- 172.21.15.116:0/3125927239 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fd3340f0370 con 0x7fd338074280 2024-03-19T01:10:40.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.741+0000 7fd33dcd2640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd3240a2910 0x7fd3240a4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd33819f700 tx=0x7fd328005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:40.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.746+0000 7fd32f7fe640 1 -- 172.21.15.116:0/3125927239 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd3340bd600 con 0x7fd338074280 2024-03-19T01:10:40.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.922+0000 7fd340737640 1 -- 172.21.15.116:0/3125927239 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd338061c60 con 0x7fd3240a2910 2024-03-19T01:10:40.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.929+0000 7fd32f7fe640 1 -- 172.21.15.116:0/3125927239 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258909 (secure 0 0 0) 0x7fd338061c60 con 0x7fd3240a2910 2024-03-19T01:10:40.933 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:40.934 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-19T01:10:40.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.932+0000 7fd340737640 1 -- 172.21.15.116:0/3125927239 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd3240a2910 msgr2=0x7fd3240a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.932+0000 7fd340737640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd3240a2910 0x7fd3240a4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd33819f700 tx=0x7fd328005f90 comp rx=0 tx=0).stop 2024-03-19T01:10:40.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.932+0000 7fd340737640 1 -- 172.21.15.116:0/3125927239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd338074280 msgr2=0x7fd33819e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:40.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.932+0000 7fd340737640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd338074280 0x7fd33819e1f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd334000c00 tx=0x7fd33402c640 comp rx=0 tx=0).stop 2024-03-19T01:10:40.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.932+0000 7fd340737640 1 -- 172.21.15.116:0/3125927239 shutdown_connections 2024-03-19T01:10:40.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.932+0000 7fd340737640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd3240a2910 0x7fd3240a4dd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.932+0000 7fd340737640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd33810de30 0x7fd3381a57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.932+0000 7fd340737640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd338075860 0x7fd33819e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.932+0000 7fd340737640 1 --2- 172.21.15.116:0/3125927239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd338074280 0x7fd33819e1f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:40.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.932+0000 7fd340737640 1 -- 172.21.15.116:0/3125927239 >> 172.21.15.116:0/3125927239 conn(0x7fd3380fcdd0 msgr2=0x7fd3381011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:40.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.932+0000 7fd340737640 1 -- 172.21.15.116:0/3125927239 shutdown_connections 2024-03-19T01:10:40.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:40.932+0000 7fd340737640 1 -- 172.21.15.116:0/3125927239 wait complete. 2024-03-19T01:10:40.944 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-03-19T01:10:39.057145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":603819,"num_objects":75,"num_object_clones":0,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":339,"num_write_kb":2989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":236,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381044,"kb_used_data":11040,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560780,"statfs":{"total":1151940427776,"available":1151550238720,"internally_reserved":0,"allocated":11304960,"data_stored":7015933,"data_compressed":108585,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13451,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":158,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.922742"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.177163+0000","last_change":"2024-03-19T01:10:33.176468+0000","last_active":"2024-03-19T01:10:33.177163+0000","last_peered":"2024-03-19T01:10:33.177163+0000","last_clean":"2024-03-19T01:10:33.177163+0000","last_became_active":"2024-03-19T01:10:33.175995+0000","last_became_peered":"2024-03-19T01:10:33.175995+0000","last_unstale":"2024-03-19T01:10:33.177163+0000","last_undegraded":"2024-03-19T01:10:33.177163+0000","last_fullsized":"2024-03-19T01:10:33.177163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:02:35.880821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948055+0000","last_change":"2024-03-19T01:10:33.171253+0000","last_active":"2024-03-19T01:10:34.948055+0000","last_peered":"2024-03-19T01:10:34.948055+0000","last_clean":"2024-03-19T01:10:34.948055+0000","last_became_active":"2024-03-19T01:10:33.170330+0000","last_became_peered":"2024-03-19T01:10:33.170330+0000","last_unstale":"2024-03-19T01:10:34.948055+0000","last_undegraded":"2024-03-19T01:10:34.948055+0000","last_fullsized":"2024-03-19T01:10:34.948055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:38.905220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947516+0000","last_change":"2024-03-19T01:10:33.178006+0000","last_active":"2024-03-19T01:10:34.947516+0000","last_peered":"2024-03-19T01:10:34.947516+0000","last_clean":"2024-03-19T01:10:34.947516+0000","last_became_active":"2024-03-19T01:10:33.177662+0000","last_became_peered":"2024-03-19T01:10:33.177662+0000","last_unstale":"2024-03-19T01:10:34.947516+0000","last_undegraded":"2024-03-19T01:10:34.947516+0000","last_fullsized":"2024-03-19T01:10:34.947516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:08.980220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.946846+0000","last_change":"2024-03-19T01:10:33.178805+0000","last_active":"2024-03-19T01:10:34.946846+0000","last_peered":"2024-03-19T01:10:34.946846+0000","last_clean":"2024-03-19T01:10:34.946846+0000","last_became_active":"2024-03-19T01:10:33.178385+0000","last_became_peered":"2024-03-19T01:10:33.178385+0000","last_unstale":"2024-03-19T01:10:34.946846+0000","last_undegraded":"2024-03-19T01:10:34.946846+0000","last_fullsized":"2024-03-19T01:10:34.946846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:06.454345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.679378+0000","last_change":"2024-03-19T01:10:33.183110+0000","last_active":"2024-03-19T01:10:34.679378+0000","last_peered":"2024-03-19T01:10:34.679378+0000","last_clean":"2024-03-19T01:10:34.679378+0000","last_became_active":"2024-03-19T01:10:33.180036+0000","last_became_peered":"2024-03-19T01:10:33.180036+0000","last_unstale":"2024-03-19T01:10:34.679378+0000","last_undegraded":"2024-03-19T01:10:34.679378+0000","last_fullsized":"2024-03-19T01:10:34.679378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:57:51.397156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.946000+0000","last_change":"2024-03-19T01:10:33.164798+0000","last_active":"2024-03-19T01:10:34.946000+0000","last_peered":"2024-03-19T01:10:34.946000+0000","last_clean":"2024-03-19T01:10:34.946000+0000","last_became_active":"2024-03-19T01:10:33.164430+0000","last_became_peered":"2024-03-19T01:10:33.164430+0000","last_unstale":"2024-03-19T01:10:34.946000+0000","last_undegraded":"2024-03-19T01:10:34.946000+0000","last_fullsized":"2024-03-19T01:10:34.946000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:22:39.679057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948918+0000","last_change":"2024-03-19T01:10:33.172946+0000","last_active":"2024-03-19T01:10:34.948918+0000","last_peered":"2024-03-19T01:10:34.948918+0000","last_clean":"2024-03-19T01:10:34.948918+0000","last_became_active":"2024-03-19T01:10:33.172617+0000","last_became_peered":"2024-03-19T01:10:33.172617+0000","last_unstale":"2024-03-19T01:10:34.948918+0000","last_undegraded":"2024-03-19T01:10:34.948918+0000","last_fullsized":"2024-03-19T01:10:34.948918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:18:20.311445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.946954+0000","last_change":"2024-03-19T01:10:33.178162+0000","last_active":"2024-03-19T01:10:34.946954+0000","last_peered":"2024-03-19T01:10:34.946954+0000","last_clean":"2024-03-19T01:10:34.946954+0000","last_became_active":"2024-03-19T01:10:33.177752+0000","last_became_peered":"2024-03-19T01:10:33.177752+0000","last_unstale":"2024-03-19T01:10:34.946954+0000","last_undegraded":"2024-03-19T01:10:34.946954+0000","last_fullsized":"2024-03-19T01:10:34.946954+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:01:35.681694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.153185+0000","last_change":"2024-03-19T01:10:33.175619+0000","last_active":"2024-03-19T01:10:34.153185+0000","last_peered":"2024-03-19T01:10:34.153185+0000","last_clean":"2024-03-19T01:10:34.153185+0000","last_became_active":"2024-03-19T01:10:33.175017+0000","last_became_peered":"2024-03-19T01:10:33.175017+0000","last_unstale":"2024-03-19T01:10:34.153185+0000","last_undegraded":"2024-03-19T01:10:34.153185+0000","last_fullsized":"2024-03-19T01:10:34.153185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:49:52.019461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.946388+0000","last_change":"2024-03-19T01:10:33.173035+0000","last_active":"2024-03-19T01:10:34.946388+0000","last_peered":"2024-03-19T01:10:34.946388+0000","last_clean":"2024-03-19T01:10:34.946388+0000","last_became_active":"2024-03-19T01:10:33.172717+0000","last_became_peered":"2024-03-19T01:10:33.172717+0000","last_unstale":"2024-03-19T01:10:34.946388+0000","last_undegraded":"2024-03-19T01:10:34.946388+0000","last_fullsized":"2024-03-19T01:10:34.946388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:38.078147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949793+0000","last_change":"2024-03-19T01:10:33.182679+0000","last_active":"2024-03-19T01:10:34.949793+0000","last_peered":"2024-03-19T01:10:34.949793+0000","last_clean":"2024-03-19T01:10:34.949793+0000","last_became_active":"2024-03-19T01:10:33.182384+0000","last_became_peered":"2024-03-19T01:10:33.182384+0000","last_unstale":"2024-03-19T01:10:34.949793+0000","last_undegraded":"2024-03-19T01:10:34.949793+0000","last_fullsized":"2024-03-19T01:10:34.949793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:37:26.542393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.946688+0000","last_change":"2024-03-19T01:10:33.178818+0000","last_active":"2024-03-19T01:10:34.946688+0000","last_peered":"2024-03-19T01:10:34.946688+0000","last_clean":"2024-03-19T01:10:34.946688+0000","last_became_active":"2024-03-19T01:10:33.178286+0000","last_became_peered":"2024-03-19T01:10:33.178286+0000","last_unstale":"2024-03-19T01:10:34.946688+0000","last_undegraded":"2024-03-19T01:10:34.946688+0000","last_fullsized":"2024-03-19T01:10:34.946688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:05:19.473738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151459+0000","last_change":"2024-03-19T01:10:33.157042+0000","last_active":"2024-03-19T01:10:34.151459+0000","last_peered":"2024-03-19T01:10:34.151459+0000","last_clean":"2024-03-19T01:10:34.151459+0000","last_became_active":"2024-03-19T01:10:33.156576+0000","last_became_peered":"2024-03-19T01:10:33.156576+0000","last_unstale":"2024-03-19T01:10:34.151459+0000","last_undegraded":"2024-03-19T01:10:34.151459+0000","last_fullsized":"2024-03-19T01:10:34.151459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:51.625850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947761+0000","last_change":"2024-03-19T01:10:33.177981+0000","last_active":"2024-03-19T01:10:34.947761+0000","last_peered":"2024-03-19T01:10:34.947761+0000","last_clean":"2024-03-19T01:10:34.947761+0000","last_became_active":"2024-03-19T01:10:33.177634+0000","last_became_peered":"2024-03-19T01:10:33.177634+0000","last_unstale":"2024-03-19T01:10:34.947761+0000","last_undegraded":"2024-03-19T01:10:34.947761+0000","last_fullsized":"2024-03-19T01:10:34.947761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:49.646938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947160+0000","last_change":"2024-03-19T01:10:33.168982+0000","last_active":"2024-03-19T01:10:34.947160+0000","last_peered":"2024-03-19T01:10:34.947160+0000","last_clean":"2024-03-19T01:10:34.947160+0000","last_became_active":"2024-03-19T01:10:33.168569+0000","last_became_peered":"2024-03-19T01:10:33.168569+0000","last_unstale":"2024-03-19T01:10:34.947160+0000","last_undegraded":"2024-03-19T01:10:34.947160+0000","last_fullsized":"2024-03-19T01:10:34.947160+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:43.389585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152470+0000","last_change":"2024-03-19T01:10:33.184056+0000","last_active":"2024-03-19T01:10:34.152470+0000","last_peered":"2024-03-19T01:10:34.152470+0000","last_clean":"2024-03-19T01:10:34.152470+0000","last_became_active":"2024-03-19T01:10:33.183712+0000","last_became_peered":"2024-03-19T01:10:33.183712+0000","last_unstale":"2024-03-19T01:10:34.152470+0000","last_undegraded":"2024-03-19T01:10:34.152470+0000","last_fullsized":"2024-03-19T01:10:34.152470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:27.268859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.180748+0000","last_change":"2024-03-19T01:10:33.180114+0000","last_active":"2024-03-19T01:10:33.180748+0000","last_peered":"2024-03-19T01:10:33.180748+0000","last_clean":"2024-03-19T01:10:33.180748+0000","last_became_active":"2024-03-19T01:10:33.179796+0000","last_became_peered":"2024-03-19T01:10:33.179796+0000","last_unstale":"2024-03-19T01:10:33.180748+0000","last_undegraded":"2024-03-19T01:10:33.180748+0000","last_fullsized":"2024-03-19T01:10:33.180748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:10:30.508075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947631+0000","last_change":"2024-03-19T01:10:33.179888+0000","last_active":"2024-03-19T01:10:34.947631+0000","last_peered":"2024-03-19T01:10:34.947631+0000","last_clean":"2024-03-19T01:10:34.947631+0000","last_became_active":"2024-03-19T01:10:33.179562+0000","last_became_peered":"2024-03-19T01:10:33.179562+0000","last_unstale":"2024-03-19T01:10:34.947631+0000","last_undegraded":"2024-03-19T01:10:34.947631+0000","last_fullsized":"2024-03-19T01:10:34.947631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:53.346043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152674+0000","last_change":"2024-03-19T01:10:33.177619+0000","last_active":"2024-03-19T01:10:34.152674+0000","last_peered":"2024-03-19T01:10:34.152674+0000","last_clean":"2024-03-19T01:10:34.152674+0000","last_became_active":"2024-03-19T01:10:33.176002+0000","last_became_peered":"2024-03-19T01:10:33.176002+0000","last_unstale":"2024-03-19T01:10:34.152674+0000","last_undegraded":"2024-03-19T01:10:34.152674+0000","last_fullsized":"2024-03-19T01:10:34.152674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:50.637964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'11","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.981539+0000","last_change":"2024-03-19T01:10:32.151875+0000","last_active":"2024-03-19T01:10:36.981539+0000","last_peered":"2024-03-19T01:10:36.981539+0000","last_clean":"2024-03-19T01:10:36.981539+0000","last_became_active":"2024-03-19T01:10:32.151448+0000","last_became_peered":"2024-03-19T01:10:32.151448+0000","last_unstale":"2024-03-19T01:10:36.981539+0000","last_undegraded":"2024-03-19T01:10:36.981539+0000","last_fullsized":"2024-03-19T01:10:36.981539+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:17.930078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947663+0000","last_change":"2024-03-19T01:10:33.184258+0000","last_active":"2024-03-19T01:10:34.947663+0000","last_peered":"2024-03-19T01:10:34.947663+0000","last_clean":"2024-03-19T01:10:34.947663+0000","last_became_active":"2024-03-19T01:10:33.183899+0000","last_became_peered":"2024-03-19T01:10:33.183899+0000","last_unstale":"2024-03-19T01:10:34.947663+0000","last_undegraded":"2024-03-19T01:10:34.947663+0000","last_fullsized":"2024-03-19T01:10:34.947663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:47.220818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948067+0000","last_change":"2024-03-19T01:10:32.151998+0000","last_active":"2024-03-19T01:10:34.948067+0000","last_peered":"2024-03-19T01:10:34.948067+0000","last_clean":"2024-03-19T01:10:34.948067+0000","last_became_active":"2024-03-19T01:10:32.151644+0000","last_became_peered":"2024-03-19T01:10:32.151644+0000","last_unstale":"2024-03-19T01:10:34.948067+0000","last_undegraded":"2024-03-19T01:10:34.948067+0000","last_fullsized":"2024-03-19T01:10:34.948067+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:20:28.660705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948099+0000","last_change":"2024-03-19T01:10:33.176759+0000","last_active":"2024-03-19T01:10:34.948099+0000","last_peered":"2024-03-19T01:10:34.948099+0000","last_clean":"2024-03-19T01:10:34.948099+0000","last_became_active":"2024-03-19T01:10:33.176409+0000","last_became_peered":"2024-03-19T01:10:33.176409+0000","last_unstale":"2024-03-19T01:10:34.948099+0000","last_undegraded":"2024-03-19T01:10:34.948099+0000","last_fullsized":"2024-03-19T01:10:34.948099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:21:08.513411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.679443+0000","last_change":"2024-03-19T01:10:32.149984+0000","last_active":"2024-03-19T01:10:34.679443+0000","last_peered":"2024-03-19T01:10:34.679443+0000","last_clean":"2024-03-19T01:10:34.679443+0000","last_became_active":"2024-03-19T01:10:32.149292+0000","last_became_peered":"2024-03-19T01:10:32.149292+0000","last_unstale":"2024-03-19T01:10:34.679443+0000","last_undegraded":"2024-03-19T01:10:34.679443+0000","last_fullsized":"2024-03-19T01:10:34.679443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:29:23.226555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152831+0000","last_change":"2024-03-19T01:10:33.174475+0000","last_active":"2024-03-19T01:10:34.152831+0000","last_peered":"2024-03-19T01:10:34.152831+0000","last_clean":"2024-03-19T01:10:34.152831+0000","last_became_active":"2024-03-19T01:10:33.174154+0000","last_became_peered":"2024-03-19T01:10:33.174154+0000","last_unstale":"2024-03-19T01:10:34.152831+0000","last_undegraded":"2024-03-19T01:10:34.152831+0000","last_fullsized":"2024-03-19T01:10:34.152831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:17.874609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.679352+0000","last_change":"2024-03-19T01:10:32.151717+0000","last_active":"2024-03-19T01:10:34.679352+0000","last_peered":"2024-03-19T01:10:34.679352+0000","last_clean":"2024-03-19T01:10:34.679352+0000","last_became_active":"2024-03-19T01:10:32.151356+0000","last_became_peered":"2024-03-19T01:10:32.151356+0000","last_unstale":"2024-03-19T01:10:34.679352+0000","last_undegraded":"2024-03-19T01:10:34.679352+0000","last_fullsized":"2024-03-19T01:10:34.679352+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:02.614226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.709298+0000","last_change":"2024-03-19T01:10:33.177202+0000","last_active":"2024-03-19T01:10:34.709298+0000","last_peered":"2024-03-19T01:10:34.709298+0000","last_clean":"2024-03-19T01:10:34.709298+0000","last_became_active":"2024-03-19T01:10:33.172004+0000","last_became_peered":"2024-03-19T01:10:33.172004+0000","last_unstale":"2024-03-19T01:10:34.709298+0000","last_undegraded":"2024-03-19T01:10:34.709298+0000","last_fullsized":"2024-03-19T01:10:34.709298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:11:36.610997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.951792+0000","last_change":"2024-03-19T01:10:32.151138+0000","last_active":"2024-03-19T01:10:34.951792+0000","last_peered":"2024-03-19T01:10:34.951792+0000","last_clean":"2024-03-19T01:10:34.951792+0000","last_became_active":"2024-03-19T01:10:32.150297+0000","last_became_peered":"2024-03-19T01:10:32.150297+0000","last_unstale":"2024-03-19T01:10:34.951792+0000","last_undegraded":"2024-03-19T01:10:34.951792+0000","last_fullsized":"2024-03-19T01:10:34.951792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:32:37.104986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.679207+0000","last_change":"2024-03-19T01:10:33.178878+0000","last_active":"2024-03-19T01:10:34.679207+0000","last_peered":"2024-03-19T01:10:34.679207+0000","last_clean":"2024-03-19T01:10:34.679207+0000","last_became_active":"2024-03-19T01:10:33.178526+0000","last_became_peered":"2024-03-19T01:10:33.178526+0000","last_unstale":"2024-03-19T01:10:34.679207+0000","last_undegraded":"2024-03-19T01:10:34.679207+0000","last_fullsized":"2024-03-19T01:10:34.679207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:40.074065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.980389+0000","last_change":"2024-03-19T01:10:32.153255+0000","last_active":"2024-03-19T01:10:36.980389+0000","last_peered":"2024-03-19T01:10:36.980389+0000","last_clean":"2024-03-19T01:10:36.980389+0000","last_became_active":"2024-03-19T01:10:32.152976+0000","last_became_peered":"2024-03-19T01:10:32.152976+0000","last_unstale":"2024-03-19T01:10:36.980389+0000","last_undegraded":"2024-03-19T01:10:36.980389+0000","last_fullsized":"2024-03-19T01:10:36.980389+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:51:00.313688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.709146+0000","last_change":"2024-03-19T01:10:33.173710+0000","last_active":"2024-03-19T01:10:34.709146+0000","last_peered":"2024-03-19T01:10:34.709146+0000","last_clean":"2024-03-19T01:10:34.709146+0000","last_became_active":"2024-03-19T01:10:33.173325+0000","last_became_peered":"2024-03-19T01:10:33.173325+0000","last_unstale":"2024-03-19T01:10:34.709146+0000","last_undegraded":"2024-03-19T01:10:34.709146+0000","last_fullsized":"2024-03-19T01:10:34.709146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:21.755723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151265+0000","last_change":"2024-03-19T01:10:32.144286+0000","last_active":"2024-03-19T01:10:34.151265+0000","last_peered":"2024-03-19T01:10:34.151265+0000","last_clean":"2024-03-19T01:10:34.151265+0000","last_became_active":"2024-03-19T01:10:32.143959+0000","last_became_peered":"2024-03-19T01:10:32.143959+0000","last_unstale":"2024-03-19T01:10:34.151265+0000","last_undegraded":"2024-03-19T01:10:34.151265+0000","last_fullsized":"2024-03-19T01:10:34.151265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:46:15.750018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947048+0000","last_change":"2024-03-19T01:10:33.178122+0000","last_active":"2024-03-19T01:10:34.947048+0000","last_peered":"2024-03-19T01:10:34.947048+0000","last_clean":"2024-03-19T01:10:34.947048+0000","last_became_active":"2024-03-19T01:10:33.175491+0000","last_became_peered":"2024-03-19T01:10:33.175491+0000","last_unstale":"2024-03-19T01:10:34.947048+0000","last_undegraded":"2024-03-19T01:10:34.947048+0000","last_fullsized":"2024-03-19T01:10:34.947048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:34.418463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.980427+0000","last_change":"2024-03-19T01:10:32.147561+0000","last_active":"2024-03-19T01:10:36.980427+0000","last_peered":"2024-03-19T01:10:36.980427+0000","last_clean":"2024-03-19T01:10:36.980427+0000","last_became_active":"2024-03-19T01:10:32.146825+0000","last_became_peered":"2024-03-19T01:10:32.146825+0000","last_unstale":"2024-03-19T01:10:36.980427+0000","last_undegraded":"2024-03-19T01:10:36.980427+0000","last_fullsized":"2024-03-19T01:10:36.980427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:20:09.833960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948195+0000","last_change":"2024-03-19T01:10:33.170492+0000","last_active":"2024-03-19T01:10:34.948195+0000","last_peered":"2024-03-19T01:10:34.948195+0000","last_clean":"2024-03-19T01:10:34.948195+0000","last_became_active":"2024-03-19T01:10:33.170013+0000","last_became_peered":"2024-03-19T01:10:33.170013+0000","last_unstale":"2024-03-19T01:10:34.948195+0000","last_undegraded":"2024-03-19T01:10:34.948195+0000","last_fullsized":"2024-03-19T01:10:34.948195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:13.046663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.950824+0000","last_change":"2024-03-19T01:10:32.147060+0000","last_active":"2024-03-19T01:10:34.950824+0000","last_peered":"2024-03-19T01:10:34.950824+0000","last_clean":"2024-03-19T01:10:34.950824+0000","last_became_active":"2024-03-19T01:10:32.146721+0000","last_became_peered":"2024-03-19T01:10:32.146721+0000","last_unstale":"2024-03-19T01:10:34.950824+0000","last_undegraded":"2024-03-19T01:10:34.950824+0000","last_fullsized":"2024-03-19T01:10:34.950824+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:13:16.930746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152960+0000","last_change":"2024-03-19T01:10:33.174431+0000","last_active":"2024-03-19T01:10:34.152960+0000","last_peered":"2024-03-19T01:10:34.152960+0000","last_clean":"2024-03-19T01:10:34.152960+0000","last_became_active":"2024-03-19T01:10:33.174040+0000","last_became_peered":"2024-03-19T01:10:33.174040+0000","last_unstale":"2024-03-19T01:10:34.152960+0000","last_undegraded":"2024-03-19T01:10:34.152960+0000","last_fullsized":"2024-03-19T01:10:34.152960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:53.850400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.951917+0000","last_change":"2024-03-19T01:10:32.149194+0000","last_active":"2024-03-19T01:10:34.951917+0000","last_peered":"2024-03-19T01:10:34.951917+0000","last_clean":"2024-03-19T01:10:34.951917+0000","last_became_active":"2024-03-19T01:10:32.148835+0000","last_became_peered":"2024-03-19T01:10:32.148835+0000","last_unstale":"2024-03-19T01:10:34.951917+0000","last_undegraded":"2024-03-19T01:10:34.951917+0000","last_fullsized":"2024-03-19T01:10:34.951917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:55.697445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948238+0000","last_change":"2024-03-19T01:10:33.180203+0000","last_active":"2024-03-19T01:10:34.948238+0000","last_peered":"2024-03-19T01:10:34.948238+0000","last_clean":"2024-03-19T01:10:34.948238+0000","last_became_active":"2024-03-19T01:10:33.179812+0000","last_became_peered":"2024-03-19T01:10:33.179812+0000","last_unstale":"2024-03-19T01:10:34.948238+0000","last_undegraded":"2024-03-19T01:10:34.948238+0000","last_fullsized":"2024-03-19T01:10:34.948238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:38.371706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.950503+0000","last_change":"2024-03-19T01:10:32.153852+0000","last_active":"2024-03-19T01:10:34.950503+0000","last_peered":"2024-03-19T01:10:34.950503+0000","last_clean":"2024-03-19T01:10:34.950503+0000","last_became_active":"2024-03-19T01:10:32.153537+0000","last_became_peered":"2024-03-19T01:10:32.153537+0000","last_unstale":"2024-03-19T01:10:34.950503+0000","last_undegraded":"2024-03-19T01:10:34.950503+0000","last_fullsized":"2024-03-19T01:10:34.950503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:48.544344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152843+0000","last_change":"2024-03-19T01:10:33.175629+0000","last_active":"2024-03-19T01:10:34.152843+0000","last_peered":"2024-03-19T01:10:34.152843+0000","last_clean":"2024-03-19T01:10:34.152843+0000","last_became_active":"2024-03-19T01:10:33.175278+0000","last_became_peered":"2024-03-19T01:10:33.175278+0000","last_unstale":"2024-03-19T01:10:34.152843+0000","last_undegraded":"2024-03-19T01:10:34.152843+0000","last_fullsized":"2024-03-19T01:10:34.152843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:42:31.534216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'13","reported_seq":28,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:37.988224+0000","last_change":"2024-03-19T01:10:32.148290+0000","last_active":"2024-03-19T01:10:37.988224+0000","last_peered":"2024-03-19T01:10:37.988224+0000","last_clean":"2024-03-19T01:10:37.988224+0000","last_became_active":"2024-03-19T01:10:32.147940+0000","last_became_peered":"2024-03-19T01:10:32.147940+0000","last_unstale":"2024-03-19T01:10:37.988224+0000","last_undegraded":"2024-03-19T01:10:37.988224+0000","last_fullsized":"2024-03-19T01:10:37.988224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:40:58.125706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8212,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.679313+0000","last_change":"2024-03-19T01:10:33.174723+0000","last_active":"2024-03-19T01:10:34.679313+0000","last_peered":"2024-03-19T01:10:34.679313+0000","last_clean":"2024-03-19T01:10:34.679313+0000","last_became_active":"2024-03-19T01:10:33.174369+0000","last_became_peered":"2024-03-19T01:10:33.174369+0000","last_unstale":"2024-03-19T01:10:34.679313+0000","last_undegraded":"2024-03-19T01:10:34.679313+0000","last_fullsized":"2024-03-19T01:10:34.679313+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:33.641050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:37.987483+0000","last_change":"2024-03-19T01:10:32.146760+0000","last_active":"2024-03-19T01:10:37.987483+0000","last_peered":"2024-03-19T01:10:37.987483+0000","last_clean":"2024-03-19T01:10:37.987483+0000","last_became_active":"2024-03-19T01:10:32.145610+0000","last_became_peered":"2024-03-19T01:10:32.145610+0000","last_unstale":"2024-03-19T01:10:37.987483+0000","last_undegraded":"2024-03-19T01:10:37.987483+0000","last_fullsized":"2024-03-19T01:10:37.987483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:10:10.959609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151336+0000","last_change":"2024-03-19T01:10:33.176948+0000","last_active":"2024-03-19T01:10:34.151336+0000","last_peered":"2024-03-19T01:10:34.151336+0000","last_clean":"2024-03-19T01:10:34.151336+0000","last_became_active":"2024-03-19T01:10:33.176604+0000","last_became_peered":"2024-03-19T01:10:33.176604+0000","last_unstale":"2024-03-19T01:10:34.151336+0000","last_undegraded":"2024-03-19T01:10:34.151336+0000","last_fullsized":"2024-03-19T01:10:34.151336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:45:45.032639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.980809+0000","last_change":"2024-03-19T01:10:32.152397+0000","last_active":"2024-03-19T01:10:36.980809+0000","last_peered":"2024-03-19T01:10:36.980809+0000","last_clean":"2024-03-19T01:10:36.980809+0000","last_became_active":"2024-03-19T01:10:32.152026+0000","last_became_peered":"2024-03-19T01:10:32.152026+0000","last_unstale":"2024-03-19T01:10:36.980809+0000","last_undegraded":"2024-03-19T01:10:36.980809+0000","last_fullsized":"2024-03-19T01:10:36.980809+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:47.789504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949920+0000","last_change":"2024-03-19T01:10:33.168529+0000","last_active":"2024-03-19T01:10:34.949920+0000","last_peered":"2024-03-19T01:10:34.949920+0000","last_clean":"2024-03-19T01:10:34.949920+0000","last_became_active":"2024-03-19T01:10:33.168198+0000","last_became_peered":"2024-03-19T01:10:33.168198+0000","last_unstale":"2024-03-19T01:10:34.949920+0000","last_undegraded":"2024-03-19T01:10:34.949920+0000","last_fullsized":"2024-03-19T01:10:34.949920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:36.030791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.979607+0000","last_change":"2024-03-19T01:10:32.145565+0000","last_active":"2024-03-19T01:10:36.979607+0000","last_peered":"2024-03-19T01:10:36.979607+0000","last_clean":"2024-03-19T01:10:36.979607+0000","last_became_active":"2024-03-19T01:10:32.145234+0000","last_became_peered":"2024-03-19T01:10:32.145234+0000","last_unstale":"2024-03-19T01:10:36.979607+0000","last_undegraded":"2024-03-19T01:10:36.979607+0000","last_fullsized":"2024-03-19T01:10:36.979607+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:35.563136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949518+0000","last_change":"2024-03-19T01:10:33.182001+0000","last_active":"2024-03-19T01:10:34.949518+0000","last_peered":"2024-03-19T01:10:34.949518+0000","last_clean":"2024-03-19T01:10:34.949518+0000","last_became_active":"2024-03-19T01:10:33.181638+0000","last_became_peered":"2024-03-19T01:10:33.181638+0000","last_unstale":"2024-03-19T01:10:34.949518+0000","last_undegraded":"2024-03-19T01:10:34.949518+0000","last_fullsized":"2024-03-19T01:10:34.949518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:59:23.155190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'10","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.951212+0000","last_change":"2024-03-19T01:10:32.154159+0000","last_active":"2024-03-19T01:10:34.951212+0000","last_peered":"2024-03-19T01:10:34.951212+0000","last_clean":"2024-03-19T01:10:34.951212+0000","last_became_active":"2024-03-19T01:10:32.153853+0000","last_became_peered":"2024-03-19T01:10:32.153853+0000","last_unstale":"2024-03-19T01:10:34.951212+0000","last_undegraded":"2024-03-19T01:10:34.951212+0000","last_fullsized":"2024-03-19T01:10:34.951212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:41:13.727588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.709146+0000","last_change":"2024-03-19T01:10:33.176659+0000","last_active":"2024-03-19T01:10:34.709146+0000","last_peered":"2024-03-19T01:10:34.709146+0000","last_clean":"2024-03-19T01:10:34.709146+0000","last_became_active":"2024-03-19T01:10:33.176319+0000","last_became_peered":"2024-03-19T01:10:33.176319+0000","last_unstale":"2024-03-19T01:10:34.709146+0000","last_undegraded":"2024-03-19T01:10:34.709146+0000","last_fullsized":"2024-03-19T01:10:34.709146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:00.311883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.679367+0000","last_change":"2024-03-19T01:10:32.144855+0000","last_active":"2024-03-19T01:10:34.679367+0000","last_peered":"2024-03-19T01:10:34.679367+0000","last_clean":"2024-03-19T01:10:34.679367+0000","last_became_active":"2024-03-19T01:10:32.144540+0000","last_became_peered":"2024-03-19T01:10:32.144540+0000","last_unstale":"2024-03-19T01:10:34.679367+0000","last_undegraded":"2024-03-19T01:10:34.679367+0000","last_fullsized":"2024-03-19T01:10:34.679367+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:22.399954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.174270+0000","last_change":"2024-03-19T01:10:33.172739+0000","last_active":"2024-03-19T01:10:33.174270+0000","last_peered":"2024-03-19T01:10:33.174270+0000","last_clean":"2024-03-19T01:10:33.174270+0000","last_became_active":"2024-03-19T01:10:33.172391+0000","last_became_peered":"2024-03-19T01:10:33.172391+0000","last_unstale":"2024-03-19T01:10:33.174270+0000","last_undegraded":"2024-03-19T01:10:33.174270+0000","last_fullsized":"2024-03-19T01:10:33.174270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:15:47.775133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947437+0000","last_change":"2024-03-19T01:10:32.144491+0000","last_active":"2024-03-19T01:10:34.947437+0000","last_peered":"2024-03-19T01:10:34.947437+0000","last_clean":"2024-03-19T01:10:34.947437+0000","last_became_active":"2024-03-19T01:10:32.144170+0000","last_became_peered":"2024-03-19T01:10:32.144170+0000","last_unstale":"2024-03-19T01:10:34.947437+0000","last_undegraded":"2024-03-19T01:10:34.947437+0000","last_fullsized":"2024-03-19T01:10:34.947437+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:07.998818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947282+0000","last_change":"2024-03-19T01:10:33.175479+0000","last_active":"2024-03-19T01:10:34.947282+0000","last_peered":"2024-03-19T01:10:34.947282+0000","last_clean":"2024-03-19T01:10:34.947282+0000","last_became_active":"2024-03-19T01:10:33.172530+0000","last_became_peered":"2024-03-19T01:10:33.172530+0000","last_unstale":"2024-03-19T01:10:34.947282+0000","last_undegraded":"2024-03-19T01:10:34.947282+0000","last_fullsized":"2024-03-19T01:10:34.947282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:16.058826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949212+0000","last_change":"2024-03-19T01:10:32.143629+0000","last_active":"2024-03-19T01:10:34.949212+0000","last_peered":"2024-03-19T01:10:34.949212+0000","last_clean":"2024-03-19T01:10:34.949212+0000","last_became_active":"2024-03-19T01:10:32.143321+0000","last_became_peered":"2024-03-19T01:10:32.143321+0000","last_unstale":"2024-03-19T01:10:34.949212+0000","last_undegraded":"2024-03-19T01:10:34.949212+0000","last_fullsized":"2024-03-19T01:10:34.949212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:47.404245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.946621+0000","last_change":"2024-03-19T01:10:33.171284+0000","last_active":"2024-03-19T01:10:34.946621+0000","last_peered":"2024-03-19T01:10:34.946621+0000","last_clean":"2024-03-19T01:10:34.946621+0000","last_became_active":"2024-03-19T01:10:33.168717+0000","last_became_peered":"2024-03-19T01:10:33.168717+0000","last_unstale":"2024-03-19T01:10:34.946621+0000","last_undegraded":"2024-03-19T01:10:34.946621+0000","last_fullsized":"2024-03-19T01:10:34.946621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:52.335303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.709185+0000","last_change":"2024-03-19T01:10:33.177261+0000","last_active":"2024-03-19T01:10:34.709185+0000","last_peered":"2024-03-19T01:10:34.709185+0000","last_clean":"2024-03-19T01:10:34.709185+0000","last_became_active":"2024-03-19T01:10:33.171836+0000","last_became_peered":"2024-03-19T01:10:33.171836+0000","last_unstale":"2024-03-19T01:10:34.709185+0000","last_undegraded":"2024-03-19T01:10:34.709185+0000","last_fullsized":"2024-03-19T01:10:34.709185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:49:14.577925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151163+0000","last_change":"2024-03-19T01:10:32.141587+0000","last_active":"2024-03-19T01:10:34.151163+0000","last_peered":"2024-03-19T01:10:34.151163+0000","last_clean":"2024-03-19T01:10:34.151163+0000","last_became_active":"2024-03-19T01:10:32.141236+0000","last_became_peered":"2024-03-19T01:10:32.141236+0000","last_unstale":"2024-03-19T01:10:34.151163+0000","last_undegraded":"2024-03-19T01:10:34.151163+0000","last_fullsized":"2024-03-19T01:10:34.151163+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:07:40.685399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948028+0000","last_change":"2024-03-19T01:10:33.174761+0000","last_active":"2024-03-19T01:10:34.948028+0000","last_peered":"2024-03-19T01:10:34.948028+0000","last_clean":"2024-03-19T01:10:34.948028+0000","last_became_active":"2024-03-19T01:10:33.174431+0000","last_became_peered":"2024-03-19T01:10:33.174431+0000","last_unstale":"2024-03-19T01:10:34.948028+0000","last_undegraded":"2024-03-19T01:10:34.948028+0000","last_fullsized":"2024-03-19T01:10:34.948028+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:21.613055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.978591+0000","last_change":"2024-03-19T01:10:32.145816+0000","last_active":"2024-03-19T01:10:36.978591+0000","last_peered":"2024-03-19T01:10:36.978591+0000","last_clean":"2024-03-19T01:10:36.978591+0000","last_became_active":"2024-03-19T01:10:32.145055+0000","last_became_peered":"2024-03-19T01:10:32.145055+0000","last_unstale":"2024-03-19T01:10:36.978591+0000","last_undegraded":"2024-03-19T01:10:36.978591+0000","last_fullsized":"2024-03-19T01:10:36.978591+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:56.428854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":338,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949358+0000","last_change":"2024-03-19T01:09:20.932686+0000","last_active":"2024-03-19T01:10:34.949358+0000","last_peered":"2024-03-19T01:10:34.949358+0000","last_clean":"2024-03-19T01:10:34.949358+0000","last_became_active":"2024-03-19T01:09:20.932170+0000","last_became_peered":"2024-03-19T01:09:20.932170+0000","last_unstale":"2024-03-19T01:10:34.949358+0000","last_undegraded":"2024-03-19T01:10:34.949358+0000","last_fullsized":"2024-03-19T01:10:34.949358+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:05:53.937682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:05:53.937682+0000","last_clean_scrub_stamp":"2024-03-19T01:05:53.937682+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:41.523200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151263+0000","last_change":"2024-03-19T01:10:33.173230+0000","last_active":"2024-03-19T01:10:34.151263+0000","last_peered":"2024-03-19T01:10:34.151263+0000","last_clean":"2024-03-19T01:10:34.151263+0000","last_became_active":"2024-03-19T01:10:33.172858+0000","last_became_peered":"2024-03-19T01:10:33.172858+0000","last_unstale":"2024-03-19T01:10:34.151263+0000","last_undegraded":"2024-03-19T01:10:34.151263+0000","last_fullsized":"2024-03-19T01:10:34.151263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:43.582211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.952180+0000","last_change":"2024-03-19T01:10:32.147419+0000","last_active":"2024-03-19T01:10:34.952180+0000","last_peered":"2024-03-19T01:10:34.952180+0000","last_clean":"2024-03-19T01:10:34.952180+0000","last_became_active":"2024-03-19T01:10:32.147111+0000","last_became_peered":"2024-03-19T01:10:32.147111+0000","last_unstale":"2024-03-19T01:10:34.952180+0000","last_undegraded":"2024-03-19T01:10:34.952180+0000","last_fullsized":"2024-03-19T01:10:34.952180+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:45:34.989763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.950168+0000","last_change":"2024-03-19T01:10:32.154233+0000","last_active":"2024-03-19T01:10:34.950168+0000","last_peered":"2024-03-19T01:10:34.950168+0000","last_clean":"2024-03-19T01:10:34.950168+0000","last_became_active":"2024-03-19T01:10:32.153939+0000","last_became_peered":"2024-03-19T01:10:32.153939+0000","last_unstale":"2024-03-19T01:10:34.950168+0000","last_undegraded":"2024-03-19T01:10:34.950168+0000","last_fullsized":"2024-03-19T01:10:34.950168+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:45.377161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151482+0000","last_change":"2024-03-19T01:10:33.168395+0000","last_active":"2024-03-19T01:10:34.151482+0000","last_peered":"2024-03-19T01:10:34.151482+0000","last_clean":"2024-03-19T01:10:34.151482+0000","last_became_active":"2024-03-19T01:10:33.168078+0000","last_became_peered":"2024-03-19T01:10:33.168078+0000","last_unstale":"2024-03-19T01:10:34.151482+0000","last_undegraded":"2024-03-19T01:10:34.151482+0000","last_fullsized":"2024-03-19T01:10:34.151482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:54:52.597591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.981779+0000","last_change":"2024-03-19T01:10:32.152489+0000","last_active":"2024-03-19T01:10:36.981779+0000","last_peered":"2024-03-19T01:10:36.981779+0000","last_clean":"2024-03-19T01:10:36.981779+0000","last_became_active":"2024-03-19T01:10:32.152126+0000","last_became_peered":"2024-03-19T01:10:32.152126+0000","last_unstale":"2024-03-19T01:10:36.981779+0000","last_undegraded":"2024-03-19T01:10:36.981779+0000","last_fullsized":"2024-03-19T01:10:36.981779+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:29:26.729261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948530+0000","last_change":"2024-03-19T01:10:33.174635+0000","last_active":"2024-03-19T01:10:34.948530+0000","last_peered":"2024-03-19T01:10:34.948530+0000","last_clean":"2024-03-19T01:10:34.948530+0000","last_became_active":"2024-03-19T01:10:33.173970+0000","last_became_peered":"2024-03-19T01:10:33.173970+0000","last_unstale":"2024-03-19T01:10:34.948530+0000","last_undegraded":"2024-03-19T01:10:34.948530+0000","last_fullsized":"2024-03-19T01:10:34.948530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:16.174169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:37.987363+0000","last_change":"2024-03-19T01:10:32.148225+0000","last_active":"2024-03-19T01:10:37.987363+0000","last_peered":"2024-03-19T01:10:37.987363+0000","last_clean":"2024-03-19T01:10:37.987363+0000","last_became_active":"2024-03-19T01:10:32.147889+0000","last_became_peered":"2024-03-19T01:10:32.147889+0000","last_unstale":"2024-03-19T01:10:37.987363+0000","last_undegraded":"2024-03-19T01:10:37.987363+0000","last_fullsized":"2024-03-19T01:10:37.987363+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:56.037018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947675+0000","last_change":"2024-03-19T01:10:33.172925+0000","last_active":"2024-03-19T01:10:34.947675+0000","last_peered":"2024-03-19T01:10:34.947675+0000","last_clean":"2024-03-19T01:10:34.947675+0000","last_became_active":"2024-03-19T01:10:33.172572+0000","last_became_peered":"2024-03-19T01:10:33.172572+0000","last_unstale":"2024-03-19T01:10:34.947675+0000","last_undegraded":"2024-03-19T01:10:34.947675+0000","last_fullsized":"2024-03-19T01:10:34.947675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:02.853337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948702+0000","last_change":"2024-03-19T01:10:32.146245+0000","last_active":"2024-03-19T01:10:34.948702+0000","last_peered":"2024-03-19T01:10:34.948702+0000","last_clean":"2024-03-19T01:10:34.948702+0000","last_became_active":"2024-03-19T01:10:32.145877+0000","last_became_peered":"2024-03-19T01:10:32.145877+0000","last_unstale":"2024-03-19T01:10:34.948702+0000","last_undegraded":"2024-03-19T01:10:34.948702+0000","last_fullsized":"2024-03-19T01:10:34.948702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:45.323155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.678959+0000","last_change":"2024-03-19T01:10:33.178538+0000","last_active":"2024-03-19T01:10:34.678959+0000","last_peered":"2024-03-19T01:10:34.678959+0000","last_clean":"2024-03-19T01:10:34.678959+0000","last_became_active":"2024-03-19T01:10:33.178170+0000","last_became_peered":"2024-03-19T01:10:33.178170+0000","last_unstale":"2024-03-19T01:10:34.678959+0000","last_undegraded":"2024-03-19T01:10:34.678959+0000","last_fullsized":"2024-03-19T01:10:34.678959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:41.913715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:37.986468+0000","last_change":"2024-03-19T01:10:32.147697+0000","last_active":"2024-03-19T01:10:37.986468+0000","last_peered":"2024-03-19T01:10:37.986468+0000","last_clean":"2024-03-19T01:10:37.986468+0000","last_became_active":"2024-03-19T01:10:32.147399+0000","last_became_peered":"2024-03-19T01:10:32.147399+0000","last_unstale":"2024-03-19T01:10:37.986468+0000","last_undegraded":"2024-03-19T01:10:37.986468+0000","last_fullsized":"2024-03-19T01:10:37.986468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:33:38.961646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.176745+0000","last_change":"2024-03-19T01:10:33.175990+0000","last_active":"2024-03-19T01:10:33.176745+0000","last_peered":"2024-03-19T01:10:33.176745+0000","last_clean":"2024-03-19T01:10:33.176745+0000","last_became_active":"2024-03-19T01:10:33.175539+0000","last_became_peered":"2024-03-19T01:10:33.175539+0000","last_unstale":"2024-03-19T01:10:33.176745+0000","last_undegraded":"2024-03-19T01:10:33.176745+0000","last_fullsized":"2024-03-19T01:10:33.176745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:11:42.608955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.979950+0000","last_change":"2024-03-19T01:10:32.151834+0000","last_active":"2024-03-19T01:10:36.979950+0000","last_peered":"2024-03-19T01:10:36.979950+0000","last_clean":"2024-03-19T01:10:36.979950+0000","last_became_active":"2024-03-19T01:10:32.151514+0000","last_became_peered":"2024-03-19T01:10:32.151514+0000","last_unstale":"2024-03-19T01:10:36.979950+0000","last_undegraded":"2024-03-19T01:10:36.979950+0000","last_fullsized":"2024-03-19T01:10:36.979950+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:40:16.839334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947540+0000","last_change":"2024-03-19T01:10:33.185206+0000","last_active":"2024-03-19T01:10:34.947540+0000","last_peered":"2024-03-19T01:10:34.947540+0000","last_clean":"2024-03-19T01:10:34.947540+0000","last_became_active":"2024-03-19T01:10:33.184901+0000","last_became_peered":"2024-03-19T01:10:33.184901+0000","last_unstale":"2024-03-19T01:10:34.947540+0000","last_undegraded":"2024-03-19T01:10:34.947540+0000","last_fullsized":"2024-03-19T01:10:34.947540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:54:47.266255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.951381+0000","last_change":"2024-03-19T01:10:32.146654+0000","last_active":"2024-03-19T01:10:34.951381+0000","last_peered":"2024-03-19T01:10:34.951381+0000","last_clean":"2024-03-19T01:10:34.951381+0000","last_became_active":"2024-03-19T01:10:32.146178+0000","last_became_peered":"2024-03-19T01:10:32.146178+0000","last_unstale":"2024-03-19T01:10:34.951381+0000","last_undegraded":"2024-03-19T01:10:34.951381+0000","last_fullsized":"2024-03-19T01:10:34.951381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:52:27.257000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949740+0000","last_change":"2024-03-19T01:10:33.181232+0000","last_active":"2024-03-19T01:10:34.949740+0000","last_peered":"2024-03-19T01:10:34.949740+0000","last_clean":"2024-03-19T01:10:34.949740+0000","last_became_active":"2024-03-19T01:10:33.180719+0000","last_became_peered":"2024-03-19T01:10:33.180719+0000","last_unstale":"2024-03-19T01:10:34.949740+0000","last_undegraded":"2024-03-19T01:10:34.949740+0000","last_fullsized":"2024-03-19T01:10:34.949740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:43.201965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'5","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.979620+0000","last_change":"2024-03-19T01:10:32.144273+0000","last_active":"2024-03-19T01:10:36.979620+0000","last_peered":"2024-03-19T01:10:36.979620+0000","last_clean":"2024-03-19T01:10:36.979620+0000","last_became_active":"2024-03-19T01:10:32.143514+0000","last_became_peered":"2024-03-19T01:10:32.143514+0000","last_unstale":"2024-03-19T01:10:36.979620+0000","last_undegraded":"2024-03-19T01:10:36.979620+0000","last_fullsized":"2024-03-19T01:10:36.979620+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:08:57.705653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949691+0000","last_change":"2024-03-19T01:10:33.178129+0000","last_active":"2024-03-19T01:10:34.949691+0000","last_peered":"2024-03-19T01:10:34.949691+0000","last_clean":"2024-03-19T01:10:34.949691+0000","last_became_active":"2024-03-19T01:10:33.177802+0000","last_became_peered":"2024-03-19T01:10:33.177802+0000","last_unstale":"2024-03-19T01:10:34.949691+0000","last_undegraded":"2024-03-19T01:10:34.949691+0000","last_fullsized":"2024-03-19T01:10:34.949691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:46:42.698865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:36.979709+0000","last_change":"2024-03-19T01:10:32.149613+0000","last_active":"2024-03-19T01:10:36.979709+0000","last_peered":"2024-03-19T01:10:36.979709+0000","last_clean":"2024-03-19T01:10:36.979709+0000","last_became_active":"2024-03-19T01:10:32.147651+0000","last_became_peered":"2024-03-19T01:10:32.147651+0000","last_unstale":"2024-03-19T01:10:36.979709+0000","last_undegraded":"2024-03-19T01:10:36.979709+0000","last_fullsized":"2024-03-19T01:10:36.979709+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:06:59.222944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.949176+0000","last_change":"2024-03-19T01:10:33.173682+0000","last_active":"2024-03-19T01:10:34.949176+0000","last_peered":"2024-03-19T01:10:34.949176+0000","last_clean":"2024-03-19T01:10:34.949176+0000","last_became_active":"2024-03-19T01:10:33.173374+0000","last_became_peered":"2024-03-19T01:10:33.173374+0000","last_unstale":"2024-03-19T01:10:34.949176+0000","last_undegraded":"2024-03-19T01:10:34.949176+0000","last_fullsized":"2024-03-19T01:10:34.949176+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:59:47.396410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151491+0000","last_change":"2024-03-19T01:10:32.149253+0000","last_active":"2024-03-19T01:10:34.151491+0000","last_peered":"2024-03-19T01:10:34.151491+0000","last_clean":"2024-03-19T01:10:34.151491+0000","last_became_active":"2024-03-19T01:10:32.148520+0000","last_became_peered":"2024-03-19T01:10:32.148520+0000","last_unstale":"2024-03-19T01:10:34.151491+0000","last_undegraded":"2024-03-19T01:10:34.151491+0000","last_fullsized":"2024-03-19T01:10:34.151491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:10:21.211510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.950410+0000","last_change":"2024-03-19T01:10:33.182684+0000","last_active":"2024-03-19T01:10:34.950410+0000","last_peered":"2024-03-19T01:10:34.950410+0000","last_clean":"2024-03-19T01:10:34.950410+0000","last_became_active":"2024-03-19T01:10:33.182269+0000","last_became_peered":"2024-03-19T01:10:33.182269+0000","last_unstale":"2024-03-19T01:10:34.950410+0000","last_undegraded":"2024-03-19T01:10:34.950410+0000","last_fullsized":"2024-03-19T01:10:34.950410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:49:33.318462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.174629+0000","last_change":"2024-03-19T01:10:33.172795+0000","last_active":"2024-03-19T01:10:33.174629+0000","last_peered":"2024-03-19T01:10:33.174629+0000","last_clean":"2024-03-19T01:10:33.174629+0000","last_became_active":"2024-03-19T01:10:33.172483+0000","last_became_peered":"2024-03-19T01:10:33.172483+0000","last_unstale":"2024-03-19T01:10:33.174629+0000","last_undegraded":"2024-03-19T01:10:33.174629+0000","last_fullsized":"2024-03-19T01:10:33.174629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:41.767064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152455+0000","last_change":"2024-03-19T01:10:33.169130+0000","last_active":"2024-03-19T01:10:34.152455+0000","last_peered":"2024-03-19T01:10:34.152455+0000","last_clean":"2024-03-19T01:10:34.152455+0000","last_became_active":"2024-03-19T01:10:33.168763+0000","last_became_peered":"2024-03-19T01:10:33.168763+0000","last_unstale":"2024-03-19T01:10:34.152455+0000","last_undegraded":"2024-03-19T01:10:34.152455+0000","last_fullsized":"2024-03-19T01:10:34.152455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:37:23.644401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152366+0000","last_change":"2024-03-19T01:10:33.178847+0000","last_active":"2024-03-19T01:10:34.152366+0000","last_peered":"2024-03-19T01:10:34.152366+0000","last_clean":"2024-03-19T01:10:34.152366+0000","last_became_active":"2024-03-19T01:10:33.178533+0000","last_became_peered":"2024-03-19T01:10:33.178533+0000","last_unstale":"2024-03-19T01:10:34.152366+0000","last_undegraded":"2024-03-19T01:10:34.152366+0000","last_fullsized":"2024-03-19T01:10:34.152366+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:52:15.401797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948621+0000","last_change":"2024-03-19T01:10:33.169936+0000","last_active":"2024-03-19T01:10:34.948621+0000","last_peered":"2024-03-19T01:10:34.948621+0000","last_clean":"2024-03-19T01:10:34.948621+0000","last_became_active":"2024-03-19T01:10:33.169594+0000","last_became_peered":"2024-03-19T01:10:33.169594+0000","last_unstale":"2024-03-19T01:10:34.948621+0000","last_undegraded":"2024-03-19T01:10:34.948621+0000","last_fullsized":"2024-03-19T01:10:34.948621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:41.932575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151417+0000","last_change":"2024-03-19T01:10:33.163701+0000","last_active":"2024-03-19T01:10:34.151417+0000","last_peered":"2024-03-19T01:10:34.151417+0000","last_clean":"2024-03-19T01:10:34.151417+0000","last_became_active":"2024-03-19T01:10:33.163374+0000","last_became_peered":"2024-03-19T01:10:33.163374+0000","last_unstale":"2024-03-19T01:10:34.151417+0000","last_undegraded":"2024-03-19T01:10:34.151417+0000","last_fullsized":"2024-03-19T01:10:34.151417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:34:26.277015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.179821+0000","last_change":"2024-03-19T01:10:33.178699+0000","last_active":"2024-03-19T01:10:33.179821+0000","last_peered":"2024-03-19T01:10:33.179821+0000","last_clean":"2024-03-19T01:10:33.179821+0000","last_became_active":"2024-03-19T01:10:33.178380+0000","last_became_peered":"2024-03-19T01:10:33.178380+0000","last_unstale":"2024-03-19T01:10:33.179821+0000","last_undegraded":"2024-03-19T01:10:33.179821+0000","last_fullsized":"2024-03-19T01:10:33.179821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:52:23.012924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152858+0000","last_change":"2024-03-19T01:10:33.179908+0000","last_active":"2024-03-19T01:10:34.152858+0000","last_peered":"2024-03-19T01:10:34.152858+0000","last_clean":"2024-03-19T01:10:34.152858+0000","last_became_active":"2024-03-19T01:10:33.179578+0000","last_became_peered":"2024-03-19T01:10:33.179578+0000","last_unstale":"2024-03-19T01:10:34.152858+0000","last_undegraded":"2024-03-19T01:10:34.152858+0000","last_fullsized":"2024-03-19T01:10:34.152858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:30:53.918078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.948348+0000","last_change":"2024-03-19T01:10:33.179788+0000","last_active":"2024-03-19T01:10:34.948348+0000","last_peered":"2024-03-19T01:10:34.948348+0000","last_clean":"2024-03-19T01:10:34.948348+0000","last_became_active":"2024-03-19T01:10:33.179473+0000","last_became_peered":"2024-03-19T01:10:33.179473+0000","last_unstale":"2024-03-19T01:10:34.948348+0000","last_undegraded":"2024-03-19T01:10:34.948348+0000","last_fullsized":"2024-03-19T01:10:34.948348+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:36.284768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.708993+0000","last_change":"2024-03-19T01:10:33.172698+0000","last_active":"2024-03-19T01:10:34.708993+0000","last_peered":"2024-03-19T01:10:34.708993+0000","last_clean":"2024-03-19T01:10:34.708993+0000","last_became_active":"2024-03-19T01:10:33.172369+0000","last_became_peered":"2024-03-19T01:10:33.172369+0000","last_unstale":"2024-03-19T01:10:34.708993+0000","last_undegraded":"2024-03-19T01:10:34.708993+0000","last_fullsized":"2024-03-19T01:10:34.708993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:09:18.961724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.947996+0000","last_change":"2024-03-19T01:10:33.184350+0000","last_active":"2024-03-19T01:10:34.947996+0000","last_peered":"2024-03-19T01:10:34.947996+0000","last_clean":"2024-03-19T01:10:34.947996+0000","last_became_active":"2024-03-19T01:10:33.180533+0000","last_became_peered":"2024-03-19T01:10:33.180533+0000","last_unstale":"2024-03-19T01:10:34.947996+0000","last_undegraded":"2024-03-19T01:10:34.947996+0000","last_fullsized":"2024-03-19T01:10:34.947996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:28.003560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.152649+0000","last_change":"2024-03-19T01:10:33.180396+0000","last_active":"2024-03-19T01:10:34.152649+0000","last_peered":"2024-03-19T01:10:34.152649+0000","last_clean":"2024-03-19T01:10:34.152649+0000","last_became_active":"2024-03-19T01:10:33.180062+0000","last_became_peered":"2024-03-19T01:10:33.180062+0000","last_unstale":"2024-03-19T01:10:34.152649+0000","last_undegraded":"2024-03-19T01:10:34.152649+0000","last_fullsized":"2024-03-19T01:10:34.152649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:15.429623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-19T01:10:34.151523+0000","last_change":"2024-03-19T01:10:33.167261+0000","last_active":"2024-03-19T01:10:34.151523+0000","last_peered":"2024-03-19T01:10:34.151523+0000","last_clean":"2024-03-19T01:10:34.151523+0000","last_became_active":"2024-03-19T01:10:33.166943+0000","last_became_peered":"2024-03-19T01:10:33.166943+0000","last_unstale":"2024-03-19T01:10:34.151523+0000","last_undegraded":"2024-03-19T01:10:34.151523+0000","last_fullsized":"2024-03-19T01:10:34.151523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:01:44.451161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-19T01:10:33.176417+0000","last_change":"2024-03-19T01:10:33.175750+0000","last_active":"2024-03-19T01:10:33.176417+0000","last_peered":"2024-03-19T01:10:33.176417+0000","last_clean":"2024-03-19T01:10:33.176417+0000","last_became_active":"2024-03-19T01:10:33.175411+0000","last_became_peered":"2024-03-19T01:10:33.175411+0000","last_unstale":"2024-03-19T01:10:33.176417+0000","last_undegraded":"2024-03-19T01:10:33.176417+0000","last_fullsized":"2024-03-19T01:10:33.176417+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:33:43.086238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284643,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":13451,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":158,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":64410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":901120,"data_stored":429951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645331,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1015808,"data_stored":1032203,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808856,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":439339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070493,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":909312,"data_stored":430555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332130,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":954368,"data_stored":1020901,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":901120,"data_stored":429951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":441106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":925696,"data_stored":439215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":442496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":438929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920818,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":929792,"data_stored":439578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":1031709,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34556,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34556,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34556,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":22647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":22867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":10339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:10:40.949 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-19T01:10:40.950 INFO:tasks.cephadm:Setting up client nodes... 2024-03-19T01:10:40.950 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-19T01:10:41.228 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:10:41.354 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:40 smithi116 ceph-mon[62701]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:41.354 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:40 smithi116 ceph-mon[62701]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:41.354 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:40 smithi116 ceph-mon[62701]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-19T01:10:41.354 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:40 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3684548049' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:41.375 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:40 smithi156 ceph-mon[66799]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:41.375 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:40 smithi156 ceph-mon[66799]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:41.375 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:40 smithi156 ceph-mon[66799]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-19T01:10:41.375 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:40 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3684548049' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:41.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:40 smithi162 ceph-mon[67178]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:41.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:40 smithi162 ceph-mon[67178]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:10:41.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:40 smithi162 ceph-mon[67178]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-19T01:10:41.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:40 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3684548049' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:42.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:41 smithi162 ceph-mon[67178]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:10:42.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:41 smithi162 ceph-mon[67178]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:10:42.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:41 smithi162 ceph-mon[67178]: pgmap v292: 97 pgs: 97 active+clean; 591 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 8.9 KiB/s wr, 27 op/s 2024-03-19T01:10:42.357 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:41 smithi116 ceph-mon[62701]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:10:42.357 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:41 smithi116 ceph-mon[62701]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:10:42.357 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:41 smithi116 ceph-mon[62701]: pgmap v292: 97 pgs: 97 active+clean; 591 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 8.9 KiB/s wr, 27 op/s 2024-03-19T01:10:42.387 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:41 smithi156 ceph-mon[66799]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:10:42.387 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:41 smithi156 ceph-mon[66799]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:10:42.387 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:41 smithi156 ceph-mon[66799]: pgmap v292: 97 pgs: 97 active+clean; 591 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 8.9 KiB/s wr, 27 op/s 2024-03-19T01:10:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.921+0000 7f3045bfc700 1 -- 172.21.15.116:0/2372959186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3040105720 msgr2=0x7f3040107b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.921+0000 7f3045bfc700 1 --2- 172.21.15.116:0/2372959186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3040105720 0x7f3040107b10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3034009990 tx=0x7f3034009ca0 comp rx=0 tx=0).stop 2024-03-19T01:10:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.921+0000 7f3045bfc700 1 -- 172.21.15.116:0/2372959186 shutdown_connections 2024-03-19T01:10:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.921+0000 7f3045bfc700 1 --2- 172.21.15.116:0/2372959186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3040108050 0x7f304010a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.921+0000 7f3045bfc700 1 --2- 172.21.15.116:0/2372959186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040102df0 0x7f30401051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.921+0000 7f3045bfc700 1 --2- 172.21.15.116:0/2372959186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3040105720 0x7f3040107b10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.921+0000 7f3045bfc700 1 -- 172.21.15.116:0/2372959186 >> 172.21.15.116:0/2372959186 conn(0x7f30400fc800 msgr2=0x7f30400fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.921+0000 7f3045bfc700 1 -- 172.21.15.116:0/2372959186 shutdown_connections 2024-03-19T01:10:42.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.921+0000 7f3045bfc700 1 -- 172.21.15.116:0/2372959186 wait complete. 2024-03-19T01:10:42.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f3045bfc700 1 Processor -- start 2024-03-19T01:10:42.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f3045bfc700 1 -- start start 2024-03-19T01:10:42.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f3045bfc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3040102df0 0x7f3040198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:42.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f3045bfc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040105720 0x7f30401989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:42.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f3045bfc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3040108050 0x7f304019fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:42.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f3045bfc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f304010df30 con 0x7f3040105720 2024-03-19T01:10:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f3045bfc700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f304010ddb0 con 0x7f3040102df0 2024-03-19T01:10:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f3045bfc700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f304010e0b0 con 0x7f3040108050 2024-03-19T01:10:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f303effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040105720 0x7f30401989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f303f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3040102df0 0x7f3040198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f303f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3040102df0 0x7f3040198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:48388/0 (socket says 172.21.15.116:48388) 2024-03-19T01:10:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f303effd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040105720 0x7f30401989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53636/0 (socket says 172.21.15.116:53636) 2024-03-19T01:10:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f303ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3040108050 0x7f304019fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.922+0000 7f303f7fe700 1 -- 172.21.15.116:0/4255314519 learned_addr learned my addr 172.21.15.116:0/4255314519 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.923+0000 7f303f7fe700 1 -- 172.21.15.116:0/4255314519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3040108050 msgr2=0x7f304019fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.923+0000 7f303f7fe700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3040108050 0x7f304019fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.923+0000 7f303f7fe700 1 -- 172.21.15.116:0/4255314519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040105720 msgr2=0x7f30401989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.923+0000 7f303f7fe700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040105720 0x7f30401989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.923+0000 7f303f7fe700 1 -- 172.21.15.116:0/4255314519 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30401a00f0 con 0x7f3040102df0 2024-03-19T01:10:42.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.923+0000 7f303effd700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040105720 0x7f30401989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:10:42.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.923+0000 7f303ffff700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3040108050 0x7f304019fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:10:42.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.923+0000 7f303f7fe700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3040102df0 0x7f3040198490 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f303000e9a0 tx=0x7f303000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:42.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.923+0000 7f303cff9700 1 -- 172.21.15.116:0/4255314519 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f303000ca10 con 0x7f3040102df0 2024-03-19T01:10:42.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.923+0000 7f3045bfc700 1 -- 172.21.15.116:0/4255314519 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30401a03e0 con 0x7f3040102df0 2024-03-19T01:10:42.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.923+0000 7f303cff9700 1 -- 172.21.15.116:0/4255314519 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f303000cbb0 con 0x7f3040102df0 2024-03-19T01:10:42.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.923+0000 7f303cff9700 1 -- 172.21.15.116:0/4255314519 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3030018710 con 0x7f3040102df0 2024-03-19T01:10:42.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.923+0000 7f3045bfc700 1 -- 172.21.15.116:0/4255314519 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30401a0970 con 0x7f3040102df0 2024-03-19T01:10:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.924+0000 7f3045bfc700 1 -- 172.21.15.116:0/4255314519 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3040050820 con 0x7f3040102df0 2024-03-19T01:10:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.925+0000 7f303cff9700 1 -- 172.21.15.116:0/4255314519 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f30300188b0 con 0x7f3040102df0 2024-03-19T01:10:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.926+0000 7f303cff9700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f30280a29d0 0x7f30280a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.926+0000 7f303effd700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f30280a29d0 0x7f30280a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.926+0000 7f303cff9700 1 -- 172.21.15.116:0/4255314519 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f30300cef00 con 0x7f3040102df0 2024-03-19T01:10:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.926+0000 7f303effd700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f30280a29d0 0x7f30280a4e90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f303400b2e0 tx=0x7f3034005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:42.934+0000 7f303cff9700 1 -- 172.21.15.116:0/4255314519 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f303009bef0 con 0x7f3040102df0 2024-03-19T01:10:43.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.220+0000 7f3045bfc700 1 -- 172.21.15.116:0/4255314519 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f3040062f20 con 0x7f3040102df0 2024-03-19T01:10:43.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.226+0000 7f303cff9700 1 -- 172.21.15.116:0/4255314519 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f303009b7f0 con 0x7f3040102df0 2024-03-19T01:10:43.226 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-03-19T01:10:43.227 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAT5vhlsylCDRAADWJU726zbexyd84feMSazw== 2024-03-19T01:10:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.228+0000 7f3045bfc700 1 -- 172.21.15.116:0/4255314519 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f30280a29d0 msgr2=0x7f30280a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.228+0000 7f3045bfc700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f30280a29d0 0x7f30280a4e90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f303400b2e0 tx=0x7f3034005fd0 comp rx=0 tx=0).stop 2024-03-19T01:10:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.228+0000 7f3045bfc700 1 -- 172.21.15.116:0/4255314519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3040102df0 msgr2=0x7f3040198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.228+0000 7f3045bfc700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3040102df0 0x7f3040198490 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f303000e9a0 tx=0x7f303000ecb0 comp rx=0 tx=0).stop 2024-03-19T01:10:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.228+0000 7f3045bfc700 1 -- 172.21.15.116:0/4255314519 shutdown_connections 2024-03-19T01:10:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.228+0000 7f3045bfc700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f30280a29d0 0x7f30280a4e90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.228+0000 7f3045bfc700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3040108050 0x7f304019fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.228+0000 7f3045bfc700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3040102df0 0x7f3040198490 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.228+0000 7f3045bfc700 1 --2- 172.21.15.116:0/4255314519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3040105720 0x7f30401989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.228+0000 7f3045bfc700 1 -- 172.21.15.116:0/4255314519 >> 172.21.15.116:0/4255314519 conn(0x7f30400fc800 msgr2=0x7f3040103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:43.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.228+0000 7f3045bfc700 1 -- 172.21.15.116:0/4255314519 shutdown_connections 2024-03-19T01:10:43.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:43.228+0000 7f3045bfc700 1 -- 172.21.15.116:0/4255314519 wait complete. 2024-03-19T01:10:43.807 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:10:43.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-19T01:10:43.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-19T01:10:43.848 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-19T01:10:43.849 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-19T01:10:43.849 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-19T01:10:44.141 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:10:44.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:44 smithi116 ceph-mon[62701]: pgmap v293: 97 pgs: 97 active+clean; 591 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 7.6 KiB/s wr, 23 op/s 2024-03-19T01:10:44.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:44 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4255314519' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T01:10:44.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:44 smithi116 ceph-mon[62701]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T01:10:44.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:44 smithi116 ceph-mon[62701]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T01:10:44.398 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:44 smithi162 ceph-mon[67178]: pgmap v293: 97 pgs: 97 active+clean; 591 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 7.6 KiB/s wr, 23 op/s 2024-03-19T01:10:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:44 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/4255314519' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T01:10:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:44 smithi162 ceph-mon[67178]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T01:10:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:44 smithi162 ceph-mon[67178]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T01:10:44.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:44 smithi156 ceph-mon[66799]: pgmap v293: 97 pgs: 97 active+clean; 591 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 7.6 KiB/s wr, 23 op/s 2024-03-19T01:10:44.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:44 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/4255314519' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T01:10:44.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:44 smithi156 ceph-mon[66799]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-19T01:10:44.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:44 smithi156 ceph-mon[66799]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-19T01:10:45.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.960+0000 7fde850f8700 1 -- 172.21.15.116:0/1296506000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde80108360 msgr2=0x7fde80108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.960+0000 7fde850f8700 1 --2- 172.21.15.116:0/1296506000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde80108360 0x7fde80108740 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fde700099c0 tx=0x7fde70009cd0 comp rx=0 tx=0).stop 2024-03-19T01:10:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.961+0000 7fde850f8700 1 -- 172.21.15.116:0/1296506000 shutdown_connections 2024-03-19T01:10:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.961+0000 7fde850f8700 1 --2- 172.21.15.116:0/1296506000 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde8006a960 0x7fde8010f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.961+0000 7fde850f8700 1 --2- 172.21.15.116:0/1296506000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde80108360 0x7fde80108740 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.961+0000 7fde850f8700 1 --2- 172.21.15.116:0/1296506000 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde80069fc0 0x7fde8006a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.961+0000 7fde850f8700 1 -- 172.21.15.116:0/1296506000 >> 172.21.15.116:0/1296506000 conn(0x7fde80078140 msgr2=0x7fde80078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.961+0000 7fde850f8700 1 -- 172.21.15.116:0/1296506000 shutdown_connections 2024-03-19T01:10:45.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.961+0000 7fde850f8700 1 -- 172.21.15.116:0/1296506000 wait complete. 2024-03-19T01:10:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.961+0000 7fde850f8700 1 Processor -- start 2024-03-19T01:10:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde850f8700 1 -- start start 2024-03-19T01:10:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde850f8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde80069fc0 0x7fde801012e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde850f8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8006a960 0x7fde80101820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde850f8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde80108360 0x7fde80101d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde850f8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fde801125a0 con 0x7fde8006a960 2024-03-19T01:10:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde850f8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fde80112420 con 0x7fde80069fc0 2024-03-19T01:10:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde850f8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fde80112720 con 0x7fde80108360 2024-03-19T01:10:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde7e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8006a960 0x7fde80101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:45.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde7ed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde80069fc0 0x7fde801012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde7ed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde80069fc0 0x7fde801012e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:48404/0 (socket says 172.21.15.116:48404) 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde7e59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8006a960 0x7fde80101820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53640/0 (socket says 172.21.15.116:53640) 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde7ed9d700 1 -- 172.21.15.116:0/243517635 learned_addr learned my addr 172.21.15.116:0/243517635 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde7f59e700 1 --2- 172.21.15.116:0/243517635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde80108360 0x7fde80101d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde7ed9d700 1 -- 172.21.15.116:0/243517635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde80108360 msgr2=0x7fde80101d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde7ed9d700 1 --2- 172.21.15.116:0/243517635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde80108360 0x7fde80101d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde7ed9d700 1 -- 172.21.15.116:0/243517635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8006a960 msgr2=0x7fde80101820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde7ed9d700 1 --2- 172.21.15.116:0/243517635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8006a960 0x7fde80101820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde7ed9d700 1 -- 172.21.15.116:0/243517635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde801a6f60 con 0x7fde80069fc0 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.962+0000 7fde7ed9d700 1 --2- 172.21.15.116:0/243517635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde80069fc0 0x7fde801012e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fde70006010 tx=0x7fde7000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.963+0000 7fde67fff700 1 -- 172.21.15.116:0/243517635 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde70003680 con 0x7fde80069fc0 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.963+0000 7fde850f8700 1 -- 172.21.15.116:0/243517635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde801a71f0 con 0x7fde80069fc0 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.963+0000 7fde67fff700 1 -- 172.21.15.116:0/243517635 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde700050d0 con 0x7fde80069fc0 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.963+0000 7fde67fff700 1 -- 172.21.15.116:0/243517635 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde70003db0 con 0x7fde80069fc0 2024-03-19T01:10:45.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.963+0000 7fde850f8700 1 -- 172.21.15.116:0/243517635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde801a7640 con 0x7fde80069fc0 2024-03-19T01:10:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.964+0000 7fde850f8700 1 -- 172.21.15.116:0/243517635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde80068fd0 con 0x7fde80069fc0 2024-03-19T01:10:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.965+0000 7fde67fff700 1 -- 172.21.15.116:0/243517635 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fde70005280 con 0x7fde80069fc0 2024-03-19T01:10:45.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.965+0000 7fde67fff700 1 --2- 172.21.15.116:0/243517635 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fde680a2b40 0x7fde680a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.965+0000 7fde7e59c700 1 --2- 172.21.15.116:0/243517635 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fde680a2b40 0x7fde680a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.965+0000 7fde67fff700 1 -- 172.21.15.116:0/243517635 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fde700cf9c0 con 0x7fde80069fc0 2024-03-19T01:10:45.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.966+0000 7fde7e59c700 1 --2- 172.21.15.116:0/243517635 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fde680a2b40 0x7fde680a5000 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fde800786b0 tx=0x7fde74009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:45.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:45.973+0000 7fde67fff700 1 -- 172.21.15.116:0/243517635 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fde7005d740 con 0x7fde80069fc0 2024-03-19T01:10:46.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.257+0000 7fde850f8700 1 -- 172.21.15.116:0/243517635 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fde8002dd70 con 0x7fde80069fc0 2024-03-19T01:10:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.262+0000 7fde67fff700 1 -- 172.21.15.116:0/243517635 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262669 (secure 0 0 0) 0x7fde70013070 con 0x7fde80069fc0 2024-03-19T01:10:46.265 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:46.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.264+0000 7fde850f8700 1 -- 172.21.15.116:0/243517635 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fde680a2b40 msgr2=0x7fde680a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:46.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.264+0000 7fde850f8700 1 --2- 172.21.15.116:0/243517635 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fde680a2b40 0x7fde680a5000 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fde800786b0 tx=0x7fde74009260 comp rx=0 tx=0).stop 2024-03-19T01:10:46.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.264+0000 7fde850f8700 1 -- 172.21.15.116:0/243517635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde80069fc0 msgr2=0x7fde801012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.264+0000 7fde850f8700 1 --2- 172.21.15.116:0/243517635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde80069fc0 0x7fde801012e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fde70006010 tx=0x7fde7000f690 comp rx=0 tx=0).stop 2024-03-19T01:10:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.264+0000 7fde850f8700 1 -- 172.21.15.116:0/243517635 shutdown_connections 2024-03-19T01:10:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.264+0000 7fde850f8700 1 --2- 172.21.15.116:0/243517635 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fde680a2b40 0x7fde680a5000 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.264+0000 7fde850f8700 1 --2- 172.21.15.116:0/243517635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fde80108360 0x7fde80101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.264+0000 7fde850f8700 1 --2- 172.21.15.116:0/243517635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde80069fc0 0x7fde801012e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.264+0000 7fde850f8700 1 --2- 172.21.15.116:0/243517635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde8006a960 0x7fde80101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.264+0000 7fde850f8700 1 -- 172.21.15.116:0/243517635 >> 172.21.15.116:0/243517635 conn(0x7fde80078140 msgr2=0x7fde800ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.265+0000 7fde850f8700 1 -- 172.21.15.116:0/243517635 shutdown_connections 2024-03-19T01:10:46.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:46.265+0000 7fde850f8700 1 -- 172.21.15.116:0/243517635 wait complete. 2024-03-19T01:10:46.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:46 smithi162 ceph-mon[67178]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 9.4 KiB/s wr, 27 op/s 2024-03-19T01:10:46.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:46 smithi116 ceph-mon[62701]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 9.4 KiB/s wr, 27 op/s 2024-03-19T01:10:46.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:46 smithi156 ceph-mon[66799]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 9.4 KiB/s wr, 27 op/s 2024-03-19T01:10:46.958 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":2389438183},{"type":"v1","addr":"172.21.15.116:6801","nonce":2389438183}]},"active_addr":"172.21.15.116:6801/2389438183","active_change":"2024-03-19T01:03:26.974997+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.116:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":1958516329}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":3839697157}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":1321758041}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":4070733263}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.116:0","nonce":363031999}]}]} 2024-03-19T01:10:46.964 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-19T01:10:46.964 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-19T01:10:46.964 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T01:10:47.226 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:10:47.348 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:47 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/243517635' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T01:10:47.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:47 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/243517635' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T01:10:47.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:47 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/243517635' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-19T01:10:48.340 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:48 smithi116 ceph-mon[62701]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 8.6 KiB/s wr, 24 op/s 2024-03-19T01:10:48.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:48 smithi162 ceph-mon[67178]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 8.6 KiB/s wr, 24 op/s 2024-03-19T01:10:48.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:48 smithi156 ceph-mon[66799]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 8.6 KiB/s wr, 24 op/s 2024-03-19T01:10:48.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.861+0000 7f3801b63700 1 -- 172.21.15.116:0/1598113550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37fc108470 msgr2=0x7f37fc108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:48.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.861+0000 7f3801b63700 1 --2- 172.21.15.116:0/1598113550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37fc108470 0x7f37fc108850 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f37ec009990 tx=0x7f37ec009ca0 comp rx=0 tx=0).stop 2024-03-19T01:10:48.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.861+0000 7f3801b63700 1 -- 172.21.15.116:0/1598113550 shutdown_connections 2024-03-19T01:10:48.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.861+0000 7f3801b63700 1 --2- 172.21.15.116:0/1598113550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37fc102d50 0x7f37fc10f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:48.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.861+0000 7f3801b63700 1 --2- 172.21.15.116:0/1598113550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37fc108470 0x7f37fc108850 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:48.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.861+0000 7f3801b63700 1 --2- 172.21.15.116:0/1598113550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37fc102390 0x7f37fc102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:48.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.861+0000 7f3801b63700 1 -- 172.21.15.116:0/1598113550 >> 172.21.15.116:0/1598113550 conn(0x7f37fc0793d0 msgr2=0x7f37fc0797e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:48.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.861+0000 7f3801b63700 1 -- 172.21.15.116:0/1598113550 shutdown_connections 2024-03-19T01:10:48.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.861+0000 7f3801b63700 1 -- 172.21.15.116:0/1598113550 wait complete. 2024-03-19T01:10:48.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.861+0000 7f3801b63700 1 Processor -- start 2024-03-19T01:10:48.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f3801b63700 1 -- start start 2024-03-19T01:10:48.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f3801b63700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37fc102390 0x7f37fc19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:48.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f3801b63700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37fc102d50 0x7f37fc19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:48.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f3801b63700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37fc108470 0x7f37fc198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:48.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f3801b63700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc1125f0 con 0x7f37fc102390 2024-03-19T01:10:48.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f3801b63700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc112470 con 0x7f37fc108470 2024-03-19T01:10:48.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f3801b63700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f37fc112770 con 0x7f37fc102d50 2024-03-19T01:10:48.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f37fb7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37fc102390 0x7f37fc19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:48.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f37fb7fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37fc102390 0x7f37fc19e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41040/0 (socket says 172.21.15.116:41040) 2024-03-19T01:10:48.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f37fb7fe700 1 -- 172.21.15.116:0/2831170320 learned_addr learned my addr 172.21.15.116:0/2831170320 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:48.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f37fbfff700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37fc108470 0x7f37fc198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:48.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f37faffd700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37fc102d50 0x7f37fc19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:48.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f37fbfff700 1 -- 172.21.15.116:0/2831170320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37fc102d50 msgr2=0x7f37fc19ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:48.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f37fbfff700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37fc102d50 0x7f37fc19ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:48.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f37fbfff700 1 -- 172.21.15.116:0/2831170320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37fc102390 msgr2=0x7f37fc19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:48.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f37fbfff700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37fc102390 0x7f37fc19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:48.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f37fbfff700 1 -- 172.21.15.116:0/2831170320 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37fc199030 con 0x7f37fc108470 2024-03-19T01:10:48.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.862+0000 7f37fb7fe700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37fc102390 0x7f37fc19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:10:48.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.863+0000 7f37fbfff700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37fc108470 0x7f37fc198850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37f400efc0 tx=0x7f37f400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:48.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.863+0000 7f37f8ff9700 1 -- 172.21.15.116:0/2831170320 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f40092e0 con 0x7f37fc108470 2024-03-19T01:10:48.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.863+0000 7f37f8ff9700 1 -- 172.21.15.116:0/2831170320 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37f4004bb0 con 0x7f37fc108470 2024-03-19T01:10:48.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.863+0000 7f3801b63700 1 -- 172.21.15.116:0/2831170320 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37fc1992c0 con 0x7f37fc108470 2024-03-19T01:10:48.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.863+0000 7f37f8ff9700 1 -- 172.21.15.116:0/2831170320 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4004280 con 0x7f37fc108470 2024-03-19T01:10:48.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.863+0000 7f3801b63700 1 -- 172.21.15.116:0/2831170320 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37fc074540 con 0x7f37fc108470 2024-03-19T01:10:48.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.864+0000 7f3801b63700 1 -- 172.21.15.116:0/2831170320 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37fc103a50 con 0x7f37fc108470 2024-03-19T01:10:48.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.865+0000 7f37f8ff9700 1 -- 172.21.15.116:0/2831170320 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f37f401b030 con 0x7f37fc108470 2024-03-19T01:10:48.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.865+0000 7f37f8ff9700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37e40a2920 0x7f37e40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:48.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.866+0000 7f37fb7fe700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37e40a2920 0x7f37e40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:48.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.866+0000 7f37f8ff9700 1 -- 172.21.15.116:0/2831170320 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f37f40ce750 con 0x7f37fc108470 2024-03-19T01:10:48.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.866+0000 7f37fb7fe700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37e40a2920 0x7f37e40a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00b2e0 tx=0x7f37ec009590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:48.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:48.869+0000 7f37f8ff9700 1 -- 172.21.15.116:0/2831170320 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f37f4022020 con 0x7f37fc108470 2024-03-19T01:10:49.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.074+0000 7f3801b63700 1 -- 172.21.15.116:0/2831170320 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37fc02dbb0 con 0x7f37fc108470 2024-03-19T01:10:49.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.077+0000 7f37f8ff9700 1 -- 172.21.15.116:0/2831170320 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22279 (secure 0 0 0) 0x7f37f4010040 con 0x7f37fc108470 2024-03-19T01:10:49.078 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:49.078 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":85,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:10:34.147770+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:10:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.079+0000 7f3801b63700 1 -- 172.21.15.116:0/2831170320 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37e40a2920 msgr2=0x7f37e40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.079+0000 7f3801b63700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37e40a2920 0x7f37e40a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00b2e0 tx=0x7f37ec009590 comp rx=0 tx=0).stop 2024-03-19T01:10:49.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.079+0000 7f3801b63700 1 -- 172.21.15.116:0/2831170320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37fc108470 msgr2=0x7f37fc198850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.079+0000 7f3801b63700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37fc108470 0x7f37fc198850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37f400efc0 tx=0x7f37f400c3a0 comp rx=0 tx=0).stop 2024-03-19T01:10:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.079+0000 7f3801b63700 1 -- 172.21.15.116:0/2831170320 shutdown_connections 2024-03-19T01:10:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.079+0000 7f3801b63700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37fc108470 0x7f37fc198850 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.079+0000 7f3801b63700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37fc102390 0x7f37fc19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.079+0000 7f3801b63700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37e40a2920 0x7f37e40a4de0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.079+0000 7f3801b63700 1 --2- 172.21.15.116:0/2831170320 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37fc102d50 0x7f37fc19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:49.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.079+0000 7f3801b63700 1 -- 172.21.15.116:0/2831170320 >> 172.21.15.116:0/2831170320 conn(0x7f37fc0793d0 msgr2=0x7f37fc101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.079+0000 7f3801b63700 1 -- 172.21.15.116:0/2831170320 shutdown_connections 2024-03-19T01:10:49.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:49.079+0000 7f3801b63700 1 -- 172.21.15.116:0/2831170320 wait complete. 2024-03-19T01:10:49.373 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:49 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2831170320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:49.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:49 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2831170320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:49.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:49 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2831170320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:49.717 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-19T01:10:49.717 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-19T01:10:49.962 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:10:50.335 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:50 smithi116 ceph-mon[62701]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 KiB/s wr, 26 op/s 2024-03-19T01:10:50.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:50 smithi162 ceph-mon[67178]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 KiB/s wr, 26 op/s 2024-03-19T01:10:50.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:50 smithi156 ceph-mon[66799]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 KiB/s wr, 26 op/s 2024-03-19T01:10:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.737+0000 7f616943f700 1 -- 172.21.15.116:0/1480862918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61641064f0 msgr2=0x7f616410e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.737+0000 7f616943f700 1 --2- 172.21.15.116:0/1480862918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61641064f0 0x7f616410e0c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f615c00b0d0 tx=0x7f615c00b3e0 comp rx=0 tx=0).stop 2024-03-19T01:10:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.737+0000 7f616943f700 1 -- 172.21.15.116:0/1480862918 shutdown_connections 2024-03-19T01:10:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.737+0000 7f616943f700 1 --2- 172.21.15.116:0/1480862918 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61641064f0 0x7f616410e0c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.737+0000 7f616943f700 1 --2- 172.21.15.116:0/1480862918 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f61641014a0 0x7f6164101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.737+0000 7f616943f700 1 --2- 172.21.15.116:0/1480862918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6164101e50 0x7f6164105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:51.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.737+0000 7f616943f700 1 -- 172.21.15.116:0/1480862918 >> 172.21.15.116:0/1480862918 conn(0x7f6164077260 msgr2=0x7f6164077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:51.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.737+0000 7f616943f700 1 -- 172.21.15.116:0/1480862918 shutdown_connections 2024-03-19T01:10:51.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.737+0000 7f616943f700 1 -- 172.21.15.116:0/1480862918 wait complete. 2024-03-19T01:10:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.738+0000 7f616943f700 1 Processor -- start 2024-03-19T01:10:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.738+0000 7f616943f700 1 -- start start 2024-03-19T01:10:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.738+0000 7f616943f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61641014a0 0x7f616419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.738+0000 7f616943f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6164101e50 0x7f616419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.738+0000 7f616943f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61641064f0 0x7f61641988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.738+0000 7f616943f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f616406b280 con 0x7f61641014a0 2024-03-19T01:10:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.738+0000 7f616943f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f616406b100 con 0x7f61641064f0 2024-03-19T01:10:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.738+0000 7f616943f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f616406b400 con 0x7f6164101e50 2024-03-19T01:10:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.738+0000 7f6162ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61641014a0 0x7f616419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.738+0000 7f6162ffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61641014a0 0x7f616419e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41060/0 (socket says 172.21.15.116:41060) 2024-03-19T01:10:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.738+0000 7f61627fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6164101e50 0x7f616419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.738+0000 7f6162ffd700 1 -- 172.21.15.116:0/4231420091 learned_addr learned my addr 172.21.15.116:0/4231420091 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:10:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.739+0000 7f61637fe700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61641064f0 0x7f61641988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:51.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.739+0000 7f61627fc700 1 -- 172.21.15.116:0/4231420091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61641064f0 msgr2=0x7f61641988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:51.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.739+0000 7f61627fc700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61641064f0 0x7f61641988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:51.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.739+0000 7f61627fc700 1 -- 172.21.15.116:0/4231420091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61641014a0 msgr2=0x7f616419e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:51.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.739+0000 7f61627fc700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61641014a0 0x7f616419e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:51.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.739+0000 7f61627fc700 1 -- 172.21.15.116:0/4231420091 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6164199030 con 0x7f6164101e50 2024-03-19T01:10:51.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.739+0000 7f61637fe700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61641064f0 0x7f61641988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:10:51.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.739+0000 7f61627fc700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6164101e50 0x7f616419ec00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f615800e9c0 tx=0x7f615800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:51.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.739+0000 7f614bfff700 1 -- 172.21.15.116:0/4231420091 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615800cbd0 con 0x7f6164101e50 2024-03-19T01:10:51.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.739+0000 7f616943f700 1 -- 172.21.15.116:0/4231420091 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6164199320 con 0x7f6164101e50 2024-03-19T01:10:51.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.739+0000 7f614bfff700 1 -- 172.21.15.116:0/4231420091 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f615800cd70 con 0x7f6164101e50 2024-03-19T01:10:51.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.739+0000 7f614bfff700 1 -- 172.21.15.116:0/4231420091 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6158018680 con 0x7f6164101e50 2024-03-19T01:10:51.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.739+0000 7f616943f700 1 -- 172.21.15.116:0/4231420091 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61641a5590 con 0x7f6164101e50 2024-03-19T01:10:51.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.740+0000 7f616943f700 1 -- 172.21.15.116:0/4231420091 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6164068fd0 con 0x7f6164101e50 2024-03-19T01:10:51.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.741+0000 7f614bfff700 1 -- 172.21.15.116:0/4231420091 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f6158047a90 con 0x7f6164101e50 2024-03-19T01:10:51.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.742+0000 7f614bfff700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f614c0a2920 0x7f614c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:10:51.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.742+0000 7f614bfff700 1 -- 172.21.15.116:0/4231420091 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f61580cece0 con 0x7f6164101e50 2024-03-19T01:10:51.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.742+0000 7f6162ffd700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f614c0a2920 0x7f614c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:10:51.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.742+0000 7f6162ffd700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f614c0a2920 0x7f614c0a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f61640779c0 tx=0x7f6154009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:10:51.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.745+0000 7f614bfff700 1 -- 172.21.15.116:0/4231420091 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f615809bcd0 con 0x7f6164101e50 2024-03-19T01:10:51.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.959+0000 7f616943f700 1 -- 172.21.15.116:0/4231420091 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f616402dbb0 con 0x7f6164101e50 2024-03-19T01:10:51.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.962+0000 7f614bfff700 1 -- 172.21.15.116:0/4231420091 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22279 (secure 0 0 0) 0x7f615809b5d0 con 0x7f6164101e50 2024-03-19T01:10:51.963 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:10:51.963 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":85,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:10:34.147770+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:10:51.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.964+0000 7f616943f700 1 -- 172.21.15.116:0/4231420091 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f614c0a2920 msgr2=0x7f614c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:51.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.964+0000 7f616943f700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f614c0a2920 0x7f614c0a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f61640779c0 tx=0x7f6154009260 comp rx=0 tx=0).stop 2024-03-19T01:10:51.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.964+0000 7f616943f700 1 -- 172.21.15.116:0/4231420091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6164101e50 msgr2=0x7f616419ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:10:51.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.964+0000 7f616943f700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6164101e50 0x7f616419ec00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f615800e9c0 tx=0x7f615800ecd0 comp rx=0 tx=0).stop 2024-03-19T01:10:51.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.964+0000 7f616943f700 1 -- 172.21.15.116:0/4231420091 shutdown_connections 2024-03-19T01:10:51.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.964+0000 7f616943f700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61641064f0 0x7f61641988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:51.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.964+0000 7f616943f700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f61641014a0 0x7f616419e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:51.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.964+0000 7f616943f700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f614c0a2920 0x7f614c0a4de0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:51.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.964+0000 7f616943f700 1 --2- 172.21.15.116:0/4231420091 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6164101e50 0x7f616419ec00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:10:51.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.964+0000 7f616943f700 1 -- 172.21.15.116:0/4231420091 >> 172.21.15.116:0/4231420091 conn(0x7f6164077260 msgr2=0x7f61640ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:10:51.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.964+0000 7f616943f700 1 -- 172.21.15.116:0/4231420091 shutdown_connections 2024-03-19T01:10:51.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:10:51.964+0000 7f616943f700 1 -- 172.21.15.116:0/4231420091 wait complete. 2024-03-19T01:10:52.323 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:52 smithi116 ceph-mon[62701]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 KiB/s wr, 17 op/s 2024-03-19T01:10:52.324 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:52 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4231420091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:52.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:52 smithi162 ceph-mon[67178]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 KiB/s wr, 17 op/s 2024-03-19T01:10:52.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:52 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/4231420091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:52.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:52 smithi156 ceph-mon[66799]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 KiB/s wr, 17 op/s 2024-03-19T01:10:52.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:52 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/4231420091' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:10:52.802 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-19T01:10:52.803 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-19T01:10:52.804 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-19T01:10:52.804 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-19T01:10:52.805 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-19T01:10:52.806 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-19T01:10:52.806 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-19T01:10:52.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-19T01:10:52.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-19T01:10:52.808 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-19T01:10:52.809 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-19T01:10:52.809 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-19T01:10:53.560 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:10:53.859 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:10:53.874 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:10:53.882 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:10:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:54 smithi162 ceph-mon[67178]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-03-19T01:10:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:54 smithi116 ceph-mon[62701]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-03-19T01:10:54.414 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:10:54.419 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:10:54.422 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:10:54.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:54 smithi156 ceph-mon[66799]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-03-19T01:10:56.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:56 smithi162 ceph-mon[67178]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.4 KiB/s wr, 15 op/s 2024-03-19T01:10:56.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:56 smithi116 ceph-mon[62701]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.4 KiB/s wr, 15 op/s 2024-03-19T01:10:56.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:56 smithi156 ceph-mon[66799]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 4.4 KiB/s wr, 15 op/s 2024-03-19T01:10:58.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:10:58 smithi162 ceph-mon[67178]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-03-19T01:10:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:10:58 smithi116 ceph-mon[62701]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-03-19T01:10:58.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:10:58 smithi156 ceph-mon[66799]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2024-03-19T01:10:59.178 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:10:59.179 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:00.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:00 smithi116 ceph-mon[62701]: pgmap v301: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-19T01:11:00.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:00 smithi156 ceph-mon[66799]: pgmap v301: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-19T01:11:00.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:00 smithi162 ceph-mon[67178]: pgmap v301: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-19T01:11:02.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.987+0000 7f1dc1506700 1 -- 172.21.15.116:0/2715227691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dbc107250 msgr2=0x7f1dbc107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:02.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.987+0000 7f1dc1506700 1 --2- 172.21.15.116:0/2715227691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dbc107250 0x7f1dbc107630 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1dac009990 tx=0x7f1dac009ca0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.987+0000 7f1dc1506700 1 -- 172.21.15.116:0/2715227691 shutdown_connections 2024-03-19T01:11:02.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.987+0000 7f1dc1506700 1 --2- 172.21.15.116:0/2715227691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dbc101bb0 0x7f1dbc10e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.987+0000 7f1dc1506700 1 --2- 172.21.15.116:0/2715227691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dbc107250 0x7f1dbc107630 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.987+0000 7f1dc1506700 1 --2- 172.21.15.116:0/2715227691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dbc101210 0x7f1dbc101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.987+0000 7f1dc1506700 1 -- 172.21.15.116:0/2715227691 >> 172.21.15.116:0/2715227691 conn(0x7f1dbc078140 msgr2=0x7f1dbc078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:02.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.987+0000 7f1dc1506700 1 -- 172.21.15.116:0/2715227691 shutdown_connections 2024-03-19T01:11:02.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.987+0000 7f1dc1506700 1 -- 172.21.15.116:0/2715227691 wait complete. 2024-03-19T01:11:02.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.988+0000 7f1dc1506700 1 Processor -- start 2024-03-19T01:11:02.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.988+0000 7f1dc1506700 1 -- start start 2024-03-19T01:11:02.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.988+0000 7f1dc1506700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dbc101210 0x7f1dbc19d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:02.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.988+0000 7f1dc1506700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dbc101bb0 0x7f1dbc19db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:02.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.988+0000 7f1dc1506700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dbc107250 0x7f1dbc1977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:02.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.988+0000 7f1dc1506700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc1114b0 con 0x7f1dbc101210 2024-03-19T01:11:02.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.988+0000 7f1dc1506700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc111330 con 0x7f1dbc107250 2024-03-19T01:11:02.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.988+0000 7f1dc1506700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1dbc111630 con 0x7f1dbc101bb0 2024-03-19T01:11:02.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1dbaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dbc101210 0x7f1dbc19d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:02.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1dbaffd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dbc101210 0x7f1dbc19d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37848/0 (socket says 172.21.15.116:37848) 2024-03-19T01:11:02.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1dba7fc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dbc101bb0 0x7f1dbc19db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:02.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1dbaffd700 1 -- 172.21.15.116:0/51786902 learned_addr learned my addr 172.21.15.116:0/51786902 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:02.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1dbb7fe700 1 --2- 172.21.15.116:0/51786902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dbc107250 0x7f1dbc1977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:02.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1dba7fc700 1 -- 172.21.15.116:0/51786902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dbc107250 msgr2=0x7f1dbc1977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1dba7fc700 1 --2- 172.21.15.116:0/51786902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1dbc107250 0x7f1dbc1977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1dba7fc700 1 -- 172.21.15.116:0/51786902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dbc101210 msgr2=0x7f1dbc19d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1dba7fc700 1 --2- 172.21.15.116:0/51786902 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1dbc101210 0x7f1dbc19d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1dba7fc700 1 -- 172.21.15.116:0/51786902 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dbc197fd0 con 0x7f1dbc101bb0 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1dba7fc700 1 --2- 172.21.15.116:0/51786902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dbc101bb0 0x7f1dbc19db60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1db000e9f0 tx=0x7f1db000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1da3fff700 1 -- 172.21.15.116:0/51786902 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db0004220 con 0x7f1dbc101bb0 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1da3fff700 1 -- 172.21.15.116:0/51786902 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1db0004d10 con 0x7f1dbc101bb0 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1dc1506700 1 -- 172.21.15.116:0/51786902 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dbc1982c0 con 0x7f1dbc101bb0 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1da3fff700 1 -- 172.21.15.116:0/51786902 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db0005020 con 0x7f1dbc101bb0 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.989+0000 7f1dc1506700 1 -- 172.21.15.116:0/51786902 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dbc069fc0 con 0x7f1dbc101bb0 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.990+0000 7f1dc1506700 1 -- 172.21.15.116:0/51786902 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1dbc06a5d0 con 0x7f1dbc101bb0 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.992+0000 7f1da3fff700 1 -- 172.21.15.116:0/51786902 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f1db0004750 con 0x7f1dbc101bb0 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.993+0000 7f1da3fff700 1 --2- 172.21.15.116:0/51786902 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1da40a2920 0x7f1da40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:02.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.993+0000 7f1dbaffd700 1 --2- 172.21.15.116:0/51786902 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1da40a2920 0x7f1da40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:02.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.993+0000 7f1da3fff700 1 -- 172.21.15.116:0/51786902 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f1db00cec90 con 0x7f1dbc101bb0 2024-03-19T01:11:02.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.993+0000 7f1da3fff700 1 --2- 172.21.15.116:0/51786902 >> [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241] conn(0x7f1da40ac100 0x7f1da40ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:02.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.993+0000 7f1da3fff700 1 -- 172.21.15.116:0/51786902 --> [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1db0012ec0 con 0x7f1da40ac100 2024-03-19T01:11:02.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.993+0000 7f1da3fff700 1 -- 172.21.15.116:0/51786902 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f1db00cf040 con 0x7f1dbc101bb0 2024-03-19T01:11:02.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.993+0000 7f1dbb7fe700 1 --2- 172.21.15.116:0/51786902 >> [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241] conn(0x7f1da40ac100 0x7f1da40ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:02.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.993+0000 7f1dbaffd700 1 --2- 172.21.15.116:0/51786902 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1da40a2920 0x7f1da40a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00b2e0 tx=0x7f1dac009590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:02.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.993+0000 7f1dbb7fe700 1 --2- 172.21.15.116:0/51786902 >> [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241] conn(0x7f1da40ac100 0x7f1da40ae520 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:02.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.994+0000 7f1da3fff700 1 -- 172.21.15.116:0/51786902 <== osd.6 v2:172.21.15.116:6818/2037449241 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1db0012ec0 con 0x7f1da40ac100 2024-03-19T01:11:02.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:02.010+0000 7f1dc1506700 1 -- 172.21.15.116:0/51786902 --> [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1dbc06aa50 con 0x7f1da40ac100 2024-03-19T01:11:02.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:02.011+0000 7f1da3fff700 1 -- 172.21.15.116:0/51786902 <== osd.6 v2:172.21.15.116:6818/2037449241 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1dbc06aa50 con 0x7f1da40ac100 2024-03-19T01:11:02.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:02.011+0000 7f1dc1506700 1 -- 172.21.15.116:0/51786902 >> [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241] conn(0x7f1da40ac100 msgr2=0x7f1da40ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:02.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:02.011+0000 7f1dc1506700 1 --2- 172.21.15.116:0/51786902 >> [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241] conn(0x7f1da40ac100 0x7f1da40ae520 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:02.011+0000 7f1dc1506700 1 -- 172.21.15.116:0/51786902 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1da40a2920 msgr2=0x7f1da40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:02.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:02.011+0000 7f1dc1506700 1 --2- 172.21.15.116:0/51786902 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1da40a2920 0x7f1da40a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00b2e0 tx=0x7f1dac009590 comp rx=0 tx=0).stop 2024-03-19T01:11:02.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:02.011+0000 7f1dc1506700 1 -- 172.21.15.116:0/51786902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dbc101bb0 msgr2=0x7f1dbc19db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:02.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:02.011+0000 7f1dc1506700 1 --2- 172.21.15.116:0/51786902 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1dbc101bb0 0x7f1dbc19db60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1db000e9f0 tx=0x7f1db000ed00 comp rx=0 tx=0).stop 2024-03-19T01:11:02.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:02.011+0000 7f1dbb7fe700 1 -- 172.21.15.116:0/51786902 reap_dead start 2024-03-19T01:11:02.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:02.011+0000 7f1dc1506700 1 -- 172.21.15.116:0/51786902 shutdown_connections 2024-03-19T01:11:02.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:02.011+0000 7f1dc1506700 1 -- 172.21.15.116:0/51786902 >> 172.21.15.116:0/51786902 conn(0x7f1dbc078140 msgr2=0x7f1dbc0ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:02.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:02.011+0000 7f1dc1506700 1 -- 172.21.15.116:0/51786902 shutdown_connections 2024-03-19T01:11:02.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:02.011+0000 7f1dc1506700 1 -- 172.21.15.116:0/51786902 wait complete. 2024-03-19T01:11:02.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:02 smithi116 ceph-mon[62701]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:11:02.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:02 smithi156 ceph-mon[66799]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:11:02.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:02 smithi162 ceph-mon[67178]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.556+0000 7f6be11a0700 1 -- 172.21.15.116:0/3197237672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc101480 msgr2=0x7f6bdc10f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.556+0000 7f6be11a0700 1 --2- 172.21.15.116:0/3197237672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc101480 0x7f6bdc10f0f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f6bd400b0a0 tx=0x7f6bd400b3b0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.556+0000 7f6be11a0700 1 -- 172.21.15.116:0/3197237672 shutdown_connections 2024-03-19T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.556+0000 7f6be11a0700 1 --2- 172.21.15.116:0/3197237672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc101480 0x7f6bdc10f0f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.556+0000 7f6be11a0700 1 --2- 172.21.15.116:0/3197237672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bdc107230 0x7f6bdc107610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.556+0000 7f6be11a0700 1 --2- 172.21.15.116:0/3197237672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bdc100ac0 0x7f6bdc100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.556+0000 7f6be11a0700 1 -- 172.21.15.116:0/3197237672 >> 172.21.15.116:0/3197237672 conn(0x7f6bdc076450 msgr2=0x7f6bdc076860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.556+0000 7f6be11a0700 1 -- 172.21.15.116:0/3197237672 shutdown_connections 2024-03-19T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.556+0000 7f6be11a0700 1 -- 172.21.15.116:0/3197237672 wait complete. 2024-03-19T01:11:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.557+0000 7f6be11a0700 1 Processor -- start 2024-03-19T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.557+0000 7f6be11a0700 1 -- start start 2024-03-19T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.557+0000 7f6be11a0700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc100ac0 0x7f6bdc19e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.557+0000 7f6be11a0700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bdc101480 0x7f6bdc19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.557+0000 7f6be11a0700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bdc107230 0x7f6bdc198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.557+0000 7f6be11a0700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc112500 con 0x7f6bdc100ac0 2024-03-19T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.557+0000 7f6be11a0700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc112380 con 0x7f6bdc107230 2024-03-19T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.557+0000 7f6be11a0700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6bdc112680 con 0x7f6bdc101480 2024-03-19T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.557+0000 7f6bdbfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc100ac0 0x7f6bdc19e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.557+0000 7f6bdbfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc100ac0 0x7f6bdc19e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37840/0 (socket says 172.21.15.116:37840) 2024-03-19T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.557+0000 7f6bdbfff700 1 -- 172.21.15.116:0/1146504698 learned_addr learned my addr 172.21.15.116:0/1146504698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.557+0000 7f6bdb7fe700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bdc101480 0x7f6bdc19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.557+0000 7f6be099f700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bdc107230 0x7f6bdc198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.558+0000 7f6bdbfff700 1 -- 172.21.15.116:0/1146504698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bdc101480 msgr2=0x7f6bdc19eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.558+0000 7f6bdbfff700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bdc101480 0x7f6bdc19eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.558+0000 7f6bdbfff700 1 -- 172.21.15.116:0/1146504698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bdc107230 msgr2=0x7f6bdc198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.558+0000 7f6bdbfff700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6bdc107230 0x7f6bdc198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.558+0000 7f6bdbfff700 1 -- 172.21.15.116:0/1146504698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bdc198fe0 con 0x7f6bdc100ac0 2024-03-19T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.558+0000 7f6bdb7fe700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6bdc101480 0x7f6bdc19eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.558+0000 7f6bdbfff700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc100ac0 0x7f6bdc19e630 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00e9c0 tx=0x7f6bcc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.558+0000 7f6bd97fa700 1 -- 172.21.15.116:0/1146504698 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00cbd0 con 0x7f6bdc100ac0 2024-03-19T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.558+0000 7f6be11a0700 1 -- 172.21.15.116:0/1146504698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bdc1992d0 con 0x7f6bdc100ac0 2024-03-19T01:11:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.558+0000 7f6bd97fa700 1 -- 172.21.15.116:0/1146504698 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6bcc00ee60 con 0x7f6bdc100ac0 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.558+0000 7f6bd97fa700 1 -- 172.21.15.116:0/1146504698 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc005630 con 0x7f6bdc100ac0 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.558+0000 7f6be11a0700 1 -- 172.21.15.116:0/1146504698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bdc1a55b0 con 0x7f6bdc100ac0 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.559+0000 7f6be11a0700 1 -- 172.21.15.116:0/1146504698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6bdc062a60 con 0x7f6bdc100ac0 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.560+0000 7f6bd97fa700 1 -- 172.21.15.116:0/1146504698 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f6bcc004750 con 0x7f6bdc100ac0 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.561+0000 7f6bd97fa700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6bc40a2920 0x7f6bc40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.561+0000 7f6bdb7fe700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6bc40a2920 0x7f6bc40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.561+0000 7f6bd97fa700 1 -- 172.21.15.116:0/1146504698 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f6bcc0cf240 con 0x7f6bdc100ac0 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.561+0000 7f6bd97fa700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607] conn(0x7f6bc40ac100 0x7f6bc40ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.561+0000 7f6bd97fa700 1 -- 172.21.15.116:0/1146504698 --> [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6bcc01fe50 con 0x7f6bc40ac100 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.561+0000 7f6bd97fa700 1 -- 172.21.15.116:0/1146504698 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f6bcc0cf720 con 0x7f6bdc100ac0 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.561+0000 7f6be099f700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607] conn(0x7f6bc40ac100 0x7f6bc40ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.561+0000 7f6bdb7fe700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6bc40a2920 0x7f6bc40a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc199ec0 tx=0x7f6bd0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.561+0000 7f6be099f700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607] conn(0x7f6bc40ac100 0x7f6bc40ae520 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.562+0000 7f6bd97fa700 1 -- 172.21.15.116:0/1146504698 <== osd.3 v2:172.21.15.116:6810/2026775607 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bcc01fe50 con 0x7f6bc40ac100 2024-03-19T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.579+0000 7f6be11a0700 1 -- 172.21.15.116:0/1146504698 --> [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6bdc062e60 con 0x7f6bc40ac100 2024-03-19T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.579+0000 7f6bd97fa700 1 -- 172.21.15.116:0/1146504698 <== osd.3 v2:172.21.15.116:6810/2026775607 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6bdc062e60 con 0x7f6bc40ac100 2024-03-19T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.579+0000 7f6be11a0700 1 -- 172.21.15.116:0/1146504698 >> [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607] conn(0x7f6bc40ac100 msgr2=0x7f6bc40ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.579+0000 7f6be11a0700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607] conn(0x7f6bc40ac100 0x7f6bc40ae520 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.579+0000 7f6be11a0700 1 -- 172.21.15.116:0/1146504698 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6bc40a2920 msgr2=0x7f6bc40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.579+0000 7f6be11a0700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6bc40a2920 0x7f6bc40a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc199ec0 tx=0x7f6bd0009260 comp rx=0 tx=0).stop 2024-03-19T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.579+0000 7f6be11a0700 1 -- 172.21.15.116:0/1146504698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc100ac0 msgr2=0x7f6bdc19e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.579+0000 7f6be11a0700 1 --2- 172.21.15.116:0/1146504698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bdc100ac0 0x7f6bdc19e630 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc00e9c0 tx=0x7f6bcc00ecd0 comp rx=0 tx=0).stop 2024-03-19T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.579+0000 7f6be099f700 1 -- 172.21.15.116:0/1146504698 reap_dead start 2024-03-19T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.579+0000 7f6be11a0700 1 -- 172.21.15.116:0/1146504698 shutdown_connections 2024-03-19T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.580+0000 7f6be11a0700 1 -- 172.21.15.116:0/1146504698 >> 172.21.15.116:0/1146504698 conn(0x7f6bdc076450 msgr2=0x7f6bdc0fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.580+0000 7f6be11a0700 1 -- 172.21.15.116:0/1146504698 shutdown_connections 2024-03-19T01:11:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:01.580+0000 7f6be11a0700 1 -- 172.21.15.116:0/1146504698 wait complete. 2024-03-19T01:11:03.446 INFO:teuthology.orchestra.run.smithi116.stdout:107374182460 2024-03-19T01:11:03.447 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-19T01:11:03.669 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:03.773 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:04.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:04 smithi116 ceph-mon[62701]: pgmap v303: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:11:04.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:04 smithi156 ceph-mon[66799]: pgmap v303: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:11:04.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:04 smithi162 ceph-mon[67178]: pgmap v303: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:11:05.298 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:05 smithi116 ceph-mon[62701]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-19T01:11:05.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:05 smithi156 ceph-mon[66799]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-19T01:11:05.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:05 smithi162 ceph-mon[67178]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-19T01:11:05.753 INFO:teuthology.orchestra.run.smithi116.stdout:197568495661 2024-03-19T01:11:05.754 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-19T01:11:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.922+0000 7f548be3b700 1 -- 172.21.15.116:0/451511046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5484101210 msgr2=0x7f5484101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.922+0000 7f548be3b700 1 --2- 172.21.15.116:0/451511046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5484101210 0x7f5484101670 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5478009990 tx=0x7f5478009ca0 comp rx=0 tx=0).stop 2024-03-19T01:11:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.922+0000 7f548be3b700 1 -- 172.21.15.116:0/451511046 shutdown_connections 2024-03-19T01:11:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.922+0000 7f548be3b700 1 --2- 172.21.15.116:0/451511046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5484101bb0 0x7f548410e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.922+0000 7f548be3b700 1 --2- 172.21.15.116:0/451511046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5484107250 0x7f5484107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.922+0000 7f548be3b700 1 --2- 172.21.15.116:0/451511046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5484101210 0x7f5484101670 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.922+0000 7f548be3b700 1 -- 172.21.15.116:0/451511046 >> 172.21.15.116:0/451511046 conn(0x7f5484078140 msgr2=0x7f5484078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:05.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.923+0000 7f548be3b700 1 -- 172.21.15.116:0/451511046 shutdown_connections 2024-03-19T01:11:05.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.923+0000 7f548be3b700 1 -- 172.21.15.116:0/451511046 wait complete. 2024-03-19T01:11:05.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.923+0000 7f548be3b700 1 Processor -- start 2024-03-19T01:11:05.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.923+0000 7f548be3b700 1 -- start start 2024-03-19T01:11:05.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f548be3b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5484101210 0x7f548419d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:05.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f548be3b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5484101bb0 0x7f548419db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:05.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f548be3b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5484107250 0x7f5484197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:05.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f548be3b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f54841114b0 con 0x7f5484101210 2024-03-19T01:11:05.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f548be3b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5484111330 con 0x7f5484101bb0 2024-03-19T01:11:05.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f548be3b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5484111630 con 0x7f5484107250 2024-03-19T01:11:05.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f5489bd7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5484101210 0x7f548419d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:05.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f54893d6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5484101bb0 0x7f548419db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:05.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f5489bd7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5484101210 0x7f548419d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37876/0 (socket says 172.21.15.116:37876) 2024-03-19T01:11:05.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f5489bd7700 1 -- 172.21.15.116:0/1356868536 learned_addr learned my addr 172.21.15.116:0/1356868536 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:05.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f548a3d8700 1 --2- 172.21.15.116:0/1356868536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5484107250 0x7f5484197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:05.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f54893d6700 1 -- 172.21.15.116:0/1356868536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5484107250 msgr2=0x7f5484197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:05.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f54893d6700 1 --2- 172.21.15.116:0/1356868536 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5484107250 0x7f5484197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:05.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f54893d6700 1 -- 172.21.15.116:0/1356868536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5484101210 msgr2=0x7f548419d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:05.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f54893d6700 1 --2- 172.21.15.116:0/1356868536 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5484101210 0x7f548419d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:05.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f54893d6700 1 -- 172.21.15.116:0/1356868536 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5484197ff0 con 0x7f5484101bb0 2024-03-19T01:11:05.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f54893d6700 1 --2- 172.21.15.116:0/1356868536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5484101bb0 0x7f548419db80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5478006010 tx=0x7f54780192d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:05.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.924+0000 7f5482ffd700 1 -- 172.21.15.116:0/1356868536 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5478027070 con 0x7f5484101bb0 2024-03-19T01:11:05.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.925+0000 7f548be3b700 1 -- 172.21.15.116:0/1356868536 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5484198280 con 0x7f5484101bb0 2024-03-19T01:11:05.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.925+0000 7f5482ffd700 1 -- 172.21.15.116:0/1356868536 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5478003d60 con 0x7f5484101bb0 2024-03-19T01:11:05.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.925+0000 7f5482ffd700 1 -- 172.21.15.116:0/1356868536 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f547800fdd0 con 0x7f5484101bb0 2024-03-19T01:11:05.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.925+0000 7f548be3b700 1 -- 172.21.15.116:0/1356868536 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5484069fc0 con 0x7f5484101bb0 2024-03-19T01:11:05.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.926+0000 7f548be3b700 1 -- 172.21.15.116:0/1356868536 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f548406a570 con 0x7f5484101bb0 2024-03-19T01:11:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.927+0000 7f5482ffd700 1 -- 172.21.15.116:0/1356868536 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f547802a070 con 0x7f5484101bb0 2024-03-19T01:11:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.927+0000 7f5482ffd700 1 --2- 172.21.15.116:0/1356868536 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f546c0a29a0 0x7f546c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.927+0000 7f5489bd7700 1 --2- 172.21.15.116:0/1356868536 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f546c0a29a0 0x7f546c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.927+0000 7f5482ffd700 1 -- 172.21.15.116:0/1356868536 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f54780cffc0 con 0x7f5484101bb0 2024-03-19T01:11:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.928+0000 7f5482ffd700 1 --2- 172.21.15.116:0/1356868536 >> [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941] conn(0x7f546c0ac180 0x7f546c0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.928+0000 7f5482ffd700 1 -- 172.21.15.116:0/1356868536 --> [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5478023e80 con 0x7f546c0ac180 2024-03-19T01:11:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.928+0000 7f5482ffd700 1 -- 172.21.15.116:0/1356868536 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f54780d04a0 con 0x7f5484101bb0 2024-03-19T01:11:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.928+0000 7f548a3d8700 1 --2- 172.21.15.116:0/1356868536 >> [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941] conn(0x7f546c0ac180 0x7f546c0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.928+0000 7f5489bd7700 1 --2- 172.21.15.116:0/1356868536 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f546c0a29a0 0x7f546c0a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f547c009550 tx=0x7f547c009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.928+0000 7f548a3d8700 1 --2- 172.21.15.116:0/1356868536 >> [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941] conn(0x7f546c0ac180 0x7f546c0ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.929+0000 7f5482ffd700 1 -- 172.21.15.116:0/1356868536 <== osd.11 v2:172.21.15.162:6824/246086941 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5478023e80 con 0x7f546c0ac180 2024-03-19T01:11:05.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.955+0000 7f548be3b700 1 -- 172.21.15.116:0/1356868536 --> [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f548406a970 con 0x7f546c0ac180 2024-03-19T01:11:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.956+0000 7f5482ffd700 1 -- 172.21.15.116:0/1356868536 <== osd.11 v2:172.21.15.162:6824/246086941 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f548406a970 con 0x7f546c0ac180 2024-03-19T01:11:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.956+0000 7f548be3b700 1 -- 172.21.15.116:0/1356868536 >> [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941] conn(0x7f546c0ac180 msgr2=0x7f546c0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.956+0000 7f548be3b700 1 --2- 172.21.15.116:0/1356868536 >> [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941] conn(0x7f546c0ac180 0x7f546c0ae5a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.956+0000 7f548be3b700 1 -- 172.21.15.116:0/1356868536 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f546c0a29a0 msgr2=0x7f546c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:05.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.956+0000 7f548be3b700 1 --2- 172.21.15.116:0/1356868536 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f546c0a29a0 0x7f546c0a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f547c009550 tx=0x7f547c009040 comp rx=0 tx=0).stop 2024-03-19T01:11:05.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.956+0000 7f548be3b700 1 -- 172.21.15.116:0/1356868536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5484101bb0 msgr2=0x7f548419db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:05.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.956+0000 7f548be3b700 1 --2- 172.21.15.116:0/1356868536 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5484101bb0 0x7f548419db80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5478006010 tx=0x7f54780192d0 comp rx=0 tx=0).stop 2024-03-19T01:11:05.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.956+0000 7f548a3d8700 1 -- 172.21.15.116:0/1356868536 reap_dead start 2024-03-19T01:11:05.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.957+0000 7f548be3b700 1 -- 172.21.15.116:0/1356868536 shutdown_connections 2024-03-19T01:11:05.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.957+0000 7f548be3b700 1 -- 172.21.15.116:0/1356868536 >> 172.21.15.116:0/1356868536 conn(0x7f5484078140 msgr2=0x7f54840fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:05.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.957+0000 7f548be3b700 1 -- 172.21.15.116:0/1356868536 shutdown_connections 2024-03-19T01:11:05.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:05.957+0000 7f548be3b700 1 -- 172.21.15.116:0/1356868536 wait complete. 2024-03-19T01:11:06.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.116+0000 7fb696dbb700 1 -- 172.21.15.116:0/1134034130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb690107240 msgr2=0x7fb690107620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:06.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.116+0000 7fb696dbb700 1 --2- 172.21.15.116:0/1134034130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb690107240 0x7fb690107620 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb684009990 tx=0x7fb684009ca0 comp rx=0 tx=0).stop 2024-03-19T01:11:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.116+0000 7fb696dbb700 1 -- 172.21.15.116:0/1134034130 shutdown_connections 2024-03-19T01:11:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.116+0000 7fb696dbb700 1 --2- 172.21.15.116:0/1134034130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb690101bc0 0x7fb69010e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.116+0000 7fb696dbb700 1 --2- 172.21.15.116:0/1134034130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb690107240 0x7fb690107620 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.116+0000 7fb696dbb700 1 --2- 172.21.15.116:0/1134034130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb690101200 0x7fb690101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.116+0000 7fb696dbb700 1 -- 172.21.15.116:0/1134034130 >> 172.21.15.116:0/1134034130 conn(0x7fb6900781c0 msgr2=0x7fb6900785d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.116+0000 7fb696dbb700 1 -- 172.21.15.116:0/1134034130 shutdown_connections 2024-03-19T01:11:06.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.116+0000 7fb696dbb700 1 -- 172.21.15.116:0/1134034130 wait complete. 2024-03-19T01:11:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.117+0000 7fb696dbb700 1 Processor -- start 2024-03-19T01:11:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.117+0000 7fb696dbb700 1 -- start start 2024-03-19T01:11:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.117+0000 7fb696dbb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb690101200 0x7fb69019d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.117+0000 7fb696dbb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb690101bc0 0x7fb69019dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:06.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb696dbb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb690107240 0x7fb6901977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb696dbb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb6901114b0 con 0x7fb690101bc0 2024-03-19T01:11:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb696dbb700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb690111330 con 0x7fb690101200 2024-03-19T01:11:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb696dbb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb690111630 con 0x7fb690107240 2024-03-19T01:11:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb68ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb690101bc0 0x7fb69019dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb68ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb690101bc0 0x7fb69019dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37886/0 (socket says 172.21.15.116:37886) 2024-03-19T01:11:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb694b57700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb690101200 0x7fb69019d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb68ffff700 1 -- 172.21.15.116:0/3436703314 learned_addr learned my addr 172.21.15.116:0/3436703314 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb695358700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb690107240 0x7fb6901977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb694b57700 1 -- 172.21.15.116:0/3436703314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb690107240 msgr2=0x7fb6901977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb694b57700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb690107240 0x7fb6901977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb694b57700 1 -- 172.21.15.116:0/3436703314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb690101bc0 msgr2=0x7fb69019dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:06.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb694b57700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb690101bc0 0x7fb69019dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb694b57700 1 -- 172.21.15.116:0/3436703314 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb690198090 con 0x7fb690101200 2024-03-19T01:11:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb68ffff700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb690101bc0 0x7fb69019dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb694b57700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb690101200 0x7fb69019d690 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb684006010 tx=0x7fb68400ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb68dffb700 1 -- 172.21.15.116:0/3436703314 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684027070 con 0x7fb690101200 2024-03-19T01:11:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb68dffb700 1 -- 172.21.15.116:0/3436703314 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb684003d70 con 0x7fb690101200 2024-03-19T01:11:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb696dbb700 1 -- 172.21.15.116:0/3436703314 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb690198320 con 0x7fb690101200 2024-03-19T01:11:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.118+0000 7fb68dffb700 1 -- 172.21.15.116:0/3436703314 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684022e00 con 0x7fb690101200 2024-03-19T01:11:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.119+0000 7fb696dbb700 1 -- 172.21.15.116:0/3436703314 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6901a4610 con 0x7fb690101200 2024-03-19T01:11:06.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.119+0000 7fb696dbb700 1 -- 172.21.15.116:0/3436703314 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb690062a60 con 0x7fb690101200 2024-03-19T01:11:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.121+0000 7fb68dffb700 1 -- 172.21.15.116:0/3436703314 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fb68402a070 con 0x7fb690101200 2024-03-19T01:11:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.121+0000 7fb68dffb700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6800a29a0 0x7fb6800a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.121+0000 7fb68ffff700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6800a29a0 0x7fb6800a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:06.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.121+0000 7fb68dffb700 1 -- 172.21.15.116:0/3436703314 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fb6840d0170 con 0x7fb690101200 2024-03-19T01:11:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.121+0000 7fb68dffb700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811] conn(0x7fb6800ac180 0x7fb6800ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.121+0000 7fb68dffb700 1 -- 172.21.15.116:0/3436703314 --> [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6800aec50 con 0x7fb6800ac180 2024-03-19T01:11:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.121+0000 7fb68dffb700 1 -- 172.21.15.116:0/3436703314 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fb6840d0520 con 0x7fb690101200 2024-03-19T01:11:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.122+0000 7fb695358700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811] conn(0x7fb6800ac180 0x7fb6800ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.122+0000 7fb68ffff700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6800a29a0 0x7fb6800a4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb690198f20 tx=0x7fb67c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.122+0000 7fb695358700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811] conn(0x7fb6800ac180 0x7fb6800ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:06.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.123+0000 7fb68dffb700 1 -- 172.21.15.116:0/3436703314 <== osd.0 v2:172.21.15.116:6802/1461395811 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb6800aec50 con 0x7fb6800ac180 2024-03-19T01:11:06.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.139+0000 7fb696dbb700 1 -- 172.21.15.116:0/3436703314 --> [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb690062e60 con 0x7fb6800ac180 2024-03-19T01:11:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.139+0000 7fb68dffb700 1 -- 172.21.15.116:0/3436703314 <== osd.0 v2:172.21.15.116:6802/1461395811 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb690062e60 con 0x7fb6800ac180 2024-03-19T01:11:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.139+0000 7fb696dbb700 1 -- 172.21.15.116:0/3436703314 >> [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811] conn(0x7fb6800ac180 msgr2=0x7fb6800ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.139+0000 7fb696dbb700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811] conn(0x7fb6800ac180 0x7fb6800ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.139+0000 7fb696dbb700 1 -- 172.21.15.116:0/3436703314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6800a29a0 msgr2=0x7fb6800a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.139+0000 7fb696dbb700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6800a29a0 0x7fb6800a4e60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb690198f20 tx=0x7fb67c009260 comp rx=0 tx=0).stop 2024-03-19T01:11:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.139+0000 7fb696dbb700 1 -- 172.21.15.116:0/3436703314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb690101200 msgr2=0x7fb69019d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.139+0000 7fb696dbb700 1 --2- 172.21.15.116:0/3436703314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb690101200 0x7fb69019d690 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb684006010 tx=0x7fb68400ba90 comp rx=0 tx=0).stop 2024-03-19T01:11:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.140+0000 7fb695358700 1 -- 172.21.15.116:0/3436703314 reap_dead start 2024-03-19T01:11:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.140+0000 7fb696dbb700 1 -- 172.21.15.116:0/3436703314 shutdown_connections 2024-03-19T01:11:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.140+0000 7fb696dbb700 1 -- 172.21.15.116:0/3436703314 >> 172.21.15.116:0/3436703314 conn(0x7fb6900781c0 msgr2=0x7fb6900ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.140+0000 7fb696dbb700 1 -- 172.21.15.116:0/3436703314 shutdown_connections 2024-03-19T01:11:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:06.140+0000 7fb696dbb700 1 -- 172.21.15.116:0/3436703314 wait complete. 2024-03-19T01:11:07.079 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:08.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:08 smithi162 ceph-mon[67178]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:11:08.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:08 smithi116 ceph-mon[62701]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:11:08.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:08 smithi156 ceph-mon[66799]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:11:09.842 INFO:teuthology.orchestra.run.smithi116.stdout:34359738444 2024-03-19T01:11:09.843 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-19T01:11:10.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.039+0000 7eff42dd6700 1 -- 172.21.15.116:0/1533290019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c06aab0 msgr2=0x7eff3c06ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:10.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.039+0000 7eff42dd6700 1 --2- 172.21.15.116:0/1533290019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c06aab0 0x7eff3c06ae90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7eff300099c0 tx=0x7eff30009cd0 comp rx=0 tx=0).stop 2024-03-19T01:11:10.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.039+0000 7eff42dd6700 1 -- 172.21.15.116:0/1533290019 shutdown_connections 2024-03-19T01:11:10.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.039+0000 7eff42dd6700 1 --2- 172.21.15.116:0/1533290019 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3c107690 0x7eff3c10f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:10.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.039+0000 7eff42dd6700 1 --2- 172.21.15.116:0/1533290019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c06aab0 0x7eff3c06ae90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.039+0000 7eff42dd6700 1 --2- 172.21.15.116:0/1533290019 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7eff3c06b460 0x7eff3c106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.039+0000 7eff42dd6700 1 -- 172.21.15.116:0/1533290019 >> 172.21.15.116:0/1533290019 conn(0x7eff3c077260 msgr2=0x7eff3c077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.039+0000 7eff42dd6700 1 -- 172.21.15.116:0/1533290019 shutdown_connections 2024-03-19T01:11:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.039+0000 7eff42dd6700 1 -- 172.21.15.116:0/1533290019 wait complete. 2024-03-19T01:11:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.040+0000 7eff42dd6700 1 Processor -- start 2024-03-19T01:11:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.040+0000 7eff42dd6700 1 -- start start 2024-03-19T01:11:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.040+0000 7eff42dd6700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c06aab0 0x7eff3c101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.040+0000 7eff42dd6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3c06b460 0x7eff3c101850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.040+0000 7eff42dd6700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7eff3c107690 0x7eff3c101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.040+0000 7eff42dd6700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c112640 con 0x7eff3c06aab0 2024-03-19T01:11:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.040+0000 7eff42dd6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c1124c0 con 0x7eff3c06b460 2024-03-19T01:11:10.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.040+0000 7eff42dd6700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7eff3c1127c0 con 0x7eff3c107690 2024-03-19T01:11:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.040+0000 7eff40b72700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c06aab0 0x7eff3c101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.040+0000 7eff40b72700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c06aab0 0x7eff3c101310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45678/0 (socket says 172.21.15.116:45678) 2024-03-19T01:11:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff40b72700 1 -- 172.21.15.116:0/1054079013 learned_addr learned my addr 172.21.15.116:0/1054079013 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff3bfff700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3c06b460 0x7eff3c101850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff41373700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7eff3c107690 0x7eff3c101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff3bfff700 1 -- 172.21.15.116:0/1054079013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7eff3c107690 msgr2=0x7eff3c101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff3bfff700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7eff3c107690 0x7eff3c101d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff3bfff700 1 -- 172.21.15.116:0/1054079013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c06aab0 msgr2=0x7eff3c101310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff3bfff700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c06aab0 0x7eff3c101310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff3bfff700 1 -- 172.21.15.116:0/1054079013 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff3c1a7050 con 0x7eff3c06b460 2024-03-19T01:11:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff40b72700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7eff3c06aab0 0x7eff3c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:10.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff3bfff700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3c06b460 0x7eff3c101850 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7eff3c103790 tx=0x7eff2800eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:10.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff39ffb700 1 -- 172.21.15.116:0/1054079013 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28004220 con 0x7eff3c06b460 2024-03-19T01:11:10.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff39ffb700 1 -- 172.21.15.116:0/1054079013 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff28004d10 con 0x7eff3c06b460 2024-03-19T01:11:10.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff42dd6700 1 -- 172.21.15.116:0/1054079013 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff3c1a7340 con 0x7eff3c06b460 2024-03-19T01:11:10.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.041+0000 7eff39ffb700 1 -- 172.21.15.116:0/1054079013 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28010700 con 0x7eff3c06b460 2024-03-19T01:11:10.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.042+0000 7eff42dd6700 1 -- 172.21.15.116:0/1054079013 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff3c1a7880 con 0x7eff3c06b460 2024-03-19T01:11:10.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.042+0000 7eff42dd6700 1 -- 172.21.15.116:0/1054079013 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7eff3c062a20 con 0x7eff3c06b460 2024-03-19T01:11:10.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.044+0000 7eff39ffb700 1 -- 172.21.15.116:0/1054079013 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7eff28004750 con 0x7eff3c06b460 2024-03-19T01:11:10.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.044+0000 7eff39ffb700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7eff2c0a29a0 0x7eff2c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:10.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.045+0000 7eff40b72700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7eff2c0a29a0 0x7eff2c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:10.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.045+0000 7eff39ffb700 1 -- 172.21.15.116:0/1054079013 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7eff280ce9e0 con 0x7eff3c06b460 2024-03-19T01:11:10.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.045+0000 7eff39ffb700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633] conn(0x7eff2c0ac180 0x7eff2c0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:10.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.045+0000 7eff39ffb700 1 -- 172.21.15.116:0/1054079013 --> [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff2c0aec50 con 0x7eff2c0ac180 2024-03-19T01:11:10.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.045+0000 7eff39ffb700 1 -- 172.21.15.116:0/1054079013 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7eff280ceec0 con 0x7eff3c06b460 2024-03-19T01:11:10.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.045+0000 7eff41373700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633] conn(0x7eff2c0ac180 0x7eff2c0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:10.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.045+0000 7eff40b72700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7eff2c0a29a0 0x7eff2c0a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7eff30009990 tx=0x7eff30005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:10.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.045+0000 7eff41373700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633] conn(0x7eff2c0ac180 0x7eff2c0ae5a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:10.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.046+0000 7eff39ffb700 1 -- 172.21.15.116:0/1054079013 <== osd.5 v2:172.21.15.162:6808/2786247633 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff2c0aec50 con 0x7eff2c0ac180 2024-03-19T01:11:10.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.063+0000 7eff42dd6700 1 -- 172.21.15.116:0/1054079013 --> [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7eff3c062ea0 con 0x7eff2c0ac180 2024-03-19T01:11:10.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.063+0000 7eff39ffb700 1 -- 172.21.15.116:0/1054079013 <== osd.5 v2:172.21.15.162:6808/2786247633 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7eff3c062ea0 con 0x7eff2c0ac180 2024-03-19T01:11:10.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.064+0000 7eff42dd6700 1 -- 172.21.15.116:0/1054079013 >> [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633] conn(0x7eff2c0ac180 msgr2=0x7eff2c0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.064+0000 7eff42dd6700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633] conn(0x7eff2c0ac180 0x7eff2c0ae5a0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.064+0000 7eff42dd6700 1 -- 172.21.15.116:0/1054079013 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7eff2c0a29a0 msgr2=0x7eff2c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.064+0000 7eff42dd6700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7eff2c0a29a0 0x7eff2c0a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7eff30009990 tx=0x7eff30005960 comp rx=0 tx=0).stop 2024-03-19T01:11:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.064+0000 7eff42dd6700 1 -- 172.21.15.116:0/1054079013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3c06b460 msgr2=0x7eff3c101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.064+0000 7eff42dd6700 1 --2- 172.21.15.116:0/1054079013 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff3c06b460 0x7eff3c101850 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7eff3c103790 tx=0x7eff2800eca0 comp rx=0 tx=0).stop 2024-03-19T01:11:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.064+0000 7eff41373700 1 -- 172.21.15.116:0/1054079013 reap_dead start 2024-03-19T01:11:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.064+0000 7eff42dd6700 1 -- 172.21.15.116:0/1054079013 shutdown_connections 2024-03-19T01:11:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.064+0000 7eff42dd6700 1 -- 172.21.15.116:0/1054079013 >> 172.21.15.116:0/1054079013 conn(0x7eff3c077260 msgr2=0x7eff3c1087c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.064+0000 7eff42dd6700 1 -- 172.21.15.116:0/1054079013 shutdown_connections 2024-03-19T01:11:10.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:10.064+0000 7eff42dd6700 1 -- 172.21.15.116:0/1054079013 wait complete. 2024-03-19T01:11:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:10 smithi162 ceph-mon[67178]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-19T01:11:10.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:10 smithi116 ceph-mon[62701]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-19T01:11:10.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:10 smithi156 ceph-mon[66799]: pgmap v306: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 597 B/s wr, 12 op/s 2024-03-19T01:11:12.356 INFO:teuthology.orchestra.run.smithi116.stdout:343597383700 2024-03-19T01:11:12.356 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-19T01:11:12.380 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:12 smithi116 ceph-mon[62701]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-19T01:11:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:12 smithi162 ceph-mon[67178]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-19T01:11:12.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:12 smithi156 ceph-mon[66799]: pgmap v307: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-19T01:11:12.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.976+0000 7f7dba79d700 1 -- 172.21.15.116:0/193720946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db4102e00 msgr2=0x7f7db4106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:12.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.976+0000 7f7dba79d700 1 --2- 172.21.15.116:0/193720946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db4102e00 0x7f7db4106e50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7da4009920 tx=0x7f7da4009c30 comp rx=0 tx=0).stop 2024-03-19T01:11:12.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.976+0000 7f7dba79d700 1 -- 172.21.15.116:0/193720946 shutdown_connections 2024-03-19T01:11:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.976+0000 7f7dba79d700 1 --2- 172.21.15.116:0/193720946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7db4107530 0x7f7db410f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.976+0000 7f7dba79d700 1 --2- 172.21.15.116:0/193720946 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db4102450 0x7f7db4102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.976+0000 7f7dba79d700 1 --2- 172.21.15.116:0/193720946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db4102e00 0x7f7db4106e50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.976+0000 7f7dba79d700 1 -- 172.21.15.116:0/193720946 >> 172.21.15.116:0/193720946 conn(0x7f7db40fdd00 msgr2=0x7f7db4100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:12.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.976+0000 7f7dba79d700 1 -- 172.21.15.116:0/193720946 shutdown_connections 2024-03-19T01:11:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.977+0000 7f7dba79d700 1 -- 172.21.15.116:0/193720946 wait complete. 2024-03-19T01:11:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7dba79d700 1 Processor -- start 2024-03-19T01:11:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7dba79d700 1 -- start start 2024-03-19T01:11:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7dba79d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db4102450 0x7f7db419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7dba79d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db4102e00 0x7f7db419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7dba79d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7db4107530 0x7f7db4198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:12.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7dba79d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7db41124f0 con 0x7f7db4102450 2024-03-19T01:11:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7dba79d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7db4112370 con 0x7f7db4107530 2024-03-19T01:11:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7dba79d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7db4112670 con 0x7f7db4102e00 2024-03-19T01:11:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7db979b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db4102450 0x7f7db419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7db979b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db4102450 0x7f7db419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45686/0 (socket says 172.21.15.116:45686) 2024-03-19T01:11:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7db979b700 1 -- 172.21.15.116:0/2333951873 learned_addr learned my addr 172.21.15.116:0/2333951873 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7db8f9a700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db4102e00 0x7f7db419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7db9f9c700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7db4107530 0x7f7db4198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7db979b700 1 -- 172.21.15.116:0/2333951873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db4102e00 msgr2=0x7f7db419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7db979b700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db4102e00 0x7f7db419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7db979b700 1 -- 172.21.15.116:0/2333951873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7db4107530 msgr2=0x7f7db4198870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7db979b700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7db4107530 0x7f7db4198870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7db979b700 1 -- 172.21.15.116:0/2333951873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db4199050 con 0x7f7db4102450 2024-03-19T01:11:12.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.978+0000 7f7db8f9a700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7db4102e00 0x7f7db419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.979+0000 7f7db979b700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db4102450 0x7f7db419e6a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7db000ec10 tx=0x7f7db000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.979+0000 7f7daa7fc700 1 -- 172.21.15.116:0/2333951873 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db000cc90 con 0x7f7db4102450 2024-03-19T01:11:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.979+0000 7f7daa7fc700 1 -- 172.21.15.116:0/2333951873 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7db000ce30 con 0x7f7db4102450 2024-03-19T01:11:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.979+0000 7f7daa7fc700 1 -- 172.21.15.116:0/2333951873 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db0010690 con 0x7f7db4102450 2024-03-19T01:11:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.979+0000 7f7dba79d700 1 -- 172.21.15.116:0/2333951873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db4199340 con 0x7f7db4102450 2024-03-19T01:11:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.979+0000 7f7dba79d700 1 -- 172.21.15.116:0/2333951873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db41a5500 con 0x7f7db4102450 2024-03-19T01:11:12.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.980+0000 7f7dba79d700 1 -- 172.21.15.116:0/2333951873 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7db4062a60 con 0x7f7db4102450 2024-03-19T01:11:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.981+0000 7f7daa7fc700 1 -- 172.21.15.116:0/2333951873 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f7db0003680 con 0x7f7db4102450 2024-03-19T01:11:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.981+0000 7f7daa7fc700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7da00a29a0 0x7f7da00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.982+0000 7f7db8f9a700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7da00a29a0 0x7f7da00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.982+0000 7f7daa7fc700 1 -- 172.21.15.116:0/2333951873 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f7db00cefe0 con 0x7f7db4102450 2024-03-19T01:11:12.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.982+0000 7f7daa7fc700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382] conn(0x7f7da00ac180 0x7f7da00ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.982+0000 7f7daa7fc700 1 -- 172.21.15.116:0/2333951873 --> [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7db0012ec0 con 0x7f7da00ac180 2024-03-19T01:11:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.982+0000 7f7daa7fc700 1 -- 172.21.15.116:0/2333951873 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f7db00cf4c0 con 0x7f7db4102450 2024-03-19T01:11:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.982+0000 7f7db9f9c700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382] conn(0x7f7da00ac180 0x7f7da00ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.982+0000 7f7db8f9a700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7da00a29a0 0x7f7da00a4e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7da4009810 tx=0x7f7da4019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.982+0000 7f7db9f9c700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382] conn(0x7f7da00ac180 0x7f7da00ae5a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:12.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:12.984+0000 7f7daa7fc700 1 -- 172.21.15.116:0/2333951873 <== osd.4 v2:172.21.15.156:6808/1262076382 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7db0012ec0 con 0x7f7da00ac180 2024-03-19T01:11:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.009+0000 7f7dba79d700 1 -- 172.21.15.116:0/2333951873 --> [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7db4062eb0 con 0x7f7da00ac180 2024-03-19T01:11:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.010+0000 7f7daa7fc700 1 -- 172.21.15.116:0/2333951873 <== osd.4 v2:172.21.15.156:6808/1262076382 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7db4062eb0 con 0x7f7da00ac180 2024-03-19T01:11:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.010+0000 7f7dba79d700 1 -- 172.21.15.116:0/2333951873 >> [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382] conn(0x7f7da00ac180 msgr2=0x7f7da00ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.010+0000 7f7dba79d700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382] conn(0x7f7da00ac180 0x7f7da00ae5a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.010+0000 7f7dba79d700 1 -- 172.21.15.116:0/2333951873 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7da00a29a0 msgr2=0x7f7da00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.010+0000 7f7dba79d700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7da00a29a0 0x7f7da00a4e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7da4009810 tx=0x7f7da4019040 comp rx=0 tx=0).stop 2024-03-19T01:11:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.010+0000 7f7dba79d700 1 -- 172.21.15.116:0/2333951873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db4102450 msgr2=0x7f7db419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.010+0000 7f7dba79d700 1 --2- 172.21.15.116:0/2333951873 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db4102450 0x7f7db419e6a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7db000ec10 tx=0x7f7db000ef20 comp rx=0 tx=0).stop 2024-03-19T01:11:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.010+0000 7f7db9f9c700 1 -- 172.21.15.116:0/2333951873 reap_dead start 2024-03-19T01:11:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.010+0000 7f7dba79d700 1 -- 172.21.15.116:0/2333951873 shutdown_connections 2024-03-19T01:11:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.010+0000 7f7dba79d700 1 -- 172.21.15.116:0/2333951873 >> 172.21.15.116:0/2333951873 conn(0x7f7db40fdd00 msgr2=0x7f7db41000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.010+0000 7f7dba79d700 1 -- 172.21.15.116:0/2333951873 shutdown_connections 2024-03-19T01:11:13.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.010+0000 7f7dba79d700 1 -- 172.21.15.116:0/2333951873 wait complete. 2024-03-19T01:11:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.185+0000 7fc191582700 1 -- 172.21.15.116:0/1165089930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc18c101210 msgr2=0x7fc18c101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.185+0000 7fc191582700 1 --2- 172.21.15.116:0/1165089930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc18c101210 0x7fc18c101670 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc180009990 tx=0x7fc180009ca0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.185+0000 7fc191582700 1 -- 172.21.15.116:0/1165089930 shutdown_connections 2024-03-19T01:11:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.185+0000 7fc191582700 1 --2- 172.21.15.116:0/1165089930 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18c101bb0 0x7fc18c10e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.185+0000 7fc191582700 1 --2- 172.21.15.116:0/1165089930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc18c107250 0x7fc18c107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.185+0000 7fc191582700 1 --2- 172.21.15.116:0/1165089930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc18c101210 0x7fc18c101670 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.185+0000 7fc191582700 1 -- 172.21.15.116:0/1165089930 >> 172.21.15.116:0/1165089930 conn(0x7fc18c078140 msgr2=0x7fc18c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.185+0000 7fc191582700 1 -- 172.21.15.116:0/1165089930 shutdown_connections 2024-03-19T01:11:13.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.185+0000 7fc191582700 1 -- 172.21.15.116:0/1165089930 wait complete. 2024-03-19T01:11:13.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.186+0000 7fc191582700 1 Processor -- start 2024-03-19T01:11:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.186+0000 7fc191582700 1 -- start start 2024-03-19T01:11:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.186+0000 7fc191582700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc18c101210 0x7fc18c19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.186+0000 7fc191582700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18c101bb0 0x7fc18c19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.186+0000 7fc191582700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc18c107250 0x7fc18c198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.186+0000 7fc191582700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc18c1114d0 con 0x7fc18c101210 2024-03-19T01:11:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.186+0000 7fc191582700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc18c111350 con 0x7fc18c107250 2024-03-19T01:11:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.186+0000 7fc191582700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc18c111650 con 0x7fc18c101bb0 2024-03-19T01:11:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.186+0000 7fc18affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc18c101210 0x7fc18c19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.186+0000 7fc18affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc18c101210 0x7fc18c19e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45716/0 (socket says 172.21.15.116:45716) 2024-03-19T01:11:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.186+0000 7fc18affd700 1 -- 172.21.15.116:0/1357596572 learned_addr learned my addr 172.21.15.116:0/1357596572 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.186+0000 7fc18b7fe700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc18c107250 0x7fc18c198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.186+0000 7fc18a7fc700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18c101bb0 0x7fc18c19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:13.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.187+0000 7fc18affd700 1 -- 172.21.15.116:0/1357596572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18c101bb0 msgr2=0x7fc18c19ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.187+0000 7fc18affd700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc18c101bb0 0x7fc18c19ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.187+0000 7fc18affd700 1 -- 172.21.15.116:0/1357596572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc18c107250 msgr2=0x7fc18c198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.187+0000 7fc18affd700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc18c107250 0x7fc18c198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.187+0000 7fc18affd700 1 -- 172.21.15.116:0/1357596572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18c199010 con 0x7fc18c101210 2024-03-19T01:11:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.187+0000 7fc18b7fe700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc18c107250 0x7fc18c198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.187+0000 7fc18affd700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc18c101210 0x7fc18c19e6b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009cb0 tx=0x7fc17c00b2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.187+0000 7fc173fff700 1 -- 172.21.15.116:0/1357596572 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c01e070 con 0x7fc18c101210 2024-03-19T01:11:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.187+0000 7fc173fff700 1 -- 172.21.15.116:0/1357596572 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc17c00bb50 con 0x7fc18c101210 2024-03-19T01:11:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.187+0000 7fc173fff700 1 -- 172.21.15.116:0/1357596572 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17c018610 con 0x7fc18c101210 2024-03-19T01:11:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.187+0000 7fc191582700 1 -- 172.21.15.116:0/1357596572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc18c199300 con 0x7fc18c101210 2024-03-19T01:11:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.187+0000 7fc191582700 1 -- 172.21.15.116:0/1357596572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc18c102df0 con 0x7fc18c101210 2024-03-19T01:11:13.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.188+0000 7fc191582700 1 -- 172.21.15.116:0/1357596572 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc18c062a20 con 0x7fc18c101210 2024-03-19T01:11:13.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.190+0000 7fc173fff700 1 -- 172.21.15.116:0/1357596572 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fc17c021070 con 0x7fc18c101210 2024-03-19T01:11:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.190+0000 7fc173fff700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc1740a2920 0x7fc1740a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.190+0000 7fc18a7fc700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc1740a2920 0x7fc1740a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.190+0000 7fc173fff700 1 -- 172.21.15.116:0/1357596572 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fc17c0d2550 con 0x7fc18c101210 2024-03-19T01:11:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.191+0000 7fc173fff700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631] conn(0x7fc1740ac100 0x7fc1740ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.191+0000 7fc173fff700 1 -- 172.21.15.116:0/1357596572 --> [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc17c021e50 con 0x7fc1740ac100 2024-03-19T01:11:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.191+0000 7fc173fff700 1 -- 172.21.15.116:0/1357596572 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fc17c0d2900 con 0x7fc18c101210 2024-03-19T01:11:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.191+0000 7fc18a7fc700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc1740a2920 0x7fc1740a4de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc18000b2e0 tx=0x7fc180005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.191+0000 7fc18b7fe700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631] conn(0x7fc1740ac100 0x7fc1740ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.191+0000 7fc18b7fe700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631] conn(0x7fc1740ac100 0x7fc1740ae520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:13.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.192+0000 7fc173fff700 1 -- 172.21.15.116:0/1357596572 <== osd.10 v2:172.21.15.156:6824/3316972631 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc17c021e50 con 0x7fc1740ac100 2024-03-19T01:11:13.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.208+0000 7fc191582700 1 -- 172.21.15.116:0/1357596572 --> [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc18c062ea0 con 0x7fc1740ac100 2024-03-19T01:11:13.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.208+0000 7fc173fff700 1 -- 172.21.15.116:0/1357596572 <== osd.10 v2:172.21.15.156:6824/3316972631 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc18c062ea0 con 0x7fc1740ac100 2024-03-19T01:11:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.209+0000 7fc191582700 1 -- 172.21.15.116:0/1357596572 >> [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631] conn(0x7fc1740ac100 msgr2=0x7fc1740ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.209+0000 7fc191582700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631] conn(0x7fc1740ac100 0x7fc1740ae520 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.209+0000 7fc191582700 1 -- 172.21.15.116:0/1357596572 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc1740a2920 msgr2=0x7fc1740a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.209+0000 7fc191582700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc1740a2920 0x7fc1740a4de0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc18000b2e0 tx=0x7fc180005960 comp rx=0 tx=0).stop 2024-03-19T01:11:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.209+0000 7fc191582700 1 -- 172.21.15.116:0/1357596572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc18c101210 msgr2=0x7fc18c19e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.209+0000 7fc191582700 1 --2- 172.21.15.116:0/1357596572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc18c101210 0x7fc18c19e6b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009cb0 tx=0x7fc17c00b2c0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.209+0000 7fc18b7fe700 1 -- 172.21.15.116:0/1357596572 reap_dead start 2024-03-19T01:11:13.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.209+0000 7fc191582700 1 -- 172.21.15.116:0/1357596572 shutdown_connections 2024-03-19T01:11:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.209+0000 7fc191582700 1 -- 172.21.15.116:0/1357596572 >> 172.21.15.116:0/1357596572 conn(0x7fc18c078140 msgr2=0x7fc18c100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.209+0000 7fc191582700 1 -- 172.21.15.116:0/1357596572 shutdown_connections 2024-03-19T01:11:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.209+0000 7fc191582700 1 -- 172.21.15.116:0/1357596572 wait complete. 2024-03-19T01:11:13.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.673+0000 7f13838a8700 1 -- 172.21.15.116:0/3692902460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f137c102f00 msgr2=0x7f137c106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.673+0000 7f13838a8700 1 --2- 172.21.15.116:0/3692902460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f137c102f00 0x7f137c106f50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f136c0099c0 tx=0x7f136c009cd0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.673+0000 7f13838a8700 1 -- 172.21.15.116:0/3692902460 shutdown_connections 2024-03-19T01:11:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.673+0000 7f13838a8700 1 --2- 172.21.15.116:0/3692902460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f137c107630 0x7f137c10f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.673+0000 7f13838a8700 1 --2- 172.21.15.116:0/3692902460 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f137c102550 0x7f137c102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.673+0000 7f13838a8700 1 --2- 172.21.15.116:0/3692902460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f137c102f00 0x7f137c106f50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.673+0000 7f13838a8700 1 -- 172.21.15.116:0/3692902460 >> 172.21.15.116:0/3692902460 conn(0x7f137c0fddc0 msgr2=0x7f137c1001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.673+0000 7f13838a8700 1 -- 172.21.15.116:0/3692902460 shutdown_connections 2024-03-19T01:11:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.673+0000 7f13838a8700 1 -- 172.21.15.116:0/3692902460 wait complete. 2024-03-19T01:11:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f13838a8700 1 Processor -- start 2024-03-19T01:11:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f13838a8700 1 -- start start 2024-03-19T01:11:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f13838a8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f137c102550 0x7f137c19e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f13838a8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f137c102f00 0x7f137c19ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f13838a8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f137c107630 0x7f137c1988f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f13838a8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f137c1125f0 con 0x7f137c102f00 2024-03-19T01:11:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f13838a8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f137c112470 con 0x7f137c102550 2024-03-19T01:11:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f13838a8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f137c112770 con 0x7f137c107630 2024-03-19T01:11:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f1380e43700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f137c102f00 0x7f137c19ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f1380e43700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f137c102f00 0x7f137c19ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45744/0 (socket says 172.21.15.116:45744) 2024-03-19T01:11:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f1380e43700 1 -- 172.21.15.116:0/216272039 learned_addr learned my addr 172.21.15.116:0/216272039 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f1381644700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f137c102550 0x7f137c19e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f1381e45700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f137c107630 0x7f137c1988f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f1380e43700 1 -- 172.21.15.116:0/216272039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f137c107630 msgr2=0x7f137c1988f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f1380e43700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f137c107630 0x7f137c1988f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f1380e43700 1 -- 172.21.15.116:0/216272039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f137c102550 msgr2=0x7f137c19e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f1380e43700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f137c102550 0x7f137c19e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.674+0000 7f1380e43700 1 -- 172.21.15.116:0/216272039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137c199050 con 0x7f137c102f00 2024-03-19T01:11:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.675+0000 7f1380e43700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f137c102f00 0x7f137c19ec70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f136c00f690 tx=0x7f136c005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.675+0000 7f13727fc700 1 -- 172.21.15.116:0/216272039 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c027070 con 0x7f137c102f00 2024-03-19T01:11:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.675+0000 7f13838a8700 1 -- 172.21.15.116:0/216272039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f137c1992e0 con 0x7f137c102f00 2024-03-19T01:11:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.675+0000 7f13727fc700 1 -- 172.21.15.116:0/216272039 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f136c003680 con 0x7f137c102f00 2024-03-19T01:11:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.675+0000 7f13727fc700 1 -- 172.21.15.116:0/216272039 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136c022e50 con 0x7f137c102f00 2024-03-19T01:11:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.675+0000 7f13838a8700 1 -- 172.21.15.116:0/216272039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f137c1a5600 con 0x7f137c102f00 2024-03-19T01:11:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.676+0000 7f13838a8700 1 -- 172.21.15.116:0/216272039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f137c062aa0 con 0x7f137c102f00 2024-03-19T01:11:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.677+0000 7f13727fc700 1 -- 172.21.15.116:0/216272039 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f136c02a070 con 0x7f137c102f00 2024-03-19T01:11:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.677+0000 7f13727fc700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f13680a2ac0 0x7f13680a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.677+0000 7f1381644700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f13680a2ac0 0x7f13680a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.678+0000 7f13727fc700 1 -- 172.21.15.116:0/216272039 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f136c0cf770 con 0x7f137c102f00 2024-03-19T01:11:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.678+0000 7f13727fc700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340] conn(0x7f13680ac2a0 0x7f13680ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.678+0000 7f13727fc700 1 -- 172.21.15.116:0/216272039 --> [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f136c001d30 con 0x7f13680ac2a0 2024-03-19T01:11:13.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.678+0000 7f13727fc700 1 -- 172.21.15.116:0/216272039 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f136c0cfc50 con 0x7f137c102f00 2024-03-19T01:11:13.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.678+0000 7f1381e45700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340] conn(0x7f13680ac2a0 0x7f13680ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:13.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.678+0000 7f1381644700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f13680a2ac0 0x7f13680a4f80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f137c199a30 tx=0x7f1378009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:13.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.678+0000 7f1381e45700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340] conn(0x7f13680ac2a0 0x7f13680ae6c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:13.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.679+0000 7f13727fc700 1 -- 172.21.15.116:0/216272039 <== osd.8 v2:172.21.15.162:6816/1669615340 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f136c001d30 con 0x7f13680ac2a0 2024-03-19T01:11:13.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.695+0000 7f13838a8700 1 -- 172.21.15.116:0/216272039 --> [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f137c062f20 con 0x7f13680ac2a0 2024-03-19T01:11:13.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.695+0000 7f13727fc700 1 -- 172.21.15.116:0/216272039 <== osd.8 v2:172.21.15.162:6816/1669615340 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f137c062f20 con 0x7f13680ac2a0 2024-03-19T01:11:13.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.696+0000 7f13838a8700 1 -- 172.21.15.116:0/216272039 >> [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340] conn(0x7f13680ac2a0 msgr2=0x7f13680ae6c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.696+0000 7f13838a8700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340] conn(0x7f13680ac2a0 0x7f13680ae6c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:13.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.696+0000 7f13838a8700 1 -- 172.21.15.116:0/216272039 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f13680a2ac0 msgr2=0x7f13680a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.696+0000 7f13838a8700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f13680a2ac0 0x7f13680a4f80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f137c199a30 tx=0x7f1378009260 comp rx=0 tx=0).stop 2024-03-19T01:11:13.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.696+0000 7f13838a8700 1 -- 172.21.15.116:0/216272039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f137c102f00 msgr2=0x7f137c19ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:13.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.696+0000 7f13838a8700 1 --2- 172.21.15.116:0/216272039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f137c102f00 0x7f137c19ec70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f136c00f690 tx=0x7f136c005f10 comp rx=0 tx=0).stop 2024-03-19T01:11:13.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.696+0000 7f1381e45700 1 -- 172.21.15.116:0/216272039 reap_dead start 2024-03-19T01:11:13.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.696+0000 7f13838a8700 1 -- 172.21.15.116:0/216272039 shutdown_connections 2024-03-19T01:11:13.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.696+0000 7f13838a8700 1 -- 172.21.15.116:0/216272039 >> 172.21.15.116:0/216272039 conn(0x7f137c0fddc0 msgr2=0x7f137c1000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:13.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.696+0000 7f13838a8700 1 -- 172.21.15.116:0/216272039 shutdown_connections 2024-03-19T01:11:13.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:13.696+0000 7f13838a8700 1 -- 172.21.15.116:0/216272039 wait complete. 2024-03-19T01:11:14.130 INFO:teuthology.orchestra.run.smithi116.stdout:141733920825 2024-03-19T01:11:14.131 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-19T01:11:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:14 smithi162 ceph-mon[67178]: pgmap v308: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-19T01:11:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:14 smithi116 ceph-mon[62701]: pgmap v308: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-19T01:11:14.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:14 smithi156 ceph-mon[66799]: pgmap v308: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-19T01:11:14.866 INFO:teuthology.orchestra.run.smithi116.stdout:163208757300 2024-03-19T01:11:14.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-19T01:11:15.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.275+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/3324374283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bb406aa90 msgr2=0x7f3bb406af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:15.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.275+0000 7f3bbb4ec700 1 --2- 172.21.15.116:0/3324374283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bb406aa90 0x7f3bb406af10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009990 tx=0x7f3ba4009ca0 comp rx=0 tx=0).stop 2024-03-19T01:11:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.275+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/3324374283 shutdown_connections 2024-03-19T01:11:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.275+0000 7f3bbb4ec700 1 --2- 172.21.15.116:0/3324374283 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bb406b450 0x7f3bb410f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.275+0000 7f3bbb4ec700 1 --2- 172.21.15.116:0/3324374283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bb4107320 0x7f3bb4107700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.275+0000 7f3bbb4ec700 1 --2- 172.21.15.116:0/3324374283 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bb406aa90 0x7f3bb406af10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.276+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/3324374283 >> 172.21.15.116:0/3324374283 conn(0x7f3bb4078270 msgr2=0x7f3bb4078680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.276+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/3324374283 shutdown_connections 2024-03-19T01:11:15.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.276+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/3324374283 wait complete. 2024-03-19T01:11:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.276+0000 7f3bbb4ec700 1 Processor -- start 2024-03-19T01:11:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.276+0000 7f3bbb4ec700 1 -- start start 2024-03-19T01:11:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bbb4ec700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bb406aa90 0x7f3bb419e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bbb4ec700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bb406b450 0x7f3bb419ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bbb4ec700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bb4107320 0x7f3bb41989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bbb4ec700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb4112770 con 0x7f3bb406aa90 2024-03-19T01:11:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bbb4ec700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb41125f0 con 0x7f3bb406b450 2024-03-19T01:11:15.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bbb4ec700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb41128f0 con 0x7f3bb4107320 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bb9288700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bb406aa90 0x7f3bb419e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bb9288700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bb406aa90 0x7f3bb419e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45760/0 (socket says 172.21.15.116:45760) 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bb8a87700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bb406b450 0x7f3bb419ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bb9288700 1 -- 172.21.15.116:0/1485815874 learned_addr learned my addr 172.21.15.116:0/1485815874 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bb9a89700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bb4107320 0x7f3bb41989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bb8a87700 1 -- 172.21.15.116:0/1485815874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bb4107320 msgr2=0x7f3bb41989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bb8a87700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3bb4107320 0x7f3bb41989f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bb8a87700 1 -- 172.21.15.116:0/1485815874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bb406aa90 msgr2=0x7f3bb419e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bb8a87700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bb406aa90 0x7f3bb419e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bb8a87700 1 -- 172.21.15.116:0/1485815874 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb4199150 con 0x7f3bb406b450 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bb9288700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bb406aa90 0x7f3bb419e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3bb8a87700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bb406b450 0x7f3bb419ed70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4006010 tx=0x7f3ba400bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.277+0000 7f3baa7fc700 1 -- 172.21.15.116:0/1485815874 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4027070 con 0x7f3bb406b450 2024-03-19T01:11:15.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.278+0000 7f3baa7fc700 1 -- 172.21.15.116:0/1485815874 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ba4003d70 con 0x7f3bb406b450 2024-03-19T01:11:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.278+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/1485815874 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bb41993e0 con 0x7f3bb406b450 2024-03-19T01:11:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.278+0000 7f3baa7fc700 1 -- 172.21.15.116:0/1485815874 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba400f460 con 0x7f3bb406b450 2024-03-19T01:11:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.278+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/1485815874 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bb41a5700 con 0x7f3bb406b450 2024-03-19T01:11:15.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.278+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/1485815874 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3bb4062a20 con 0x7f3bb406b450 2024-03-19T01:11:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.280+0000 7f3baa7fc700 1 -- 172.21.15.116:0/1485815874 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f3ba402a070 con 0x7f3bb406b450 2024-03-19T01:11:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.280+0000 7f3baa7fc700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3ba00a2920 0x7f3ba00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.280+0000 7f3bb9288700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3ba00a2920 0x7f3ba00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.280+0000 7f3baa7fc700 1 -- 172.21.15.116:0/1485815874 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f3ba40d0520 con 0x7f3bb406b450 2024-03-19T01:11:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.280+0000 7f3baa7fc700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908] conn(0x7f3ba00ac100 0x7f3ba00ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.280+0000 7f3baa7fc700 1 -- 172.21.15.116:0/1485815874 --> [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3ba4009030 con 0x7f3ba00ac100 2024-03-19T01:11:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.280+0000 7f3baa7fc700 1 -- 172.21.15.116:0/1485815874 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f3ba40d0a00 con 0x7f3bb406b450 2024-03-19T01:11:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.281+0000 7f3bb9a89700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908] conn(0x7f3ba00ac100 0x7f3ba00ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:15.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.281+0000 7f3bb9288700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3ba00a2920 0x7f3ba00a4de0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4199b30 tx=0x7f3bb0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.281+0000 7f3bb9a89700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908] conn(0x7f3ba00ac100 0x7f3ba00ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:15.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.282+0000 7f3baa7fc700 1 -- 172.21.15.116:0/1485815874 <== osd.1 v2:172.21.15.156:6800/2727091908 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3ba4009030 con 0x7f3ba00ac100 2024-03-19T01:11:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.298+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/1485815874 --> [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3bb4062e20 con 0x7f3ba00ac100 2024-03-19T01:11:15.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.299+0000 7f3baa7fc700 1 -- 172.21.15.116:0/1485815874 <== osd.1 v2:172.21.15.156:6800/2727091908 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3bb4062e20 con 0x7f3ba00ac100 2024-03-19T01:11:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.299+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/1485815874 >> [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908] conn(0x7f3ba00ac100 msgr2=0x7f3ba00ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.299+0000 7f3bbb4ec700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908] conn(0x7f3ba00ac100 0x7f3ba00ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.299+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/1485815874 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3ba00a2920 msgr2=0x7f3ba00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.299+0000 7f3bbb4ec700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3ba00a2920 0x7f3ba00a4de0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4199b30 tx=0x7f3bb0009260 comp rx=0 tx=0).stop 2024-03-19T01:11:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.299+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/1485815874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bb406b450 msgr2=0x7f3bb419ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:15.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.299+0000 7f3bbb4ec700 1 --2- 172.21.15.116:0/1485815874 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3bb406b450 0x7f3bb419ed70 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4006010 tx=0x7f3ba400bab0 comp rx=0 tx=0).stop 2024-03-19T01:11:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.299+0000 7f3bb9a89700 1 -- 172.21.15.116:0/1485815874 reap_dead start 2024-03-19T01:11:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.299+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/1485815874 shutdown_connections 2024-03-19T01:11:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.299+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/1485815874 >> 172.21.15.116:0/1485815874 conn(0x7f3bb4078270 msgr2=0x7f3bb40ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.299+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/1485815874 shutdown_connections 2024-03-19T01:11:15.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:15.299+0000 7f3bbb4ec700 1 -- 172.21.15.116:0/1485815874 wait complete. 2024-03-19T01:11:15.841 INFO:teuthology.orchestra.run.smithi116.stdout:309237645339 2024-03-19T01:11:15.841 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-19T01:11:16.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:16 smithi162 ceph-mon[67178]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-19T01:11:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:16 smithi116 ceph-mon[62701]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-19T01:11:16.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:16 smithi156 ceph-mon[66799]: pgmap v309: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 170 B/s wr, 11 op/s 2024-03-19T01:11:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.964+0000 7f6e99d27700 1 -- 172.21.15.116:0/744712422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e94101460 msgr2=0x7f6e94101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.964+0000 7f6e99d27700 1 --2- 172.21.15.116:0/744712422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e94101460 0x7f6e94101840 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6e84009990 tx=0x7f6e84009ca0 comp rx=0 tx=0).stop 2024-03-19T01:11:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.964+0000 7f6e99d27700 1 -- 172.21.15.116:0/744712422 shutdown_connections 2024-03-19T01:11:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.964+0000 7f6e99d27700 1 --2- 172.21.15.116:0/744712422 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e94106520 0x7f6e9410e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.964+0000 7f6e99d27700 1 --2- 172.21.15.116:0/744712422 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e94101460 0x7f6e94101840 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.964+0000 7f6e99d27700 1 --2- 172.21.15.116:0/744712422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e94101e10 0x7f6e94105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.964+0000 7f6e99d27700 1 -- 172.21.15.116:0/744712422 >> 172.21.15.116:0/744712422 conn(0x7f6e94078180 msgr2=0x7f6e94078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.964+0000 7f6e99d27700 1 -- 172.21.15.116:0/744712422 shutdown_connections 2024-03-19T01:11:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.964+0000 7f6e99d27700 1 -- 172.21.15.116:0/744712422 wait complete. 2024-03-19T01:11:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.965+0000 7f6e99d27700 1 Processor -- start 2024-03-19T01:11:17.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.965+0000 7f6e99d27700 1 -- start start 2024-03-19T01:11:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e99d27700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e94101460 0x7f6e941998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e99d27700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e94101e10 0x7f6e94199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e99d27700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e94106520 0x7f6e941a4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e99d27700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6e941114e0 con 0x7f6e94101460 2024-03-19T01:11:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e99d27700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6e94111360 con 0x7f6e94106520 2024-03-19T01:11:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e99d27700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6e94111660 con 0x7f6e94101e10 2024-03-19T01:11:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e937fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e94101460 0x7f6e941998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e937fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e94101460 0x7f6e941998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45634/0 (socket says 172.21.15.116:45634) 2024-03-19T01:11:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e937fe700 1 -- 172.21.15.116:0/2124890069 learned_addr learned my addr 172.21.15.116:0/2124890069 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e92ffd700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e94101e10 0x7f6e94199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e93fff700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e94106520 0x7f6e941a4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e92ffd700 1 -- 172.21.15.116:0/2124890069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e94106520 msgr2=0x7f6e941a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:17.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e92ffd700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e94106520 0x7f6e941a4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e92ffd700 1 -- 172.21.15.116:0/2124890069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e94101460 msgr2=0x7f6e941998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e92ffd700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e94101460 0x7f6e941998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e92ffd700 1 -- 172.21.15.116:0/2124890069 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e941a4ea0 con 0x7f6e94101e10 2024-03-19T01:11:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e937fe700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e94101460 0x7f6e941998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e92ffd700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e94101e10 0x7f6e94199e20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800e9c0 tx=0x7f6e8800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e90ff9700 1 -- 172.21.15.116:0/2124890069 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e8800c9c0 con 0x7f6e94101e10 2024-03-19T01:11:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e90ff9700 1 -- 172.21.15.116:0/2124890069 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e8800cb60 con 0x7f6e94101e10 2024-03-19T01:11:17.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.966+0000 7f6e99d27700 1 -- 172.21.15.116:0/2124890069 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e941a50d0 con 0x7f6e94101e10 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.967+0000 7f6e90ff9700 1 -- 172.21.15.116:0/2124890069 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88018680 con 0x7f6e94101e10 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.967+0000 7f6e99d27700 1 -- 172.21.15.116:0/2124890069 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e941a5610 con 0x7f6e94101e10 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.967+0000 7f6e99d27700 1 -- 172.21.15.116:0/2124890069 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6e94062a20 con 0x7f6e94101e10 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.969+0000 7f6e90ff9700 1 -- 172.21.15.116:0/2124890069 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f6e88047a90 con 0x7f6e94101e10 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.969+0000 7f6e90ff9700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6e7c0a2920 0x7f6e7c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.969+0000 7f6e937fe700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6e7c0a2920 0x7f6e7c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.969+0000 7f6e90ff9700 1 -- 172.21.15.116:0/2124890069 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f6e880cebc0 con 0x7f6e94101e10 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.969+0000 7f6e90ff9700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116] conn(0x7f6e7c0ac100 0x7f6e7c0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.969+0000 7f6e90ff9700 1 -- 172.21.15.116:0/2124890069 --> [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6e88023e50 con 0x7f6e7c0ac100 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.969+0000 7f6e90ff9700 1 -- 172.21.15.116:0/2124890069 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f6e880cef70 con 0x7f6e94101e10 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.969+0000 7f6e93fff700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116] conn(0x7f6e7c0ac100 0x7f6e7c0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.969+0000 7f6e937fe700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6e7c0a2920 0x7f6e7c0a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6e8400b2e0 tx=0x7f6e84009590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.970+0000 7f6e93fff700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116] conn(0x7f6e7c0ac100 0x7f6e7c0ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:17.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.970+0000 7f6e90ff9700 1 -- 172.21.15.116:0/2124890069 <== osd.9 v2:172.21.15.116:6826/96304116 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6e88023e50 con 0x7f6e7c0ac100 2024-03-19T01:11:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.996+0000 7f6e99d27700 1 -- 172.21.15.116:0/2124890069 --> [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6e94062ea0 con 0x7f6e7c0ac100 2024-03-19T01:11:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.996+0000 7f6e90ff9700 1 -- 172.21.15.116:0/2124890069 <== osd.9 v2:172.21.15.116:6826/96304116 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6e94062ea0 con 0x7f6e7c0ac100 2024-03-19T01:11:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.997+0000 7f6e99d27700 1 -- 172.21.15.116:0/2124890069 >> [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116] conn(0x7f6e7c0ac100 msgr2=0x7f6e7c0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.997+0000 7f6e99d27700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116] conn(0x7f6e7c0ac100 0x7f6e7c0ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.997+0000 7f6e99d27700 1 -- 172.21.15.116:0/2124890069 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6e7c0a2920 msgr2=0x7f6e7c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.997+0000 7f6e99d27700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6e7c0a2920 0x7f6e7c0a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6e8400b2e0 tx=0x7f6e84009590 comp rx=0 tx=0).stop 2024-03-19T01:11:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.997+0000 7f6e99d27700 1 -- 172.21.15.116:0/2124890069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e94101e10 msgr2=0x7f6e94199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.997+0000 7f6e99d27700 1 --2- 172.21.15.116:0/2124890069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e94101e10 0x7f6e94199e20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800e9c0 tx=0x7f6e8800ecd0 comp rx=0 tx=0).stop 2024-03-19T01:11:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.997+0000 7f6e93fff700 1 -- 172.21.15.116:0/2124890069 reap_dead start 2024-03-19T01:11:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.997+0000 7f6e99d27700 1 -- 172.21.15.116:0/2124890069 shutdown_connections 2024-03-19T01:11:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.997+0000 7f6e99d27700 1 -- 172.21.15.116:0/2124890069 >> 172.21.15.116:0/2124890069 conn(0x7f6e94078180 msgr2=0x7f6e940ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.997+0000 7f6e99d27700 1 -- 172.21.15.116:0/2124890069 shutdown_connections 2024-03-19T01:11:17.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:16.997+0000 7f6e99d27700 1 -- 172.21.15.116:0/2124890069 wait complete. 2024-03-19T01:11:18.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:18 smithi162 ceph-mon[67178]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-19T01:11:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:18 smithi116 ceph-mon[62701]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-19T01:11:18.455 INFO:teuthology.orchestra.run.smithi116.stdout:55834574920 2024-03-19T01:11:18.456 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-19T01:11:18.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:18 smithi156 ceph-mon[66799]: pgmap v310: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-03-19T01:11:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.691+0000 7f91f7cc9700 1 -- 172.21.15.116:0/3600853302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91f0101210 msgr2=0x7f91f0101670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.691+0000 7f91f7cc9700 1 --2- 172.21.15.116:0/3600853302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91f0101210 0x7f91f0101670 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f91e00098f0 tx=0x7f91e0009c00 comp rx=0 tx=0).stop 2024-03-19T01:11:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.692+0000 7f91f7cc9700 1 -- 172.21.15.116:0/3600853302 shutdown_connections 2024-03-19T01:11:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.692+0000 7f91f7cc9700 1 --2- 172.21.15.116:0/3600853302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91f0101bb0 0x7f91f010e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.692+0000 7f91f7cc9700 1 --2- 172.21.15.116:0/3600853302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91f0107250 0x7f91f0107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.692+0000 7f91f7cc9700 1 --2- 172.21.15.116:0/3600853302 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91f0101210 0x7f91f0101670 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.692+0000 7f91f7cc9700 1 -- 172.21.15.116:0/3600853302 >> 172.21.15.116:0/3600853302 conn(0x7f91f0078140 msgr2=0x7f91f0078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.692+0000 7f91f7cc9700 1 -- 172.21.15.116:0/3600853302 shutdown_connections 2024-03-19T01:11:18.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.692+0000 7f91f7cc9700 1 -- 172.21.15.116:0/3600853302 wait complete. 2024-03-19T01:11:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.693+0000 7f91f7cc9700 1 Processor -- start 2024-03-19T01:11:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.693+0000 7f91f7cc9700 1 -- start start 2024-03-19T01:11:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.693+0000 7f91f7cc9700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91f0101210 0x7f91f01a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.693+0000 7f91f7cc9700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91f0101bb0 0x7f91f01a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.693+0000 7f91f7cc9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91f0107250 0x7f91f019f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.693+0000 7f91f5a65700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91f0101210 0x7f91f01a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.693+0000 7f91f5264700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91f0101bb0 0x7f91f01a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.693+0000 7f91f5264700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91f0101bb0 0x7f91f01a52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45666/0 (socket says 172.21.15.116:45666) 2024-03-19T01:11:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.693+0000 7f91f5a65700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91f0101210 0x7f91f01a4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37114/0 (socket says 172.21.15.116:37114) 2024-03-19T01:11:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.693+0000 7f91f5264700 1 -- 172.21.15.116:0/1935156252 learned_addr learned my addr 172.21.15.116:0/1935156252 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.694+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f91f0111500 con 0x7f91f0101bb0 2024-03-19T01:11:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.694+0000 7f91f6266700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91f0107250 0x7f91f019f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.694+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f91f0111380 con 0x7f91f0107250 2024-03-19T01:11:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.694+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f91f0111680 con 0x7f91f0101210 2024-03-19T01:11:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.694+0000 7f91f5264700 1 -- 172.21.15.116:0/1935156252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91f0101210 msgr2=0x7f91f01a4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.694+0000 7f91f5264700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91f0101210 0x7f91f01a4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.694+0000 7f91f5264700 1 -- 172.21.15.116:0/1935156252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91f0107250 msgr2=0x7f91f019f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.694+0000 7f91f5264700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91f0107250 0x7f91f019f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.694+0000 7f91f5264700 1 -- 172.21.15.116:0/1935156252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f019f9b0 con 0x7f91f0101bb0 2024-03-19T01:11:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.695+0000 7f91f5264700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91f0101bb0 0x7f91f01a52e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f91e0005950 tx=0x7f91e000fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.695+0000 7f91f5a65700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f91f0101210 0x7f91f01a4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:11:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.695+0000 7f91e6ffd700 1 -- 172.21.15.116:0/1935156252 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e0005140 con 0x7f91f0101bb0 2024-03-19T01:11:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.695+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f019fc40 con 0x7f91f0101bb0 2024-03-19T01:11:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.695+0000 7f91e6ffd700 1 -- 172.21.15.116:0/1935156252 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f91e000b560 con 0x7f91f0101bb0 2024-03-19T01:11:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.695+0000 7f91e6ffd700 1 -- 172.21.15.116:0/1935156252 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e0003aa0 con 0x7f91f0101bb0 2024-03-19T01:11:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.696+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f01abc00 con 0x7f91f0101bb0 2024-03-19T01:11:18.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.697+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f91f01054d0 con 0x7f91f0101bb0 2024-03-19T01:11:18.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.698+0000 7f91e6ffd700 1 -- 172.21.15.116:0/1935156252 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f91e0003c40 con 0x7f91f0101bb0 2024-03-19T01:11:18.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.699+0000 7f91e6ffd700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f91dc0a2920 0x7f91dc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:18.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.699+0000 7f91e6ffd700 1 -- 172.21.15.116:0/1935156252 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f91e00cfc80 con 0x7f91f0101bb0 2024-03-19T01:11:18.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.699+0000 7f91e6ffd700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514] conn(0x7f91dc0ac100 0x7f91dc0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:18.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.699+0000 7f91e6ffd700 1 -- 172.21.15.116:0/1935156252 --> [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91dc0aebd0 con 0x7f91dc0ac100 2024-03-19T01:11:18.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.699+0000 7f91e6ffd700 1 -- 172.21.15.116:0/1935156252 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7f91e00d6020 con 0x7f91f0101bb0 2024-03-19T01:11:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.699+0000 7f91f6266700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514] conn(0x7f91dc0ac100 0x7f91dc0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.699+0000 7f91f5a65700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f91dc0a2920 0x7f91dc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.700+0000 7f91f6266700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514] conn(0x7f91dc0ac100 0x7f91dc0ae520 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.700+0000 7f91f5a65700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f91dc0a2920 0x7f91dc0a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009d40 tx=0x7f91ec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.700+0000 7f91e6ffd700 1 -- 172.21.15.116:0/1935156252 <== osd.7 v2:172.21.15.156:6816/3540949514 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f91dc0aebd0 con 0x7f91dc0ac100 2024-03-19T01:11:18.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.721+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 --> [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f91f0102900 con 0x7f91dc0ac100 2024-03-19T01:11:18.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.721+0000 7f91e6ffd700 1 -- 172.21.15.116:0/1935156252 <== osd.7 v2:172.21.15.156:6816/3540949514 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f91f0102900 con 0x7f91dc0ac100 2024-03-19T01:11:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.721+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 >> [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514] conn(0x7f91dc0ac100 msgr2=0x7f91dc0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.721+0000 7f91f7cc9700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514] conn(0x7f91dc0ac100 0x7f91dc0ae520 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.721+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f91dc0a2920 msgr2=0x7f91dc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.721+0000 7f91f7cc9700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f91dc0a2920 0x7f91dc0a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f91ec009d40 tx=0x7f91ec009260 comp rx=0 tx=0).stop 2024-03-19T01:11:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.721+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91f0101bb0 msgr2=0x7f91f01a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.721+0000 7f91f7cc9700 1 --2- 172.21.15.116:0/1935156252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91f0101bb0 0x7f91f01a52e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f91e0005950 tx=0x7f91e000fb20 comp rx=0 tx=0).stop 2024-03-19T01:11:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.721+0000 7f91f6266700 1 -- 172.21.15.116:0/1935156252 reap_dead start 2024-03-19T01:11:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.721+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 shutdown_connections 2024-03-19T01:11:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.721+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 >> 172.21.15.116:0/1935156252 conn(0x7f91f0078140 msgr2=0x7f91f006a1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:18.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.722+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 shutdown_connections 2024-03-19T01:11:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.722+0000 7f91f7cc9700 1 -- 172.21.15.116:0/1935156252 wait complete. 2024-03-19T01:11:19.142 INFO:teuthology.orchestra.run.smithi116.stdout:253403070501 2024-03-19T01:11:19.142 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-19T01:11:19.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.124+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/4072169319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb44101c70 msgr2=0x7fcb44102050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.124+0000 7fcb4c5c3700 1 --2- 172.21.15.116:0/4072169319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb44101c70 0x7fcb44102050 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcb400099c0 tx=0x7fcb40009cd0 comp rx=0 tx=0).stop 2024-03-19T01:11:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.124+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/4072169319 shutdown_connections 2024-03-19T01:11:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.124+0000 7fcb4c5c3700 1 --2- 172.21.15.116:0/4072169319 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb44106590 0x7fcb4410e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.124+0000 7fcb4c5c3700 1 --2- 172.21.15.116:0/4072169319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb44101c70 0x7fcb44102050 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.124+0000 7fcb4c5c3700 1 --2- 172.21.15.116:0/4072169319 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb44102620 0x7fcb44105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.124+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/4072169319 >> 172.21.15.116:0/4072169319 conn(0x7fcb44077810 msgr2=0x7fcb440fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.124+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/4072169319 shutdown_connections 2024-03-19T01:11:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.124+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/4072169319 wait complete. 2024-03-19T01:11:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4c5c3700 1 Processor -- start 2024-03-19T01:11:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4c5c3700 1 -- start start 2024-03-19T01:11:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4c5c3700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb44101c70 0x7fcb44199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4c5c3700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb44102620 0x7fcb44199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4c5c3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb44106590 0x7fcb441a49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:19.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4c5c3700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcb44111570 con 0x7fcb44101c70 2024-03-19T01:11:19.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4c5c3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcb441113f0 con 0x7fcb44106590 2024-03-19T01:11:19.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4c5c3700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fcb441116f0 con 0x7fcb44102620 2024-03-19T01:11:19.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4a35f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb44101c70 0x7fcb44199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:19.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4a35f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb44101c70 0x7fcb44199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45658/0 (socket says 172.21.15.116:45658) 2024-03-19T01:11:19.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4a35f700 1 -- 172.21.15.116:0/56879950 learned_addr learned my addr 172.21.15.116:0/56879950 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:19.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb49b5e700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb44102620 0x7fcb44199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:19.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4ab60700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb44106590 0x7fcb441a49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4a35f700 1 -- 172.21.15.116:0/56879950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb44102620 msgr2=0x7fcb44199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4a35f700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fcb44102620 0x7fcb44199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4a35f700 1 -- 172.21.15.116:0/56879950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb44106590 msgr2=0x7fcb441a49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4a35f700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb44106590 0x7fcb441a49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.125+0000 7fcb4a35f700 1 -- 172.21.15.116:0/56879950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb441a4f30 con 0x7fcb44101c70 2024-03-19T01:11:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.126+0000 7fcb4a35f700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb44101c70 0x7fcb44199970 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcb40006010 tx=0x7fcb4000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.130+0000 7fcb3b7fe700 1 -- 172.21.15.116:0/56879950 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb40003680 con 0x7fcb44101c70 2024-03-19T01:11:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.130+0000 7fcb3b7fe700 1 -- 172.21.15.116:0/56879950 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb400050d0 con 0x7fcb44101c70 2024-03-19T01:11:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.130+0000 7fcb3b7fe700 1 -- 172.21.15.116:0/56879950 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb40003db0 con 0x7fcb44101c70 2024-03-19T01:11:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.130+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/56879950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb441a5100 con 0x7fcb44101c70 2024-03-19T01:11:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.130+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/56879950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb441a5550 con 0x7fcb44101c70 2024-03-19T01:11:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.130+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/56879950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcb44062a20 con 0x7fcb44101c70 2024-03-19T01:11:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.132+0000 7fcb3b7fe700 1 -- 172.21.15.116:0/56879950 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fcb40005280 con 0x7fcb44101c70 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.132+0000 7fcb3b7fe700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fcb300a2b40 0x7fcb300a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.132+0000 7fcb49b5e700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fcb300a2b40 0x7fcb300a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.133+0000 7fcb3b7fe700 1 -- 172.21.15.116:0/56879950 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7fcb400cfa40 con 0x7fcb44101c70 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.133+0000 7fcb3b7fe700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758] conn(0x7fcb300ac320 0x7fcb300ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.133+0000 7fcb3b7fe700 1 -- 172.21.15.116:0/56879950 --> [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb4000fe50 con 0x7fcb300ac320 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.133+0000 7fcb3b7fe700 1 -- 172.21.15.116:0/56879950 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) ==== 24+0+0 (secure 0 0 0) 0x7fcb400cff20 con 0x7fcb44101c70 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.133+0000 7fcb49b5e700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fcb300a2b40 0x7fcb300a5000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcb440fff50 tx=0x7fcb34009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.133+0000 7fcb4ab60700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758] conn(0x7fcb300ac320 0x7fcb300ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.133+0000 7fcb4ab60700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758] conn(0x7fcb300ac320 0x7fcb300ae740 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.134+0000 7fcb3b7fe700 1 -- 172.21.15.116:0/56879950 <== osd.2 v2:172.21.15.162:6800/2120908758 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcb4000fe50 con 0x7fcb300ac320 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.157+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/56879950 --> [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcb44062e20 con 0x7fcb300ac320 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.157+0000 7fcb3b7fe700 1 -- 172.21.15.116:0/56879950 <== osd.2 v2:172.21.15.162:6800/2120908758 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcb44062e20 con 0x7fcb300ac320 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.157+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/56879950 >> [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758] conn(0x7fcb300ac320 msgr2=0x7fcb300ae740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.158+0000 7fcb4c5c3700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758] conn(0x7fcb300ac320 0x7fcb300ae740 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.158+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/56879950 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fcb300a2b40 msgr2=0x7fcb300a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.158+0000 7fcb4c5c3700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fcb300a2b40 0x7fcb300a5000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcb440fff50 tx=0x7fcb34009260 comp rx=0 tx=0).stop 2024-03-19T01:11:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.158+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/56879950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb44101c70 msgr2=0x7fcb44199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.158+0000 7fcb4c5c3700 1 --2- 172.21.15.116:0/56879950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb44101c70 0x7fcb44199970 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcb40006010 tx=0x7fcb4000f690 comp rx=0 tx=0).stop 2024-03-19T01:11:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.158+0000 7fcb4ab60700 1 -- 172.21.15.116:0/56879950 reap_dead start 2024-03-19T01:11:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.158+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/56879950 shutdown_connections 2024-03-19T01:11:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.158+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/56879950 >> 172.21.15.116:0/56879950 conn(0x7fcb44077810 msgr2=0x7fcb440fcf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.158+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/56879950 shutdown_connections 2024-03-19T01:11:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:18.158+0000 7fcb4c5c3700 1 -- 172.21.15.116:0/56879950 wait complete. 2024-03-19T01:11:19.855 INFO:teuthology.orchestra.run.smithi116.stdout:77309411396 2024-03-19T01:11:19.856 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-19T01:11:20.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:20 smithi162 ceph-mon[67178]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-19T01:11:20.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:20 smithi116 ceph-mon[62701]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-19T01:11:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:20 smithi156 ceph-mon[66799]: pgmap v311: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-19T01:11:20.550 INFO:teuthology.orchestra.run.smithi116.stdout:219043332139 2024-03-19T01:11:20.550 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-19T01:11:20.613 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:20.617 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:20.622 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:21.007 INFO:teuthology.orchestra.run.smithi116.stdout:287762808864 2024-03-19T01:11:21.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-19T01:11:22.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:22 smithi162 ceph-mon[67178]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T01:11:22.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:22 smithi116 ceph-mon[62701]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T01:11:22.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:22 smithi156 ceph-mon[66799]: pgmap v312: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T01:11:24.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:24 smithi162 ceph-mon[67178]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:11:24.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:24 smithi116 ceph-mon[62701]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:11:24.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:24 smithi156 ceph-mon[66799]: pgmap v313: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:11:24.580 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:24.582 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:24.583 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:24.589 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:24.592 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:24.609 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:24.613 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:26.393 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:26 smithi116 ceph-mon[62701]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:11:26.398 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:26 smithi162 ceph-mon[67178]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:11:26.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:26 smithi156 ceph-mon[66799]: pgmap v314: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:11:27.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-19T01:11:27.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-03-19T01:11:27.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-03-19T01:11:27.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-19T01:11:27.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-19T01:11:27.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]: dispatch 2024-03-19T01:11:27.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-03-19T01:11:27.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-03-19T01:11:27.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-19T01:11:27.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:27 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-03-19T01:11:27.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-19T01:11:27.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-03-19T01:11:27.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-03-19T01:11:27.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-19T01:11:27.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-19T01:11:27.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]: dispatch 2024-03-19T01:11:27.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-03-19T01:11:27.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-03-19T01:11:27.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-19T01:11:27.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:27 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-03-19T01:11:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-19T01:11:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]: dispatch 2024-03-19T01:11:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-03-19T01:11:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-19T01:11:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-19T01:11:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]: dispatch 2024-03-19T01:11:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-03-19T01:11:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-03-19T01:11:27.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-19T01:11:27.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:27 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]: dispatch 2024-03-19T01:11:27.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.914+0000 7f1add993700 1 -- 172.21.15.116:0/1992464737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8108340 msgr2=0x7f1ad8108720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:27.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.914+0000 7f1add993700 1 --2- 172.21.15.116:0/1992464737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8108340 0x7f1ad8108720 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1ac80099c0 tx=0x7f1ac8009cd0 comp rx=0 tx=0).stop 2024-03-19T01:11:27.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.914+0000 7f1add993700 1 -- 172.21.15.116:0/1992464737 shutdown_connections 2024-03-19T01:11:27.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.914+0000 7f1add993700 1 --2- 172.21.15.116:0/1992464737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad8102d20 0x7f1ad810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:27.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.914+0000 7f1add993700 1 --2- 172.21.15.116:0/1992464737 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8108340 0x7f1ad8108720 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:27.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.914+0000 7f1add993700 1 --2- 172.21.15.116:0/1992464737 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ad8102360 0x7f1ad81027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:27.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.914+0000 7f1add993700 1 -- 172.21.15.116:0/1992464737 >> 172.21.15.116:0/1992464737 conn(0x7f1ad8079440 msgr2=0x7f1ad8079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.915+0000 7f1add993700 1 -- 172.21.15.116:0/1992464737 shutdown_connections 2024-03-19T01:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.915+0000 7f1add993700 1 -- 172.21.15.116:0/1992464737 wait complete. 2024-03-19T01:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.915+0000 7f1add993700 1 Processor -- start 2024-03-19T01:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.915+0000 7f1add993700 1 -- start start 2024-03-19T01:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1add993700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad8102360 0x7f1ad819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1add993700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ad8102d20 0x7f1ad819ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1add993700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8108340 0x7f1ad8198810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1add993700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad81125f0 con 0x7f1ad8108340 2024-03-19T01:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1add993700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad8112470 con 0x7f1ad8102360 2024-03-19T01:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1add993700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad8112770 con 0x7f1ad8102d20 2024-03-19T01:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1ad77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8108340 0x7f1ad8198810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:27.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1ad77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8108340 0x7f1ad8198810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36870/0 (socket says 172.21.15.116:36870) 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1ad6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad8102360 0x7f1ad819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1ad77fe700 1 -- 172.21.15.116:0/491875355 learned_addr learned my addr 172.21.15.116:0/491875355 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1ad67fc700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ad8102d20 0x7f1ad819ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1ad77fe700 1 -- 172.21.15.116:0/491875355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ad8102d20 msgr2=0x7f1ad819ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1ad77fe700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ad8102d20 0x7f1ad819ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1ad77fe700 1 -- 172.21.15.116:0/491875355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad8102360 msgr2=0x7f1ad819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1ad77fe700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad8102360 0x7f1ad819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1ad77fe700 1 -- 172.21.15.116:0/491875355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad8198ff0 con 0x7f1ad8108340 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1ad67fc700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ad8102d20 0x7f1ad819ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1ad6ffd700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad8102360 0x7f1ad819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1ad77fe700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8108340 0x7f1ad8198810 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000ef30 tx=0x7f1ad000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1adc991700 1 -- 172.21.15.116:0/491875355 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad00092e0 con 0x7f1ad8108340 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1add993700 1 -- 172.21.15.116:0/491875355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad8199280 con 0x7f1ad8108340 2024-03-19T01:11:27.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.916+0000 7f1adc991700 1 -- 172.21.15.116:0/491875355 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ad0004bb0 con 0x7f1ad8108340 2024-03-19T01:11:27.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.917+0000 7f1adc991700 1 -- 172.21.15.116:0/491875355 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0004280 con 0x7f1ad8108340 2024-03-19T01:11:27.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.917+0000 7f1add993700 1 -- 172.21.15.116:0/491875355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad81a54f0 con 0x7f1ad8108340 2024-03-19T01:11:27.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.918+0000 7f1add993700 1 -- 172.21.15.116:0/491875355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad81039f0 con 0x7f1ad8108340 2024-03-19T01:11:27.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.919+0000 7f1adc991700 1 -- 172.21.15.116:0/491875355 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f1ad001b030 con 0x7f1ad8108340 2024-03-19T01:11:27.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.919+0000 7f1adc991700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1ac00a2920 0x7f1ac00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:27.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.919+0000 7f1ad6ffd700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1ac00a2920 0x7f1ac00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:27.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.919+0000 7f1adc991700 1 -- 172.21.15.116:0/491875355 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 8269+0+0 (secure 0 0 0) 0x7f1ad00ce6e0 con 0x7f1ad8108340 2024-03-19T01:11:27.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.920+0000 7f1ad6ffd700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1ac00a2920 0x7f1ac00a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8000c00 tx=0x7f1ac8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:27.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:27.924+0000 7f1adc991700 1 -- 172.21.15.116:0/491875355 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1ad0022020 con 0x7f1ad8108340 2024-03-19T01:11:28.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.138+0000 7f1add993700 1 -- 172.21.15.116:0/491875355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f1ad802dbb0 con 0x7f1ad8108340 2024-03-19T01:11:28.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.138+0000 7f1adc991700 1 -- 172.21.15.116:0/491875355 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1ad0010040 con 0x7f1ad8108340 2024-03-19T01:11:28.139 INFO:teuthology.orchestra.run.smithi116.stdout:197568495666 2024-03-19T01:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.140+0000 7f1add993700 1 -- 172.21.15.116:0/491875355 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1ac00a2920 msgr2=0x7f1ac00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.140+0000 7f1add993700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1ac00a2920 0x7f1ac00a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8000c00 tx=0x7f1ac8005fb0 comp rx=0 tx=0).stop 2024-03-19T01:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.140+0000 7f1add993700 1 -- 172.21.15.116:0/491875355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8108340 msgr2=0x7f1ad8198810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.140+0000 7f1add993700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8108340 0x7f1ad8198810 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1ad000ef30 tx=0x7f1ad000c3a0 comp rx=0 tx=0).stop 2024-03-19T01:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.140+0000 7f1add993700 1 -- 172.21.15.116:0/491875355 shutdown_connections 2024-03-19T01:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.140+0000 7f1add993700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ad8108340 0x7f1ad8198810 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.140+0000 7f1add993700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad8102360 0x7f1ad819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.140+0000 7f1add993700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1ac00a2920 0x7f1ac00a4de0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.140+0000 7f1add993700 1 --2- 172.21.15.116:0/491875355 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ad8102d20 0x7f1ad819ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.140+0000 7f1add993700 1 -- 172.21.15.116:0/491875355 >> 172.21.15.116:0/491875355 conn(0x7f1ad8079440 msgr2=0x7f1ad8100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.140+0000 7f1add993700 1 -- 172.21.15.116:0/491875355 shutdown_connections 2024-03-19T01:11:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:28.140+0000 7f1add993700 1 -- 172.21.15.116:0/491875355 wait complete. 2024-03-19T01:11:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:28 smithi116 ceph-mon[62701]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:11:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-19T01:11:28.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-03-19T01:11:28.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-03-19T01:11:28.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-19T01:11:28.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-19T01:11:28.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]': finished 2024-03-19T01:11:28.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-03-19T01:11:28.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-03-19T01:11:28.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-19T01:11:28.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:28 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-03-19T01:11:28.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:28 smithi116 ceph-mon[62701]: osdmap e86: 12 total, 12 up, 12 in 2024-03-19T01:11:28.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:28 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/491875355' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-19T01:11:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:28 smithi156 ceph-mon[66799]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:11:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-19T01:11:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-03-19T01:11:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-03-19T01:11:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-19T01:11:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-19T01:11:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]': finished 2024-03-19T01:11:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-03-19T01:11:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-03-19T01:11:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-19T01:11:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:28 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-03-19T01:11:28.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:28 smithi156 ceph-mon[66799]: osdmap e86: 12 total, 12 up, 12 in 2024-03-19T01:11:28.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:28 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/491875355' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-19T01:11:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:28 smithi162 ceph-mon[67178]: pgmap v315: 97 pgs: 97 active+clean; 643 KiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:11:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-19T01:11:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 1]}]': finished 2024-03-19T01:11:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-03-19T01:11:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-19T01:11:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-19T01:11:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [5, 6]}]': finished 2024-03-19T01:11:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-03-19T01:11:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-03-19T01:11:28.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-19T01:11:28.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:28 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 7]}]': finished 2024-03-19T01:11:28.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:28 smithi162 ceph-mon[67178]: osdmap e86: 12 total, 12 up, 12 in 2024-03-19T01:11:28.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:28 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/491875355' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-19T01:11:29.217 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495661 got 197568495666 for osd.6 2024-03-19T01:11:29.218 DEBUG:teuthology.parallel:result is None 2024-03-19T01:11:29.425 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:29 smithi116 ceph-mon[62701]: osdmap e87: 12 total, 12 up, 12 in 2024-03-19T01:11:29.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:29 smithi156 ceph-mon[66799]: osdmap e87: 12 total, 12 up, 12 in 2024-03-19T01:11:29.648 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:29 smithi162 ceph-mon[67178]: osdmap e87: 12 total, 12 up, 12 in 2024-03-19T01:11:30.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.351+0000 7f0d32b2a700 1 -- 172.21.15.116:0/1308799092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d2c107250 msgr2=0x7f0d2c107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.351+0000 7f0d32b2a700 1 --2- 172.21.15.116:0/1308799092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d2c107250 0x7f0d2c107630 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0d20009990 tx=0x7f0d20009ca0 comp rx=0 tx=0).stop 2024-03-19T01:11:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 -- 172.21.15.116:0/1308799092 shutdown_connections 2024-03-19T01:11:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 --2- 172.21.15.116:0/1308799092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d2c101bb0 0x7f0d2c10e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 --2- 172.21.15.116:0/1308799092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d2c107250 0x7f0d2c107630 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 --2- 172.21.15.116:0/1308799092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d2c101210 0x7f0d2c101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 -- 172.21.15.116:0/1308799092 >> 172.21.15.116:0/1308799092 conn(0x7f0d2c078140 msgr2=0x7f0d2c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 -- 172.21.15.116:0/1308799092 shutdown_connections 2024-03-19T01:11:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 -- 172.21.15.116:0/1308799092 wait complete. 2024-03-19T01:11:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 Processor -- start 2024-03-19T01:11:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 -- start start 2024-03-19T01:11:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d2c101210 0x7f0d2c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d2c101bb0 0x7f0d2c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d2c107250 0x7f0d2c1988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2c1114e0 con 0x7f0d2c107250 2024-03-19T01:11:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2c111360 con 0x7f0d2c101210 2024-03-19T01:11:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.352+0000 7f0d32b2a700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2c111660 con 0x7f0d2c101bb0 2024-03-19T01:11:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d310c7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d2c107250 0x7f0d2c1988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d308c6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d2c101210 0x7f0d2c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d2bfff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d2c101bb0 0x7f0d2c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d308c6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d2c101210 0x7f0d2c19e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:46096/0 (socket says 172.21.15.116:46096) 2024-03-19T01:11:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d308c6700 1 -- 172.21.15.116:0/2192049140 learned_addr learned my addr 172.21.15.116:0/2192049140 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d310c7700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d2c107250 0x7f0d2c1988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36904/0 (socket says 172.21.15.116:36904) 2024-03-19T01:11:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d308c6700 1 -- 172.21.15.116:0/2192049140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d2c101bb0 msgr2=0x7f0d2c19ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d308c6700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d2c101bb0 0x7f0d2c19ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d308c6700 1 -- 172.21.15.116:0/2192049140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d2c107250 msgr2=0x7f0d2c1988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:30.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d308c6700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d2c107250 0x7f0d2c1988e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:30.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d308c6700 1 -- 172.21.15.116:0/2192049140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2c199040 con 0x7f0d2c101210 2024-03-19T01:11:30.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d2bfff700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d2c101bb0 0x7f0d2c19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:30.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d310c7700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d2c107250 0x7f0d2c1988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:30.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d308c6700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d2c101210 0x7f0d2c19e6d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0d20005910 tx=0x7f0d2000bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:30.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d29ffb700 1 -- 172.21.15.116:0/2192049140 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d20027070 con 0x7f0d2c101210 2024-03-19T01:11:30.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d32b2a700 1 -- 172.21.15.116:0/2192049140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d2c1992d0 con 0x7f0d2c101210 2024-03-19T01:11:30.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d29ffb700 1 -- 172.21.15.116:0/2192049140 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d20004650 con 0x7f0d2c101210 2024-03-19T01:11:30.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.353+0000 7f0d29ffb700 1 -- 172.21.15.116:0/2192049140 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2000f460 con 0x7f0d2c101210 2024-03-19T01:11:30.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.354+0000 7f0d32b2a700 1 -- 172.21.15.116:0/2192049140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d2c1a55a0 con 0x7f0d2c101210 2024-03-19T01:11:30.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.355+0000 7f0d32b2a700 1 -- 172.21.15.116:0/2192049140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d2c068fd0 con 0x7f0d2c101210 2024-03-19T01:11:30.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.356+0000 7f0d29ffb700 1 -- 172.21.15.116:0/2192049140 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f0d2002a070 con 0x7f0d2c101210 2024-03-19T01:11:30.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.356+0000 7f0d29ffb700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0d1c0a2920 0x7f0d1c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:30.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.357+0000 7f0d2bfff700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0d1c0a2920 0x7f0d1c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:30.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.357+0000 7f0d29ffb700 1 -- 172.21.15.116:0/2192049140 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0d200cfb90 con 0x7f0d2c101210 2024-03-19T01:11:30.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.357+0000 7f0d2bfff700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0d1c0a2920 0x7f0d1c0a4de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c06b0a0 tx=0x7f0d18009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:30.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.360+0000 7f0d29ffb700 1 -- 172.21.15.116:0/2192049140 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0d20013070 con 0x7f0d2c101210 2024-03-19T01:11:30.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:30 smithi116 ceph-mon[62701]: pgmap v318: 97 pgs: 3 peering, 94 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 4 op/s; 146 B/s, 0 objects/s recovering 2024-03-19T01:11:30.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:30 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4097553309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T01:11:30.481 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:30 smithi156 ceph-mon[66799]: pgmap v318: 97 pgs: 3 peering, 94 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 4 op/s; 146 B/s, 0 objects/s recovering 2024-03-19T01:11:30.481 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:30 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/4097553309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T01:11:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.599+0000 7f0d32b2a700 1 -- 172.21.15.116:0/2192049140 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0d2c1a5770 con 0x7f0d2c101210 2024-03-19T01:11:30.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.600+0000 7f0d29ffb700 1 -- 172.21.15.116:0/2192049140 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0d2009cb40 con 0x7f0d2c101210 2024-03-19T01:11:30.609 INFO:teuthology.orchestra.run.smithi116.stdout:34359738448 2024-03-19T01:11:30.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.602+0000 7f0d32b2a700 1 -- 172.21.15.116:0/2192049140 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0d1c0a2920 msgr2=0x7f0d1c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:30.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.602+0000 7f0d32b2a700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0d1c0a2920 0x7f0d1c0a4de0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c06b0a0 tx=0x7f0d18009260 comp rx=0 tx=0).stop 2024-03-19T01:11:30.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.602+0000 7f0d32b2a700 1 -- 172.21.15.116:0/2192049140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d2c101210 msgr2=0x7f0d2c19e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:30.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.602+0000 7f0d32b2a700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d2c101210 0x7f0d2c19e6d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0d20005910 tx=0x7f0d2000bab0 comp rx=0 tx=0).stop 2024-03-19T01:11:30.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.602+0000 7f0d32b2a700 1 -- 172.21.15.116:0/2192049140 shutdown_connections 2024-03-19T01:11:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.602+0000 7f0d32b2a700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d2c107250 0x7f0d2c1988e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.602+0000 7f0d32b2a700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d2c101210 0x7f0d2c19e6d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.602+0000 7f0d32b2a700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0d1c0a2920 0x7f0d1c0a4de0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.602+0000 7f0d32b2a700 1 --2- 172.21.15.116:0/2192049140 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d2c101bb0 0x7f0d2c19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.602+0000 7f0d32b2a700 1 -- 172.21.15.116:0/2192049140 >> 172.21.15.116:0/2192049140 conn(0x7f0d2c078140 msgr2=0x7f0d2c1054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.602+0000 7f0d32b2a700 1 -- 172.21.15.116:0/2192049140 shutdown_connections 2024-03-19T01:11:30.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:30.602+0000 7f0d32b2a700 1 -- 172.21.15.116:0/2192049140 wait complete. 2024-03-19T01:11:30.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:30 smithi162 ceph-mon[67178]: pgmap v318: 97 pgs: 3 peering, 94 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 4 op/s; 146 B/s, 0 objects/s recovering 2024-03-19T01:11:30.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:30 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/4097553309' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-19T01:11:31.367 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:31 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2192049140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T01:11:31.367 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:31 smithi116 ceph-mon[62701]: pgmap v319: 97 pgs: 3 peering, 94 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 5 op/s; 160 B/s, 1 keys/s, 1 objects/s recovering 2024-03-19T01:11:31.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:31 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2192049140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T01:11:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:31 smithi156 ceph-mon[66799]: pgmap v319: 97 pgs: 3 peering, 94 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 5 op/s; 160 B/s, 1 keys/s, 1 objects/s recovering 2024-03-19T01:11:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:31 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2192049140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-19T01:11:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:31 smithi162 ceph-mon[67178]: pgmap v319: 97 pgs: 3 peering, 94 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 5 op/s; 160 B/s, 1 keys/s, 1 objects/s recovering 2024-03-19T01:11:32.361 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:32 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3149511312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T01:11:32.362 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:32 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:11:32.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.818+0000 7fc1698ae700 1 -- 172.21.15.116:0/3185956656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc164102550 msgr2=0x7fc164102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:32.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.818+0000 7fc1698ae700 1 --2- 172.21.15.116:0/3185956656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc164102550 0x7fc164102930 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc1540099c0 tx=0x7fc154009cd0 comp rx=0 tx=0).stop 2024-03-19T01:11:32.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.818+0000 7fc1698ae700 1 -- 172.21.15.116:0/3185956656 shutdown_connections 2024-03-19T01:11:32.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.818+0000 7fc1698ae700 1 --2- 172.21.15.116:0/3185956656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc164107630 0x7fc16410f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.818+0000 7fc1698ae700 1 --2- 172.21.15.116:0/3185956656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc164102550 0x7fc164102930 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.818+0000 7fc1698ae700 1 --2- 172.21.15.116:0/3185956656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc164102f00 0x7fc164106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.818+0000 7fc1698ae700 1 -- 172.21.15.116:0/3185956656 >> 172.21.15.116:0/3185956656 conn(0x7fc1640fddc0 msgr2=0x7fc1641001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.818+0000 7fc1698ae700 1 -- 172.21.15.116:0/3185956656 shutdown_connections 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.818+0000 7fc1698ae700 1 -- 172.21.15.116:0/3185956656 wait complete. 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.819+0000 7fc1698ae700 1 Processor -- start 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.819+0000 7fc1698ae700 1 -- start start 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.819+0000 7fc1698ae700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc164102550 0x7fc16419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.819+0000 7fc1698ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc164102f00 0x7fc16419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.819+0000 7fc1698ae700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc164107630 0x7fc1641988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.819+0000 7fc1698ae700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc164112630 con 0x7fc164102f00 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.819+0000 7fc1698ae700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc1641124b0 con 0x7fc164102550 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.819+0000 7fc1698ae700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc1641127b0 con 0x7fc164107630 2024-03-19T01:11:32.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.819+0000 7fc1627fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc164102f00 0x7fc16419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.819+0000 7fc162ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc164102550 0x7fc16419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.819+0000 7fc1627fc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc164102f00 0x7fc16419ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36912/0 (socket says 172.21.15.116:36912) 2024-03-19T01:11:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.819+0000 7fc1627fc700 1 -- 172.21.15.116:0/3149511312 learned_addr learned my addr 172.21.15.116:0/3149511312 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.819+0000 7fc1637fe700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc164107630 0x7fc1641988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.820+0000 7fc1637fe700 1 -- 172.21.15.116:0/3149511312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc164102550 msgr2=0x7fc16419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.820+0000 7fc1637fe700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc164102550 0x7fc16419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.820+0000 7fc1637fe700 1 -- 172.21.15.116:0/3149511312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc164102f00 msgr2=0x7fc16419ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:32.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.820+0000 7fc1637fe700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc164102f00 0x7fc16419ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:32.430 INFO:teuthology.orchestra.run.smithi116.stdout:163208757304 2024-03-19T01:11:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.820+0000 7fc1637fe700 1 -- 172.21.15.116:0/3149511312 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc164199030 con 0x7fc164107630 2024-03-19T01:11:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.820+0000 7fc1627fc700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc164102f00 0x7fc16419ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.820+0000 7fc162ffd700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc164102550 0x7fc16419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.820+0000 7fc1637fe700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc164107630 0x7fc1641988d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00efc0 tx=0x7fc15c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.820+0000 7fc1688ac700 1 -- 172.21.15.116:0/3149511312 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c0092e0 con 0x7fc164107630 2024-03-19T01:11:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.820+0000 7fc1688ac700 1 -- 172.21.15.116:0/3149511312 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc15c004bb0 con 0x7fc164107630 2024-03-19T01:11:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.820+0000 7fc1688ac700 1 -- 172.21.15.116:0/3149511312 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc15c004280 con 0x7fc164107630 2024-03-19T01:11:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.820+0000 7fc1698ae700 1 -- 172.21.15.116:0/3149511312 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1641992c0 con 0x7fc164107630 2024-03-19T01:11:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.820+0000 7fc1698ae700 1 -- 172.21.15.116:0/3149511312 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1641a55e0 con 0x7fc164107630 2024-03-19T01:11:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.821+0000 7fc1698ae700 1 -- 172.21.15.116:0/3149511312 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc164199e00 con 0x7fc164107630 2024-03-19T01:11:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.823+0000 7fc1688ac700 1 -- 172.21.15.116:0/3149511312 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fc15c01b030 con 0x7fc164107630 2024-03-19T01:11:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.823+0000 7fc1688ac700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc14c0a29a0 0x7fc14c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.823+0000 7fc162ffd700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc14c0a29a0 0x7fc14c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.823+0000 7fc1688ac700 1 -- 172.21.15.116:0/3149511312 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc15c0ce770 con 0x7fc164107630 2024-03-19T01:11:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.824+0000 7fc162ffd700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc14c0a29a0 0x7fc14c0a4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc154000c00 tx=0x7fc154005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:31.826+0000 7fc1688ac700 1 -- 172.21.15.116:0/3149511312 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc15c022020 con 0x7fc164107630 2024-03-19T01:11:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.073+0000 7fc1698ae700 1 -- 172.21.15.116:0/3149511312 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc164068fb0 con 0x7fc164107630 2024-03-19T01:11:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.073+0000 7fc1688ac700 1 -- 172.21.15.116:0/3149511312 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc15c010040 con 0x7fc164107630 2024-03-19T01:11:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.075+0000 7fc1698ae700 1 -- 172.21.15.116:0/3149511312 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc14c0a29a0 msgr2=0x7fc14c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.075+0000 7fc1698ae700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc14c0a29a0 0x7fc14c0a4e60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc154000c00 tx=0x7fc154005fb0 comp rx=0 tx=0).stop 2024-03-19T01:11:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.075+0000 7fc1698ae700 1 -- 172.21.15.116:0/3149511312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc164107630 msgr2=0x7fc1641988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.075+0000 7fc1698ae700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc164107630 0x7fc1641988d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00efc0 tx=0x7fc15c00c3a0 comp rx=0 tx=0).stop 2024-03-19T01:11:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.075+0000 7fc1698ae700 1 -- 172.21.15.116:0/3149511312 shutdown_connections 2024-03-19T01:11:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.075+0000 7fc1698ae700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc14c0a29a0 0x7fc14c0a4e60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.075+0000 7fc1698ae700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc164107630 0x7fc1641988d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.075+0000 7fc1698ae700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc164102550 0x7fc16419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.075+0000 7fc1698ae700 1 --2- 172.21.15.116:0/3149511312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc164102f00 0x7fc16419ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.075+0000 7fc1698ae700 1 -- 172.21.15.116:0/3149511312 >> 172.21.15.116:0/3149511312 conn(0x7fc1640fddc0 msgr2=0x7fc1641000e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.076+0000 7fc1698ae700 1 -- 172.21.15.116:0/3149511312 shutdown_connections 2024-03-19T01:11:32.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:32.076+0000 7fc1698ae700 1 -- 172.21.15.116:0/3149511312 wait complete. 2024-03-19T01:11:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:32 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3149511312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T01:11:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:32 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:11:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:32 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3149511312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-19T01:11:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:32 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:11:33.050 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757300 got 163208757304 for osd.5 2024-03-19T01:11:33.050 DEBUG:teuthology.parallel:result is None 2024-03-19T01:11:33.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:33 smithi156 ceph-mon[66799]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 op/s; 160 B/s, 2 keys/s, 2 objects/s recovering 2024-03-19T01:11:33.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:33 smithi162 ceph-mon[67178]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 op/s; 160 B/s, 2 keys/s, 2 objects/s recovering 2024-03-19T01:11:33.512 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:33 smithi116 ceph-mon[62701]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 op/s; 160 B/s, 2 keys/s, 2 objects/s recovering 2024-03-19T01:11:33.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.513+0000 7f23a1896700 1 -- 172.21.15.116:0/1878108452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c101bb0 msgr2=0x7f239c10e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.513+0000 7f23a1896700 1 --2- 172.21.15.116:0/1878108452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c101bb0 0x7f239c10e090 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f239400b0a0 tx=0x7f239400b3b0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.513+0000 7f23a1896700 1 -- 172.21.15.116:0/1878108452 shutdown_connections 2024-03-19T01:11:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.513+0000 7f23a1896700 1 --2- 172.21.15.116:0/1878108452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c101bb0 0x7f239c10e090 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.513+0000 7f23a1896700 1 --2- 172.21.15.116:0/1878108452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f239c107250 0x7f239c107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.513+0000 7f23a1896700 1 --2- 172.21.15.116:0/1878108452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f239c101210 0x7f239c101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.513+0000 7f23a1896700 1 -- 172.21.15.116:0/1878108452 >> 172.21.15.116:0/1878108452 conn(0x7f239c078140 msgr2=0x7f239c078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.513+0000 7f23a1896700 1 -- 172.21.15.116:0/1878108452 shutdown_connections 2024-03-19T01:11:33.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.513+0000 7f23a1896700 1 -- 172.21.15.116:0/1878108452 wait complete. 2024-03-19T01:11:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.514+0000 7f23a1896700 1 Processor -- start 2024-03-19T01:11:33.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.514+0000 7f23a1896700 1 -- start start 2024-03-19T01:11:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.514+0000 7f23a1896700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c101210 0x7f239c199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:33.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.514+0000 7f23a1896700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f239c101bb0 0x7f239c199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.514+0000 7f23a1896700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f239c107250 0x7f239c1a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.514+0000 7f23a1896700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f239c1114a0 con 0x7f239c101210 2024-03-19T01:11:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.514+0000 7f23a1896700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f239c111320 con 0x7f239c107250 2024-03-19T01:11:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.514+0000 7f23a1896700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f239c111620 con 0x7f239c101bb0 2024-03-19T01:11:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.514+0000 7f239affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c101210 0x7f239c199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.514+0000 7f239affd700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c101210 0x7f239c199940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36930/0 (socket says 172.21.15.116:36930) 2024-03-19T01:11:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.514+0000 7f239affd700 1 -- 172.21.15.116:0/3177125733 learned_addr learned my addr 172.21.15.116:0/3177125733 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.514+0000 7f239a7fc700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f239c101bb0 0x7f239c199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:33.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.515+0000 7f239b7fe700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f239c107250 0x7f239c1a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.515+0000 7f239affd700 1 -- 172.21.15.116:0/3177125733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f239c101bb0 msgr2=0x7f239c199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.515+0000 7f239affd700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f239c101bb0 0x7f239c199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.515+0000 7f239affd700 1 -- 172.21.15.116:0/3177125733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f239c107250 msgr2=0x7f239c1a49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.515+0000 7f239affd700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f239c107250 0x7f239c1a49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.515+0000 7f239affd700 1 -- 172.21.15.116:0/3177125733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f239c1a4f00 con 0x7f239c101210 2024-03-19T01:11:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.515+0000 7f239affd700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c101210 0x7f239c199940 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f239000e9a0 tx=0x7f239000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.515+0000 7f23a0894700 1 -- 172.21.15.116:0/3177125733 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239000ca10 con 0x7f239c101210 2024-03-19T01:11:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.515+0000 7f23a1896700 1 -- 172.21.15.116:0/3177125733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f239c197730 con 0x7f239c101210 2024-03-19T01:11:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.515+0000 7f23a0894700 1 -- 172.21.15.116:0/3177125733 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f239000cbb0 con 0x7f239c101210 2024-03-19T01:11:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.515+0000 7f23a0894700 1 -- 172.21.15.116:0/3177125733 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2390018680 con 0x7f239c101210 2024-03-19T01:11:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.515+0000 7f23a1896700 1 -- 172.21.15.116:0/3177125733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f239c197c20 con 0x7f239c101210 2024-03-19T01:11:33.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.516+0000 7f23a1896700 1 -- 172.21.15.116:0/3177125733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f239c068fd0 con 0x7f239c101210 2024-03-19T01:11:33.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.517+0000 7f23a0894700 1 -- 172.21.15.116:0/3177125733 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f2390010430 con 0x7f239c101210 2024-03-19T01:11:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.518+0000 7f23a0894700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f23840a29a0 0x7f23840a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.518+0000 7f239a7fc700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f23840a29a0 0x7f23840a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.518+0000 7f23a0894700 1 -- 172.21.15.116:0/3177125733 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f23900cf2d0 con 0x7f239c101210 2024-03-19T01:11:33.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.518+0000 7f239a7fc700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f23840a29a0 0x7f23840a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f239c078640 tx=0x7f238c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:33.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.525+0000 7f23a0894700 1 -- 172.21.15.116:0/3177125733 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f239009c190 con 0x7f239c101210 2024-03-19T01:11:33.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.730+0000 7fc1f3419700 1 -- 172.21.15.116:0/3653848115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec101c70 msgr2=0x7fc1ec102050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:33.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.730+0000 7fc1f3419700 1 --2- 172.21.15.116:0/3653848115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec101c70 0x7fc1ec102050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80099c0 tx=0x7fc1e8009cd0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.730+0000 7fc1f3419700 1 -- 172.21.15.116:0/3653848115 shutdown_connections 2024-03-19T01:11:33.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.730+0000 7fc1f3419700 1 --2- 172.21.15.116:0/3653848115 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc1ec106590 0x7fc1ec10e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.730+0000 7fc1f3419700 1 --2- 172.21.15.116:0/3653848115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec101c70 0x7fc1ec102050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.730+0000 7fc1f3419700 1 --2- 172.21.15.116:0/3653848115 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1ec102620 0x7fc1ec105eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.730+0000 7fc1f3419700 1 -- 172.21.15.116:0/3653848115 >> 172.21.15.116:0/3653848115 conn(0x7fc1ec077810 msgr2=0x7fc1ec0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:33.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.730+0000 7fc1f3419700 1 -- 172.21.15.116:0/3653848115 shutdown_connections 2024-03-19T01:11:33.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.730+0000 7fc1f3419700 1 -- 172.21.15.116:0/3653848115 wait complete. 2024-03-19T01:11:33.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.731+0000 7fc1f3419700 1 Processor -- start 2024-03-19T01:11:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.731+0000 7fc1f3419700 1 -- start start 2024-03-19T01:11:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.731+0000 7fc1f3419700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1ec101c70 0x7fc1ec0fccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.731+0000 7fc1f3419700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec102620 0x7fc1ec0fd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.731+0000 7fc1f3419700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc1ec106590 0x7fc1ec0fd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.731+0000 7fc1f3419700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec111570 con 0x7fc1ec102620 2024-03-19T01:11:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.731+0000 7fc1f3419700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec1113f0 con 0x7fc1ec106590 2024-03-19T01:11:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.731+0000 7fc1f3419700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc1ec1116f0 con 0x7fc1ec101c70 2024-03-19T01:11:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.731+0000 7fc1f09b4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec102620 0x7fc1ec0fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.731+0000 7fc1f09b4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec102620 0x7fc1ec0fd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36952/0 (socket says 172.21.15.116:36952) 2024-03-19T01:11:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.731+0000 7fc1f11b5700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1ec101c70 0x7fc1ec0fccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.731+0000 7fc1f09b4700 1 -- 172.21.15.116:0/3977590673 learned_addr learned my addr 172.21.15.116:0/3977590673 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.731+0000 7fc1f19b6700 1 --2- 172.21.15.116:0/3977590673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc1ec106590 0x7fc1ec0fd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.732+0000 7fc1f09b4700 1 -- 172.21.15.116:0/3977590673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1ec101c70 msgr2=0x7fc1ec0fccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.732+0000 7fc1f09b4700 1 --2- 172.21.15.116:0/3977590673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1ec101c70 0x7fc1ec0fccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.732+0000 7fc1f09b4700 1 -- 172.21.15.116:0/3977590673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc1ec106590 msgr2=0x7fc1ec0fd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.732+0000 7fc1f09b4700 1 --2- 172.21.15.116:0/3977590673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc1ec106590 0x7fc1ec0fd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.732+0000 7fc1f09b4700 1 -- 172.21.15.116:0/3977590673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1ec1a7040 con 0x7fc1ec102620 2024-03-19T01:11:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.732+0000 7fc1f09b4700 1 --2- 172.21.15.116:0/3977590673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec102620 0x7fc1ec0fd230 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc00d950 tx=0x7fc1dc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.732+0000 7fc1e27fc700 1 -- 172.21.15.116:0/3977590673 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc016070 con 0x7fc1ec102620 2024-03-19T01:11:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.732+0000 7fc1f3419700 1 -- 172.21.15.116:0/3977590673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1ec1a7330 con 0x7fc1ec102620 2024-03-19T01:11:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.732+0000 7fc1f3419700 1 -- 172.21.15.116:0/3977590673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1ec1a7870 con 0x7fc1ec102620 2024-03-19T01:11:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.732+0000 7fc1e27fc700 1 -- 172.21.15.116:0/3977590673 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1dc004d10 con 0x7fc1ec102620 2024-03-19T01:11:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.732+0000 7fc1e27fc700 1 -- 172.21.15.116:0/3977590673 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1dc005650 con 0x7fc1ec102620 2024-03-19T01:11:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.733+0000 7fc1f3419700 1 -- 172.21.15.116:0/3977590673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1ec102a20 con 0x7fc1ec102620 2024-03-19T01:11:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.734+0000 7fc1e27fc700 1 -- 172.21.15.116:0/3977590673 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fc1dc004750 con 0x7fc1ec102620 2024-03-19T01:11:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.735+0000 7fc1e27fc700 1 --2- 172.21.15.116:0/3977590673 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc1d80a2920 0x7fc1d80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.735+0000 7fc1f11b5700 1 --2- 172.21.15.116:0/3977590673 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc1d80a2920 0x7fc1d80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.735+0000 7fc1e27fc700 1 -- 172.21.15.116:0/3977590673 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc1dc0cedd0 con 0x7fc1ec102620 2024-03-19T01:11:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.735+0000 7fc1f11b5700 1 --2- 172.21.15.116:0/3977590673 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc1d80a2920 0x7fc1d80a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8000c00 tx=0x7fc1e8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.738+0000 7f23a1896700 1 -- 172.21.15.116:0/3177125733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f239c02dbb0 con 0x7f239c101210 2024-03-19T01:11:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.740+0000 7f23a0894700 1 -- 172.21.15.116:0/3177125733 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f239009ba90 con 0x7f239c101210 2024-03-19T01:11:33.741 INFO:teuthology.orchestra.run.smithi116.stdout:309237645342 2024-03-19T01:11:33.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.740+0000 7fc1e27fc700 1 -- 172.21.15.116:0/3977590673 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc1dc01f020 con 0x7fc1ec102620 2024-03-19T01:11:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.743+0000 7f23a1896700 1 -- 172.21.15.116:0/3177125733 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f23840a29a0 msgr2=0x7f23840a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.743+0000 7f23a1896700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f23840a29a0 0x7f23840a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f239c078640 tx=0x7f238c009260 comp rx=0 tx=0).stop 2024-03-19T01:11:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.743+0000 7f23a1896700 1 -- 172.21.15.116:0/3177125733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c101210 msgr2=0x7f239c199940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.743+0000 7f23a1896700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c101210 0x7f239c199940 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f239000e9a0 tx=0x7f239000ecb0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.743+0000 7f23a1896700 1 -- 172.21.15.116:0/3177125733 shutdown_connections 2024-03-19T01:11:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.743+0000 7f23a1896700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f239c107250 0x7f239c1a49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.743+0000 7f23a1896700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f239c101210 0x7f239c199940 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.743+0000 7f23a1896700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f23840a29a0 0x7f23840a4e60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.743+0000 7f23a1896700 1 --2- 172.21.15.116:0/3177125733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f239c101bb0 0x7f239c199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.743+0000 7f23a1896700 1 -- 172.21.15.116:0/3177125733 >> 172.21.15.116:0/3177125733 conn(0x7f239c078140 msgr2=0x7f239c0ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.743+0000 7f23a1896700 1 -- 172.21.15.116:0/3177125733 shutdown_connections 2024-03-19T01:11:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.743+0000 7f23a1896700 1 -- 172.21.15.116:0/3177125733 wait complete. 2024-03-19T01:11:33.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.965+0000 7fc1f3419700 1 -- 172.21.15.116:0/3977590673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fc1ec0fe220 con 0x7fc1ec102620 2024-03-19T01:11:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.966+0000 7fc1e27fc700 1 -- 172.21.15.116:0/3977590673 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc1dc09bcf0 con 0x7fc1ec102620 2024-03-19T01:11:33.967 INFO:teuthology.orchestra.run.smithi116.stdout:141733920829 2024-03-19T01:11:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.968+0000 7fc1f3419700 1 -- 172.21.15.116:0/3977590673 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc1d80a2920 msgr2=0x7fc1d80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.968+0000 7fc1f3419700 1 --2- 172.21.15.116:0/3977590673 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc1d80a2920 0x7fc1d80a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8000c00 tx=0x7fc1e8005fb0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.968+0000 7fc1f3419700 1 -- 172.21.15.116:0/3977590673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec102620 msgr2=0x7fc1ec0fd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:33.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.969+0000 7fc1f3419700 1 --2- 172.21.15.116:0/3977590673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec102620 0x7fc1ec0fd230 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc00d950 tx=0x7fc1dc00dc60 comp rx=0 tx=0).stop 2024-03-19T01:11:33.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.969+0000 7fc1f3419700 1 -- 172.21.15.116:0/3977590673 shutdown_connections 2024-03-19T01:11:33.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.969+0000 7fc1f3419700 1 --2- 172.21.15.116:0/3977590673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc1ec106590 0x7fc1ec0fd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.969+0000 7fc1f3419700 1 --2- 172.21.15.116:0/3977590673 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc1d80a2920 0x7fc1d80a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.969+0000 7fc1f3419700 1 --2- 172.21.15.116:0/3977590673 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc1ec101c70 0x7fc1ec0fccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.969+0000 7fc1f3419700 1 --2- 172.21.15.116:0/3977590673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1ec102620 0x7fc1ec0fd230 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:33.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.969+0000 7fc1f3419700 1 -- 172.21.15.116:0/3977590673 >> 172.21.15.116:0/3977590673 conn(0x7fc1ec077810 msgr2=0x7fc1ec0ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:33.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.969+0000 7fc1f3419700 1 -- 172.21.15.116:0/3977590673 shutdown_connections 2024-03-19T01:11:33.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:33.969+0000 7fc1f3419700 1 -- 172.21.15.116:0/3977590673 wait complete. 2024-03-19T01:11:34.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:34 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3177125733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-19T01:11:34.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:34 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:34.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:34 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:34.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:34 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:34.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:34 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3977590673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T01:11:34.471 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:34 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:34.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:34 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3177125733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-19T01:11:34.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:34 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:34 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:34 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:34 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3977590673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T01:11:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:34 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:34 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3177125733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-19T01:11:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:34 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:34 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:34 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:34 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3977590673' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-19T01:11:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:34 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:34.779 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920825 got 141733920829 for osd.4 2024-03-19T01:11:34.779 DEBUG:teuthology.parallel:result is None 2024-03-19T01:11:35.354 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:35 smithi116 ceph-mon[62701]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5 op/s; 160 B/s, 2 keys/s, 2 objects/s recovering 2024-03-19T01:11:35.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:35 smithi156 ceph-mon[66799]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5 op/s; 160 B/s, 2 keys/s, 2 objects/s recovering 2024-03-19T01:11:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:35 smithi162 ceph-mon[67178]: pgmap v321: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 5 op/s; 160 B/s, 2 keys/s, 2 objects/s recovering 2024-03-19T01:11:35.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.893+0000 7f9697799700 1 -- 172.21.15.116:0/3328974511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690107680 msgr2=0x7f969010f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:35.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.893+0000 7f9697799700 1 --2- 172.21.15.116:0/3328974511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690107680 0x7f969010f200 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f968000b0d0 tx=0x7f968000b3e0 comp rx=0 tx=0).stop 2024-03-19T01:11:35.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.893+0000 7f9697799700 1 -- 172.21.15.116:0/3328974511 shutdown_connections 2024-03-19T01:11:35.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.893+0000 7f9697799700 1 --2- 172.21.15.116:0/3328974511 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9690107680 0x7f969010f200 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:35.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.893+0000 7f9697799700 1 --2- 172.21.15.116:0/3328974511 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f969006ac10 0x7f969006aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:35.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.893+0000 7f9697799700 1 --2- 172.21.15.116:0/3328974511 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f969006b5c0 0x7f9690106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:35.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.893+0000 7f9697799700 1 -- 172.21.15.116:0/3328974511 >> 172.21.15.116:0/3328974511 conn(0x7f9690078180 msgr2=0x7f9690078590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:35.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.893+0000 7f9697799700 1 -- 172.21.15.116:0/3328974511 shutdown_connections 2024-03-19T01:11:35.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.893+0000 7f9697799700 1 -- 172.21.15.116:0/3328974511 wait complete. 2024-03-19T01:11:35.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9697799700 1 Processor -- start 2024-03-19T01:11:35.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9697799700 1 -- start start 2024-03-19T01:11:35.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9697799700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f969006ac10 0x7f969019e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:35.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9697799700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f969006b5c0 0x7f969019ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:35.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9697799700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9690107680 0x7f9690198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:35.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9697799700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9690112690 con 0x7f969006ac10 2024-03-19T01:11:35.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9697799700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9690112510 con 0x7f969006b5c0 2024-03-19T01:11:35.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9697799700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9690112810 con 0x7f9690107680 2024-03-19T01:11:35.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9695535700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f969006ac10 0x7f969019e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:35.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9694d34700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f969006b5c0 0x7f969019ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:35.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9695535700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f969006ac10 0x7f969019e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36970/0 (socket says 172.21.15.116:36970) 2024-03-19T01:11:35.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9695535700 1 -- 172.21.15.116:0/1447341562 learned_addr learned my addr 172.21.15.116:0/1447341562 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9695d36700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9690107680 0x7f9690198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9695535700 1 -- 172.21.15.116:0/1447341562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9690107680 msgr2=0x7f9690198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9695535700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9690107680 0x7f9690198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9695535700 1 -- 172.21.15.116:0/1447341562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f969006b5c0 msgr2=0x7f969019ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:35.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9695535700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f969006b5c0 0x7f969019ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9695535700 1 -- 172.21.15.116:0/1447341562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96901990b0 con 0x7f969006ac10 2024-03-19T01:11:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.894+0000 7f9695d36700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9690107680 0x7f9690198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:11:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.895+0000 7f9695535700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f969006ac10 0x7f969019e740 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f968800e9c0 tx=0x7f968800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.895+0000 7f968e7fc700 1 -- 172.21.15.116:0/1447341562 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f968800cbd0 con 0x7f969006ac10 2024-03-19T01:11:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.895+0000 7f9697799700 1 -- 172.21.15.116:0/1447341562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96901993a0 con 0x7f969006ac10 2024-03-19T01:11:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.895+0000 7f968e7fc700 1 -- 172.21.15.116:0/1447341562 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f968800cd70 con 0x7f969006ac10 2024-03-19T01:11:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.895+0000 7f968e7fc700 1 -- 172.21.15.116:0/1447341562 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9688010430 con 0x7f969006ac10 2024-03-19T01:11:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.895+0000 7f9697799700 1 -- 172.21.15.116:0/1447341562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96901a5610 con 0x7f969006ac10 2024-03-19T01:11:35.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.896+0000 7f9677fff700 1 -- 172.21.15.116:0/1447341562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9690103a30 con 0x7f969006ac10 2024-03-19T01:11:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.897+0000 7f968e7fc700 1 -- 172.21.15.116:0/1447341562 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f9688004750 con 0x7f969006ac10 2024-03-19T01:11:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.898+0000 7f968e7fc700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f96780a2920 0x7f96780a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.898+0000 7f9694d34700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f96780a2920 0x7f96780a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.898+0000 7f968e7fc700 1 -- 172.21.15.116:0/1447341562 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f96880cee90 con 0x7f969006ac10 2024-03-19T01:11:35.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.899+0000 7f9694d34700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f96780a2920 0x7f96780a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9690100d90 tx=0x7f9684009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:35.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:35.903+0000 7f968e7fc700 1 -- 172.21.15.116:0/1447341562 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f968809bdb0 con 0x7f969006ac10 2024-03-19T01:11:36.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.117+0000 7f9677fff700 1 -- 172.21.15.116:0/1447341562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f9690199e30 con 0x7f969006ac10 2024-03-19T01:11:36.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.118+0000 7f968e7fc700 1 -- 172.21.15.116:0/1447341562 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f96880bb030 con 0x7f969006ac10 2024-03-19T01:11:36.119 INFO:teuthology.orchestra.run.smithi116.stdout:77309411400 2024-03-19T01:11:36.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.120+0000 7f9677fff700 1 -- 172.21.15.116:0/1447341562 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f96780a2920 msgr2=0x7f96780a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:36.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.120+0000 7f9677fff700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f96780a2920 0x7f96780a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9690100d90 tx=0x7f9684009260 comp rx=0 tx=0).stop 2024-03-19T01:11:36.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.120+0000 7f9677fff700 1 -- 172.21.15.116:0/1447341562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f969006ac10 msgr2=0x7f969019e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:36.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.120+0000 7f9677fff700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f969006ac10 0x7f969019e740 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f968800e9c0 tx=0x7f968800ecd0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.120+0000 7f9677fff700 1 -- 172.21.15.116:0/1447341562 shutdown_connections 2024-03-19T01:11:36.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.120+0000 7f9677fff700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f96780a2920 0x7f96780a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.120+0000 7f9677fff700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9690107680 0x7f9690198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.120+0000 7f9677fff700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f969006ac10 0x7f969019e740 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.120+0000 7f9677fff700 1 --2- 172.21.15.116:0/1447341562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f969006b5c0 0x7f969019ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.120+0000 7f9677fff700 1 -- 172.21.15.116:0/1447341562 >> 172.21.15.116:0/1447341562 conn(0x7f9690078180 msgr2=0x7f96900ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:36.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.120+0000 7f9677fff700 1 -- 172.21.15.116:0/1447341562 shutdown_connections 2024-03-19T01:11:36.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.120+0000 7f9677fff700 1 -- 172.21.15.116:0/1447341562 wait complete. 2024-03-19T01:11:36.319 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:36 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1447341562' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T01:11:36.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.419+0000 7fb6266bb700 1 -- 172.21.15.116:0/1386474293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb620101bb0 msgr2=0x7fb62010e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.419+0000 7fb6266bb700 1 --2- 172.21.15.116:0/1386474293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb620101bb0 0x7fb62010e090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb61800b0a0 tx=0x7fb61800b3b0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.419+0000 7fb6266bb700 1 -- 172.21.15.116:0/1386474293 shutdown_connections 2024-03-19T01:11:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.419+0000 7fb6266bb700 1 --2- 172.21.15.116:0/1386474293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb620101bb0 0x7fb62010e090 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.419+0000 7fb6266bb700 1 --2- 172.21.15.116:0/1386474293 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb620107250 0x7fb620107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.419+0000 7fb6266bb700 1 --2- 172.21.15.116:0/1386474293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb620101210 0x7fb620101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.419+0000 7fb6266bb700 1 -- 172.21.15.116:0/1386474293 >> 172.21.15.116:0/1386474293 conn(0x7fb620078140 msgr2=0x7fb620078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.419+0000 7fb6266bb700 1 -- 172.21.15.116:0/1386474293 shutdown_connections 2024-03-19T01:11:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.419+0000 7fb6266bb700 1 -- 172.21.15.116:0/1386474293 wait complete. 2024-03-19T01:11:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb6266bb700 1 Processor -- start 2024-03-19T01:11:36.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb6266bb700 1 -- start start 2024-03-19T01:11:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb6266bb700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb620101210 0x7fb62019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb6266bb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb620101bb0 0x7fb62019ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb6266bb700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb620107250 0x7fb6201988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb6266bb700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb620112600 con 0x7fb620107250 2024-03-19T01:11:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb6266bb700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb620112480 con 0x7fb620101bb0 2024-03-19T01:11:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb6266bb700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb620112780 con 0x7fb620101210 2024-03-19T01:11:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb624c58700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb620107250 0x7fb6201988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:36.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb61f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb620101bb0 0x7fb62019ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb61ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb620101210 0x7fb62019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb61f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb620101bb0 0x7fb62019ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:46182/0 (socket says 172.21.15.116:46182) 2024-03-19T01:11:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb61ffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb620101210 0x7fb62019e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:49868/0 (socket says 172.21.15.116:49868) 2024-03-19T01:11:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb624c58700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb620107250 0x7fb6201988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36996/0 (socket says 172.21.15.116:36996) 2024-03-19T01:11:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.420+0000 7fb61f7fe700 1 -- 172.21.15.116:0/2885711160 learned_addr learned my addr 172.21.15.116:0/2885711160 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.421+0000 7fb61f7fe700 1 -- 172.21.15.116:0/2885711160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb620101210 msgr2=0x7fb62019e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.421+0000 7fb61f7fe700 1 --2- 172.21.15.116:0/2885711160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb620101210 0x7fb62019e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.421+0000 7fb61f7fe700 1 -- 172.21.15.116:0/2885711160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb620107250 msgr2=0x7fb6201988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.421+0000 7fb61f7fe700 1 --2- 172.21.15.116:0/2885711160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb620107250 0x7fb6201988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.421+0000 7fb61f7fe700 1 -- 172.21.15.116:0/2885711160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb620199080 con 0x7fb620101bb0 2024-03-19T01:11:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.421+0000 7fb61ffff700 1 --2- 172.21.15.116:0/2885711160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb620101210 0x7fb62019e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.421+0000 7fb61f7fe700 1 --2- 172.21.15.116:0/2885711160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb620101bb0 0x7fb62019ec10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb61400e9a0 tx=0x7fb61400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:36.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.421+0000 7fb61d7fa700 1 -- 172.21.15.116:0/2885711160 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb61400ca10 con 0x7fb620101bb0 2024-03-19T01:11:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.421+0000 7fb61d7fa700 1 -- 172.21.15.116:0/2885711160 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb61400cbb0 con 0x7fb620101bb0 2024-03-19T01:11:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.421+0000 7fb6266bb700 1 -- 172.21.15.116:0/2885711160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb620199370 con 0x7fb620101bb0 2024-03-19T01:11:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.421+0000 7fb61d7fa700 1 -- 172.21.15.116:0/2885711160 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614018680 con 0x7fb620101bb0 2024-03-19T01:11:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.421+0000 7fb6266bb700 1 -- 172.21.15.116:0/2885711160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6201a5530 con 0x7fb620101bb0 2024-03-19T01:11:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.422+0000 7fb6266bb700 1 -- 172.21.15.116:0/2885711160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb620102900 con 0x7fb620101bb0 2024-03-19T01:11:36.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.423+0000 7fb61d7fa700 1 -- 172.21.15.116:0/2885711160 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fb614010430 con 0x7fb620101bb0 2024-03-19T01:11:36.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.423+0000 7fb61d7fa700 1 --2- 172.21.15.116:0/2885711160 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6080a29a0 0x7fb6080a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:36.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.424+0000 7fb61ffff700 1 --2- 172.21.15.116:0/2885711160 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6080a29a0 0x7fb6080a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:36.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.424+0000 7fb61d7fa700 1 -- 172.21.15.116:0/2885711160 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb6140cedc0 con 0x7fb620101bb0 2024-03-19T01:11:36.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.424+0000 7fb61ffff700 1 --2- 172.21.15.116:0/2885711160 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6080a29a0 0x7fb6080a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb610009fd0 tx=0x7fb610009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:36.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.428+0000 7fb61d7fa700 1 -- 172.21.15.116:0/2885711160 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb61409bce0 con 0x7fb620101bb0 2024-03-19T01:11:36.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:36 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1447341562' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T01:11:36.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:36 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1447341562' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-19T01:11:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.684+0000 7fb6266bb700 1 -- 172.21.15.116:0/2885711160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb62002dbb0 con 0x7fb620101bb0 2024-03-19T01:11:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.685+0000 7fb61d7fa700 1 -- 172.21.15.116:0/2885711160 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb61409b5e0 con 0x7fb620101bb0 2024-03-19T01:11:36.686 INFO:teuthology.orchestra.run.smithi116.stdout:55834574924 2024-03-19T01:11:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.687+0000 7fb6266bb700 1 -- 172.21.15.116:0/2885711160 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6080a29a0 msgr2=0x7fb6080a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.687+0000 7fb6266bb700 1 --2- 172.21.15.116:0/2885711160 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6080a29a0 0x7fb6080a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb610009fd0 tx=0x7fb610009260 comp rx=0 tx=0).stop 2024-03-19T01:11:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.687+0000 7fb6266bb700 1 -- 172.21.15.116:0/2885711160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb620101bb0 msgr2=0x7fb62019ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.687+0000 7fb6266bb700 1 --2- 172.21.15.116:0/2885711160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb620101bb0 0x7fb62019ec10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb61400e9a0 tx=0x7fb61400ecb0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.687+0000 7fb6266bb700 1 -- 172.21.15.116:0/2885711160 shutdown_connections 2024-03-19T01:11:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.687+0000 7fb6266bb700 1 --2- 172.21.15.116:0/2885711160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb620107250 0x7fb6201988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.687+0000 7fb6266bb700 1 --2- 172.21.15.116:0/2885711160 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6080a29a0 0x7fb6080a4e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.687+0000 7fb6266bb700 1 --2- 172.21.15.116:0/2885711160 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb620101210 0x7fb62019e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.687+0000 7fb6266bb700 1 --2- 172.21.15.116:0/2885711160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb620101bb0 0x7fb62019ec10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.687+0000 7fb6266bb700 1 -- 172.21.15.116:0/2885711160 >> 172.21.15.116:0/2885711160 conn(0x7fb620078140 msgr2=0x7fb6200ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.687+0000 7fb6266bb700 1 -- 172.21.15.116:0/2885711160 shutdown_connections 2024-03-19T01:11:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:36.687+0000 7fb6266bb700 1 -- 172.21.15.116:0/2885711160 wait complete. 2024-03-19T01:11:37.293 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:37 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2885711160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T01:11:37.294 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:37 smithi116 ceph-mon[62701]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4 op/s; 129 B/s, 1 keys/s, 1 objects/s recovering 2024-03-19T01:11:37.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:37 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2885711160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T01:11:37.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:37 smithi156 ceph-mon[66799]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4 op/s; 129 B/s, 1 keys/s, 1 objects/s recovering 2024-03-19T01:11:37.603 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411396 got 77309411400 for osd.2 2024-03-19T01:11:37.603 DEBUG:teuthology.parallel:result is None 2024-03-19T01:11:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:37 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2885711160' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-19T01:11:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:37 smithi162 ceph-mon[67178]: pgmap v322: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 4 op/s; 129 B/s, 1 keys/s, 1 objects/s recovering 2024-03-19T01:11:37.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.670+0000 7fb045b29700 1 -- 172.21.15.116:0/114543658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0401072a0 msgr2=0x7fb040107680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:37.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.670+0000 7fb045b29700 1 --2- 172.21.15.116:0/114543658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0401072a0 0x7fb040107680 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb030009990 tx=0x7fb030009ca0 comp rx=0 tx=0).stop 2024-03-19T01:11:37.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.671+0000 7fb045b29700 1 -- 172.21.15.116:0/114543658 shutdown_connections 2024-03-19T01:11:37.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.671+0000 7fb045b29700 1 --2- 172.21.15.116:0/114543658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb040101bb0 0x7fb04010e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:37.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.671+0000 7fb045b29700 1 --2- 172.21.15.116:0/114543658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0401072a0 0x7fb040107680 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:37.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.671+0000 7fb045b29700 1 --2- 172.21.15.116:0/114543658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb0401011f0 0x7fb040101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:37.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.671+0000 7fb045b29700 1 -- 172.21.15.116:0/114543658 >> 172.21.15.116:0/114543658 conn(0x7fb040078220 msgr2=0x7fb040078630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:37.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.671+0000 7fb045b29700 1 -- 172.21.15.116:0/114543658 shutdown_connections 2024-03-19T01:11:37.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.671+0000 7fb045b29700 1 -- 172.21.15.116:0/114543658 wait complete. 2024-03-19T01:11:37.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.671+0000 7fb045b29700 1 Processor -- start 2024-03-19T01:11:37.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.671+0000 7fb045b29700 1 -- start start 2024-03-19T01:11:37.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb045b29700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0401011f0 0x7fb0401998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb045b29700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb040101bb0 0x7fb040199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb045b29700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0401072a0 0x7fb0401a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb045b29700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb0401114e0 con 0x7fb0401072a0 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb045b29700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb040111360 con 0x7fb040101bb0 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb045b29700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb040111660 con 0x7fb0401011f0 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0401072a0 0x7fb0401a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03ffff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0401072a0 0x7fb0401a4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36886/0 (socket says 172.21.15.116:36886) 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03ffff700 1 -- 172.21.15.116:0/4097553309 learned_addr learned my addr 172.21.15.116:0/4097553309 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03f7fe700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0401011f0 0x7fb0401998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03effd700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb040101bb0 0x7fb040199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03ffff700 1 -- 172.21.15.116:0/4097553309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0401011f0 msgr2=0x7fb0401998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03ffff700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0401011f0 0x7fb0401998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03ffff700 1 -- 172.21.15.116:0/4097553309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb040101bb0 msgr2=0x7fb040199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:37.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03ffff700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb040101bb0 0x7fb040199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:37.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03ffff700 1 -- 172.21.15.116:0/4097553309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0401a4e80 con 0x7fb0401072a0 2024-03-19T01:11:37.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03ffff700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0401072a0 0x7fb0401a4940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb03800ef30 tx=0x7fb03800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:37.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03cff9700 1 -- 172.21.15.116:0/4097553309 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0380092e0 con 0x7fb0401072a0 2024-03-19T01:11:37.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb045b29700 1 -- 172.21.15.116:0/4097553309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0401a5050 con 0x7fb0401072a0 2024-03-19T01:11:37.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03cff9700 1 -- 172.21.15.116:0/4097553309 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb038004bb0 con 0x7fb0401072a0 2024-03-19T01:11:37.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb03cff9700 1 -- 172.21.15.116:0/4097553309 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb038004280 con 0x7fb0401072a0 2024-03-19T01:11:37.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.672+0000 7fb045b29700 1 -- 172.21.15.116:0/4097553309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0401a5560 con 0x7fb0401072a0 2024-03-19T01:11:37.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.673+0000 7fb045b29700 1 -- 172.21.15.116:0/4097553309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb040068fd0 con 0x7fb0401072a0 2024-03-19T01:11:37.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.674+0000 7fb03cff9700 1 -- 172.21.15.116:0/4097553309 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7fb03801b030 con 0x7fb0401072a0 2024-03-19T01:11:37.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.675+0000 7fb03cff9700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb0280a2920 0x7fb0280a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:37.799 INFO:teuthology.orchestra.run.smithi116.stdout:107374182466 2024-03-19T01:11:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.675+0000 7fb03f7fe700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb0280a2920 0x7fb0280a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.675+0000 7fb03cff9700 1 -- 172.21.15.116:0/4097553309 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb0380ce660 con 0x7fb0401072a0 2024-03-19T01:11:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.676+0000 7fb03f7fe700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb0280a2920 0x7fb0280a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb03000b2e0 tx=0x7fb030005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.681+0000 7fb03cff9700 1 -- 172.21.15.116:0/4097553309 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb038022020 con 0x7fb0401072a0 2024-03-19T01:11:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.892+0000 7fb045b29700 1 -- 172.21.15.116:0/4097553309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb04002dbb0 con 0x7fb0401072a0 2024-03-19T01:11:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.892+0000 7fb03cff9700 1 -- 172.21.15.116:0/4097553309 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb038010040 con 0x7fb0401072a0 2024-03-19T01:11:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.894+0000 7fb045b29700 1 -- 172.21.15.116:0/4097553309 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb0280a2920 msgr2=0x7fb0280a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.894+0000 7fb045b29700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb0280a2920 0x7fb0280a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb03000b2e0 tx=0x7fb030005960 comp rx=0 tx=0).stop 2024-03-19T01:11:37.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.894+0000 7fb045b29700 1 -- 172.21.15.116:0/4097553309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0401072a0 msgr2=0x7fb0401a4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.894+0000 7fb045b29700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0401072a0 0x7fb0401a4940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb03800ef30 tx=0x7fb03800c3a0 comp rx=0 tx=0).stop 2024-03-19T01:11:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.894+0000 7fb045b29700 1 -- 172.21.15.116:0/4097553309 shutdown_connections 2024-03-19T01:11:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.894+0000 7fb045b29700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0401072a0 0x7fb0401a4940 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.894+0000 7fb045b29700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb0280a2920 0x7fb0280a4de0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.894+0000 7fb045b29700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb0401011f0 0x7fb0401998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.894+0000 7fb045b29700 1 --2- 172.21.15.116:0/4097553309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb040101bb0 0x7fb040199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.894+0000 7fb045b29700 1 -- 172.21.15.116:0/4097553309 >> 172.21.15.116:0/4097553309 conn(0x7fb040078220 msgr2=0x7fb040100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.895+0000 7fb045b29700 1 -- 172.21.15.116:0/4097553309 shutdown_connections 2024-03-19T01:11:37.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:29.895+0000 7fb045b29700 1 -- 172.21.15.116:0/4097553309 wait complete. 2024-03-19T01:11:38.003 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:38.003 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:38.420 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182460 got 107374182466 for osd.3 2024-03-19T01:11:38.420 DEBUG:teuthology.parallel:result is None 2024-03-19T01:11:38.835 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738444 got 34359738448 for osd.0 2024-03-19T01:11:38.835 DEBUG:teuthology.parallel:result is None 2024-03-19T01:11:40.269 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:40 smithi116 ceph-mon[62701]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 5 op/s; 117 B/s, 1 keys/s, 1 objects/s recovering 2024-03-19T01:11:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:40 smithi162 ceph-mon[67178]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 5 op/s; 117 B/s, 1 keys/s, 1 objects/s recovering 2024-03-19T01:11:40.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:40 smithi156 ceph-mon[66799]: pgmap v323: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 5 op/s; 117 B/s, 1 keys/s, 1 objects/s recovering 2024-03-19T01:11:40.728 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574920 got 55834574924 for osd.1 2024-03-19T01:11:40.728 DEBUG:teuthology.parallel:result is None 2024-03-19T01:11:42.388 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:42 smithi116 ceph-mon[62701]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s; 9 B/s, 1 keys/s, 1 objects/s recovering 2024-03-19T01:11:42.389 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645339 got 309237645342 for osd.10 2024-03-19T01:11:42.389 DEBUG:teuthology.parallel:result is None 2024-03-19T01:11:42.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:42 smithi162 ceph-mon[67178]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s; 9 B/s, 1 keys/s, 1 objects/s recovering 2024-03-19T01:11:42.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:42 smithi156 ceph-mon[66799]: pgmap v324: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s; 9 B/s, 1 keys/s, 1 objects/s recovering 2024-03-19T01:11:42.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.836+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/399740987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bbc06a960 msgr2=0x7f7bbc10f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:42.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.836+0000 7f7bc0bc8700 1 --2- 172.21.15.116:0/399740987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bbc06a960 0x7f7bbc10f1a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7bb400b0d0 tx=0x7f7bb400b3e0 comp rx=0 tx=0).stop 2024-03-19T01:11:42.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.836+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/399740987 shutdown_connections 2024-03-19T01:11:42.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.836+0000 7f7bc0bc8700 1 --2- 172.21.15.116:0/399740987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bbc06a960 0x7f7bbc10f1a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:42.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.836+0000 7f7bc0bc8700 1 --2- 172.21.15.116:0/399740987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bbc108360 0x7f7bbc108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:42.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.836+0000 7f7bc0bc8700 1 --2- 172.21.15.116:0/399740987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7bbc069fc0 0x7f7bbc06a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:42.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.836+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/399740987 >> 172.21.15.116:0/399740987 conn(0x7f7bbc078140 msgr2=0x7f7bbc078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:42.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.836+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/399740987 shutdown_connections 2024-03-19T01:11:42.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.836+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/399740987 wait complete. 2024-03-19T01:11:42.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bc0bc8700 1 Processor -- start 2024-03-19T01:11:42.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bc0bc8700 1 -- start start 2024-03-19T01:11:42.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bc0bc8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bbc069fc0 0x7f7bbc19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:42.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bc0bc8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bbc06a960 0x7f7bbc19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:42.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bc0bc8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7bbc108360 0x7f7bbc198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:42.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bc0bc8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1125e0 con 0x7f7bbc069fc0 2024-03-19T01:11:42.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bc0bc8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc112460 con 0x7f7bbc108360 2024-03-19T01:11:42.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bc0bc8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc112760 con 0x7f7bbc06a960 2024-03-19T01:11:42.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bba59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bbc069fc0 0x7f7bbc19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:42.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bba59c700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bbc069fc0 0x7f7bbc19e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47378/0 (socket says 172.21.15.116:47378) 2024-03-19T01:11:42.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bba59c700 1 -- 172.21.15.116:0/1499418362 learned_addr learned my addr 172.21.15.116:0/1499418362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:42.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bb9d9b700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bbc06a960 0x7f7bbc19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:42.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bbad9d700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7bbc108360 0x7f7bbc198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:42.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bba59c700 1 -- 172.21.15.116:0/1499418362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bbc06a960 msgr2=0x7f7bbc19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:42.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bba59c700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bbc06a960 0x7f7bbc19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:42.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bba59c700 1 -- 172.21.15.116:0/1499418362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7bbc108360 msgr2=0x7f7bbc198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:42.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bba59c700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7bbc108360 0x7f7bbc198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:42.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bba59c700 1 -- 172.21.15.116:0/1499418362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bbc198fc0 con 0x7f7bbc069fc0 2024-03-19T01:11:42.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bb9d9b700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bbc06a960 0x7f7bbc19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:42.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.837+0000 7f7bba59c700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bbc069fc0 0x7f7bbc19e6a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00e9c0 tx=0x7f7bac00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:42.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.838+0000 7f7bab7fe700 1 -- 172.21.15.116:0/1499418362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac00cbd0 con 0x7f7bbc069fc0 2024-03-19T01:11:42.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.838+0000 7f7bab7fe700 1 -- 172.21.15.116:0/1499418362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bac00ee60 con 0x7f7bbc069fc0 2024-03-19T01:11:42.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.838+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/1499418362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bbc1992b0 con 0x7f7bbc069fc0 2024-03-19T01:11:42.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.838+0000 7f7bab7fe700 1 -- 172.21.15.116:0/1499418362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac005630 con 0x7f7bbc069fc0 2024-03-19T01:11:42.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.838+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/1499418362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bbc1a5570 con 0x7f7bbc069fc0 2024-03-19T01:11:42.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.839+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/1499418362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bbc068fd0 con 0x7f7bbc069fc0 2024-03-19T01:11:42.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.840+0000 7f7bab7fe700 1 -- 172.21.15.116:0/1499418362 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f7bac004750 con 0x7f7bbc069fc0 2024-03-19T01:11:42.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.841+0000 7f7bab7fe700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ba40a2920 0x7f7ba40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:42.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.841+0000 7f7bb9d9b700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ba40a2920 0x7f7ba40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:42.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.841+0000 7f7bab7fe700 1 -- 172.21.15.116:0/1499418362 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7bac0cf040 con 0x7f7bbc069fc0 2024-03-19T01:11:42.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.841+0000 7f7bb9d9b700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ba40a2920 0x7f7ba40a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc078720 tx=0x7f7bb0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:42.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:42.844+0000 7f7bab7fe700 1 -- 172.21.15.116:0/1499418362 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7bac053740 con 0x7f7bbc069fc0 2024-03-19T01:11:43.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.048+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/1499418362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f7bbc02dbb0 con 0x7f7bbc069fc0 2024-03-19T01:11:43.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.048+0000 7f7bab7fe700 1 -- 172.21.15.116:0/1499418362 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f7bac09bf60 con 0x7f7bbc069fc0 2024-03-19T01:11:43.049 INFO:teuthology.orchestra.run.smithi116.stdout:343597383707 2024-03-19T01:11:43.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.050+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/1499418362 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ba40a2920 msgr2=0x7f7ba40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:43.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.050+0000 7f7bc0bc8700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ba40a2920 0x7f7ba40a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc078720 tx=0x7f7bb0009260 comp rx=0 tx=0).stop 2024-03-19T01:11:43.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.050+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/1499418362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bbc069fc0 msgr2=0x7f7bbc19e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:43.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.050+0000 7f7bc0bc8700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bbc069fc0 0x7f7bbc19e6a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7bac00e9c0 tx=0x7f7bac00ecd0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.050+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/1499418362 shutdown_connections 2024-03-19T01:11:43.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.050+0000 7f7bc0bc8700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7bbc108360 0x7f7bbc198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.050+0000 7f7bc0bc8700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7bbc069fc0 0x7f7bbc19e6a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.050+0000 7f7bc0bc8700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ba40a2920 0x7f7ba40a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.050+0000 7f7bc0bc8700 1 --2- 172.21.15.116:0/1499418362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7bbc06a960 0x7f7bbc19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.050+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/1499418362 >> 172.21.15.116:0/1499418362 conn(0x7f7bbc078140 msgr2=0x7f7bbc0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:43.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.051+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/1499418362 shutdown_connections 2024-03-19T01:11:43.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.051+0000 7f7bc0bc8700 1 -- 172.21.15.116:0/1499418362 wait complete. 2024-03-19T01:11:43.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.060+0000 7f3daaf43700 1 -- 172.21.15.116:0/1076999299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da406b460 msgr2=0x7f3da4106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:43.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.060+0000 7f3daaf43700 1 --2- 172.21.15.116:0/1076999299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da406b460 0x7f3da4106f60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3d900099c0 tx=0x7f3d90009cd0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.061+0000 7f3daaf43700 1 -- 172.21.15.116:0/1076999299 shutdown_connections 2024-03-19T01:11:43.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.061+0000 7f3daaf43700 1 --2- 172.21.15.116:0/1076999299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da4107690 0x7f3da410f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.061+0000 7f3daaf43700 1 --2- 172.21.15.116:0/1076999299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da406aab0 0x7f3da406ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.061+0000 7f3daaf43700 1 --2- 172.21.15.116:0/1076999299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da406b460 0x7f3da4106f60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.061+0000 7f3daaf43700 1 -- 172.21.15.116:0/1076999299 >> 172.21.15.116:0/1076999299 conn(0x7f3da4077260 msgr2=0x7f3da4077670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:43.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.061+0000 7f3daaf43700 1 -- 172.21.15.116:0/1076999299 shutdown_connections 2024-03-19T01:11:43.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.061+0000 7f3daaf43700 1 -- 172.21.15.116:0/1076999299 wait complete. 2024-03-19T01:11:43.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.061+0000 7f3daaf43700 1 Processor -- start 2024-03-19T01:11:43.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3daaf43700 1 -- start start 2024-03-19T01:11:43.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3daaf43700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da406aab0 0x7f3da4101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:43.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3daaf43700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da406b460 0x7f3da4101850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:43.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3daaf43700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da4107690 0x7f3da4101d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:43.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3daaf43700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3da4112600 con 0x7f3da406aab0 2024-03-19T01:11:43.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3daaf43700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3da4112480 con 0x7f3da406b460 2024-03-19T01:11:43.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3daaf43700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3da4112780 con 0x7f3da4107690 2024-03-19T01:11:43.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3da3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da406b460 0x7f3da4101850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:43.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3da8cdf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da406aab0 0x7f3da4101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:43.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3da3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da406b460 0x7f3da4101850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:57026/0 (socket says 172.21.15.116:57026) 2024-03-19T01:11:43.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3da3fff700 1 -- 172.21.15.116:0/2271831458 learned_addr learned my addr 172.21.15.116:0/2271831458 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:43.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3da94e0700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da4107690 0x7f3da4101d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:43.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3da3fff700 1 -- 172.21.15.116:0/2271831458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da4107690 msgr2=0x7f3da4101d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:43.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3da3fff700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da4107690 0x7f3da4101d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3da3fff700 1 -- 172.21.15.116:0/2271831458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da406aab0 msgr2=0x7f3da4101310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:43.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3da3fff700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da406aab0 0x7f3da4101310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3da3fff700 1 -- 172.21.15.116:0/2271831458 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da41a7020 con 0x7f3da406b460 2024-03-19T01:11:43.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3da94e0700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da4107690 0x7f3da4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:43.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.062+0000 7f3da8cdf700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da406aab0 0x7f3da4101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:43.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.063+0000 7f3da3fff700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da406b460 0x7f3da4101850 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3d900098e0 tx=0x7f3d90005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:43.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.063+0000 7f3da1ffb700 1 -- 172.21.15.116:0/2271831458 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d90027070 con 0x7f3da406b460 2024-03-19T01:11:43.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.063+0000 7f3da1ffb700 1 -- 172.21.15.116:0/2271831458 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d90003680 con 0x7f3da406b460 2024-03-19T01:11:43.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.063+0000 7f3da1ffb700 1 -- 172.21.15.116:0/2271831458 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d90022e40 con 0x7f3da406b460 2024-03-19T01:11:43.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.063+0000 7f3daaf43700 1 -- 172.21.15.116:0/2271831458 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3da41a72b0 con 0x7f3da406b460 2024-03-19T01:11:43.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.063+0000 7f3daaf43700 1 -- 172.21.15.116:0/2271831458 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3da41a7760 con 0x7f3da406b460 2024-03-19T01:11:43.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.064+0000 7f3daaf43700 1 -- 172.21.15.116:0/2271831458 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3da4068fd0 con 0x7f3da406b460 2024-03-19T01:11:43.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.065+0000 7f3da1ffb700 1 -- 172.21.15.116:0/2271831458 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f3d9002a070 con 0x7f3da406b460 2024-03-19T01:11:43.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.066+0000 7f3da1ffb700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3d940a29a0 0x7f3d940a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:43.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.066+0000 7f3da8cdf700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3d940a29a0 0x7f3d940a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:43.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.066+0000 7f3da1ffb700 1 -- 172.21.15.116:0/2271831458 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f3d900cf5a0 con 0x7f3da406b460 2024-03-19T01:11:43.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.066+0000 7f3da8cdf700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3d940a29a0 0x7f3d940a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3d98011490 tx=0x7f3d98009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:43.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.074+0000 7f3da1ffb700 1 -- 172.21.15.116:0/2271831458 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3d90013070 con 0x7f3da406b460 2024-03-19T01:11:43.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.281+0000 7f3daaf43700 1 -- 172.21.15.116:0/2271831458 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f3da402dbb0 con 0x7f3da406b460 2024-03-19T01:11:43.283 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:43 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1499418362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-19T01:11:43.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.282+0000 7f3da1ffb700 1 -- 172.21.15.116:0/2271831458 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3d9009c4f0 con 0x7f3da406b460 2024-03-19T01:11:43.283 INFO:teuthology.orchestra.run.smithi116.stdout:253403070507 2024-03-19T01:11:43.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.284+0000 7f3daaf43700 1 -- 172.21.15.116:0/2271831458 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3d940a29a0 msgr2=0x7f3d940a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:43.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.284+0000 7f3daaf43700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3d940a29a0 0x7f3d940a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3d98011490 tx=0x7f3d98009260 comp rx=0 tx=0).stop 2024-03-19T01:11:43.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.284+0000 7f3daaf43700 1 -- 172.21.15.116:0/2271831458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da406b460 msgr2=0x7f3da4101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:43.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.284+0000 7f3daaf43700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da406b460 0x7f3da4101850 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3d900098e0 tx=0x7f3d90005f10 comp rx=0 tx=0).stop 2024-03-19T01:11:43.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.284+0000 7f3daaf43700 1 -- 172.21.15.116:0/2271831458 shutdown_connections 2024-03-19T01:11:43.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.284+0000 7f3daaf43700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3d940a29a0 0x7f3d940a4e60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.284+0000 7f3daaf43700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3da4107690 0x7f3da4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.284+0000 7f3daaf43700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3da406aab0 0x7f3da4101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.284+0000 7f3daaf43700 1 --2- 172.21.15.116:0/2271831458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3da406b460 0x7f3da4101850 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:43.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.284+0000 7f3daaf43700 1 -- 172.21.15.116:0/2271831458 >> 172.21.15.116:0/2271831458 conn(0x7f3da4077260 msgr2=0x7f3da40ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:43.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.284+0000 7f3daaf43700 1 -- 172.21.15.116:0/2271831458 shutdown_connections 2024-03-19T01:11:43.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:43.284+0000 7f3daaf43700 1 -- 172.21.15.116:0/2271831458 wait complete. 2024-03-19T01:11:43.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:43 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1499418362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-19T01:11:43.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:43 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1499418362' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-19T01:11:43.951 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070501 got 253403070507 for osd.8 2024-03-19T01:11:43.951 DEBUG:teuthology.parallel:result is None 2024-03-19T01:11:44.179 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:44 smithi116 ceph-mon[62701]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-19T01:11:44.179 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:44 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2271831458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-19T01:11:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:44 smithi162 ceph-mon[67178]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-19T01:11:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:44 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2271831458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-19T01:11:44.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:44 smithi156 ceph-mon[66799]: pgmap v325: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-19T01:11:44.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:44 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2271831458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-19T01:11:45.416 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383700 got 343597383707 for osd.11 2024-03-19T01:11:45.417 DEBUG:teuthology.parallel:result is None 2024-03-19T01:11:45.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.917+0000 7f713fe1b700 1 -- 172.21.15.116:0/4294609191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7138101bb0 msgr2=0x7f713810e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:45.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.917+0000 7f713fe1b700 1 --2- 172.21.15.116:0/4294609191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7138101bb0 0x7f713810e090 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f713000b0a0 tx=0x7f713000b3b0 comp rx=0 tx=0).stop 2024-03-19T01:11:45.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.917+0000 7f713fe1b700 1 -- 172.21.15.116:0/4294609191 shutdown_connections 2024-03-19T01:11:45.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.917+0000 7f713fe1b700 1 --2- 172.21.15.116:0/4294609191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7138101bb0 0x7f713810e090 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:45.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.917+0000 7f713fe1b700 1 --2- 172.21.15.116:0/4294609191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7138107250 0x7f7138107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:45.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.917+0000 7f713fe1b700 1 --2- 172.21.15.116:0/4294609191 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7138101210 0x7f7138101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:45.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.917+0000 7f713fe1b700 1 -- 172.21.15.116:0/4294609191 >> 172.21.15.116:0/4294609191 conn(0x7f7138078140 msgr2=0x7f7138078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:45.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.917+0000 7f713fe1b700 1 -- 172.21.15.116:0/4294609191 shutdown_connections 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.917+0000 7f713fe1b700 1 -- 172.21.15.116:0/4294609191 wait complete. 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713fe1b700 1 Processor -- start 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713fe1b700 1 -- start start 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713fe1b700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7138101210 0x7f7138199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713fe1b700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7138101bb0 0x7f7138199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713fe1b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7138107250 0x7f71381a4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713fe1b700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f71381114d0 con 0x7f7138101210 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713fe1b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7138111350 con 0x7f7138107250 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713fe1b700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7138111650 con 0x7f7138101bb0 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713dbb7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7138101210 0x7f7138199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713dbb7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7138101210 0x7f7138199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47408/0 (socket says 172.21.15.116:47408) 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713dbb7700 1 -- 172.21.15.116:0/3102866287 learned_addr learned my addr 172.21.15.116:0/3102866287 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713e3b8700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7138107250 0x7f71381a4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:45.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713d3b6700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7138101bb0 0x7f7138199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:45.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713dbb7700 1 -- 172.21.15.116:0/3102866287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7138101bb0 msgr2=0x7f7138199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:45.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713dbb7700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7138101bb0 0x7f7138199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:45.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713dbb7700 1 -- 172.21.15.116:0/3102866287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7138107250 msgr2=0x7f71381a4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:45.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713dbb7700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7138107250 0x7f71381a4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:45.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713dbb7700 1 -- 172.21.15.116:0/3102866287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71381a4e50 con 0x7f7138101210 2024-03-19T01:11:45.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.918+0000 7f713d3b6700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7138101bb0 0x7f7138199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:11:45.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.919+0000 7f713dbb7700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7138101210 0x7f7138199890 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f713400e9a0 tx=0x7f713400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:45.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.919+0000 7f712effd700 1 -- 172.21.15.116:0/3102866287 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713400cbd0 con 0x7f7138101210 2024-03-19T01:11:45.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.919+0000 7f712effd700 1 -- 172.21.15.116:0/3102866287 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f713400cd70 con 0x7f7138101210 2024-03-19T01:11:45.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.919+0000 7f713fe1b700 1 -- 172.21.15.116:0/3102866287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71381a50e0 con 0x7f7138101210 2024-03-19T01:11:45.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.919+0000 7f712effd700 1 -- 172.21.15.116:0/3102866287 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7134018750 con 0x7f7138101210 2024-03-19T01:11:45.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.919+0000 7f713fe1b700 1 -- 172.21.15.116:0/3102866287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71381a5640 con 0x7f7138101210 2024-03-19T01:11:45.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.920+0000 7f713fe1b700 1 -- 172.21.15.116:0/3102866287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7138068fd0 con 0x7f7138101210 2024-03-19T01:11:45.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.921+0000 7f712effd700 1 -- 172.21.15.116:0/3102866287 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f7134010430 con 0x7f7138101210 2024-03-19T01:11:45.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.921+0000 7f712effd700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f71240a29a0 0x7f71240a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:45.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.922+0000 7f713d3b6700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f71240a29a0 0x7f71240a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:45.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.922+0000 7f712effd700 1 -- 172.21.15.116:0/3102866287 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f71340cf260 con 0x7f7138101210 2024-03-19T01:11:45.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.922+0000 7f713d3b6700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f71240a29a0 0x7f71240a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7138078740 tx=0x7f7128009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:45.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:45.929+0000 7f712effd700 1 -- 172.21.15.116:0/3102866287 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f713409c180 con 0x7f7138101210 2024-03-19T01:11:46.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.137+0000 7f713fe1b700 1 -- 172.21.15.116:0/3102866287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f713802dcd0 con 0x7f7138101210 2024-03-19T01:11:46.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.140+0000 7f712effd700 1 -- 172.21.15.116:0/3102866287 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f713409ba80 con 0x7f7138101210 2024-03-19T01:11:46.141 INFO:teuthology.orchestra.run.smithi116.stdout:287762808870 2024-03-19T01:11:46.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.143+0000 7f713fe1b700 1 -- 172.21.15.116:0/3102866287 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f71240a29a0 msgr2=0x7f71240a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:46.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.143+0000 7f713fe1b700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f71240a29a0 0x7f71240a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7138078740 tx=0x7f7128009260 comp rx=0 tx=0).stop 2024-03-19T01:11:46.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.143+0000 7f713fe1b700 1 -- 172.21.15.116:0/3102866287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7138101210 msgr2=0x7f7138199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:46.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.143+0000 7f713fe1b700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7138101210 0x7f7138199890 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f713400e9a0 tx=0x7f713400ecb0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.143+0000 7f713fe1b700 1 -- 172.21.15.116:0/3102866287 shutdown_connections 2024-03-19T01:11:46.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.143+0000 7f713fe1b700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7138107250 0x7f71381a4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.143+0000 7f713fe1b700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7138101210 0x7f7138199890 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.143+0000 7f713fe1b700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f71240a29a0 0x7f71240a4e60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.143+0000 7f713fe1b700 1 --2- 172.21.15.116:0/3102866287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7138101bb0 0x7f7138199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.143+0000 7f713fe1b700 1 -- 172.21.15.116:0/3102866287 >> 172.21.15.116:0/3102866287 conn(0x7f7138078140 msgr2=0x7f71380ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:46.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.143+0000 7f713fe1b700 1 -- 172.21.15.116:0/3102866287 shutdown_connections 2024-03-19T01:11:46.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.143+0000 7f713fe1b700 1 -- 172.21.15.116:0/3102866287 wait complete. 2024-03-19T01:11:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.269+0000 7f16cc512700 1 -- 172.21.15.116:0/2792486418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c4107340 msgr2=0x7f16c4107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.269+0000 7f16cc512700 1 --2- 172.21.15.116:0/2792486418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c4107340 0x7f16c4107720 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f16c00098f0 tx=0x7f16c0009c00 comp rx=0 tx=0).stop 2024-03-19T01:11:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.269+0000 7f16cc512700 1 -- 172.21.15.116:0/2792486418 shutdown_connections 2024-03-19T01:11:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.269+0000 7f16cc512700 1 --2- 172.21.15.116:0/2792486418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16c4101eb0 0x7f16c410e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.269+0000 7f16cc512700 1 --2- 172.21.15.116:0/2792486418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c4107340 0x7f16c4107720 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.269+0000 7f16cc512700 1 --2- 172.21.15.116:0/2792486418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16c41014f0 0x7f16c4101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.269+0000 7f16cc512700 1 -- 172.21.15.116:0/2792486418 >> 172.21.15.116:0/2792486418 conn(0x7f16c40780d0 msgr2=0x7f16c40784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.269+0000 7f16cc512700 1 -- 172.21.15.116:0/2792486418 shutdown_connections 2024-03-19T01:11:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.270+0000 7f16cc512700 1 -- 172.21.15.116:0/2792486418 wait complete. 2024-03-19T01:11:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.270+0000 7f16cc512700 1 Processor -- start 2024-03-19T01:11:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.270+0000 7f16cc512700 1 -- start start 2024-03-19T01:11:46.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16cc512700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16c41014f0 0x7f16c41998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:46.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16cc512700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c4101eb0 0x7f16c4199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:46.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16cc512700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16c4107340 0x7f16c41a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:46.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16cc512700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f16c4111520 con 0x7f16c41014f0 2024-03-19T01:11:46.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16cc512700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f16c41113a0 con 0x7f16c4107340 2024-03-19T01:11:46.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16cc512700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f16c41116a0 con 0x7f16c4101eb0 2024-03-19T01:11:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16ca2ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16c41014f0 0x7f16c41998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16ca2ae700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16c41014f0 0x7f16c41998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47426/0 (socket says 172.21.15.116:47426) 2024-03-19T01:11:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16ca2ae700 1 -- 172.21.15.116:0/2810856342 learned_addr learned my addr 172.21.15.116:0/2810856342 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16caaaf700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16c4107340 0x7f16c41a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16c9aad700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c4101eb0 0x7f16c4199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16ca2ae700 1 -- 172.21.15.116:0/2810856342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c4101eb0 msgr2=0x7f16c4199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16ca2ae700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c4101eb0 0x7f16c4199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16ca2ae700 1 -- 172.21.15.116:0/2810856342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16c4107340 msgr2=0x7f16c41a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16ca2ae700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16c4107340 0x7f16c41a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16ca2ae700 1 -- 172.21.15.116:0/2810856342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c41a4e80 con 0x7f16c41014f0 2024-03-19T01:11:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16ca2ae700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16c41014f0 0x7f16c41998c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009230 tx=0x7f16c000ff20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:46.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16bb7fe700 1 -- 172.21.15.116:0/2810856342 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0005490 con 0x7f16c41014f0 2024-03-19T01:11:46.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16cc512700 1 -- 172.21.15.116:0/2810856342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c41a5050 con 0x7f16c41014f0 2024-03-19T01:11:46.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16bb7fe700 1 -- 172.21.15.116:0/2810856342 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c0005630 con 0x7f16c41014f0 2024-03-19T01:11:46.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16bb7fe700 1 -- 172.21.15.116:0/2810856342 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c0003de0 con 0x7f16c41014f0 2024-03-19T01:11:46.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.271+0000 7f16cc512700 1 -- 172.21.15.116:0/2810856342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c41a5500 con 0x7f16c41014f0 2024-03-19T01:11:46.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.272+0000 7f16cc512700 1 -- 172.21.15.116:0/2810856342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c4068fd0 con 0x7f16c41014f0 2024-03-19T01:11:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.274+0000 7f16bb7fe700 1 -- 172.21.15.116:0/2810856342 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f16c000b560 con 0x7f16c41014f0 2024-03-19T01:11:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.275+0000 7f16bb7fe700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f16b00a29a0 0x7f16b00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.275+0000 7f16c9aad700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f16b00a29a0 0x7f16b00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.275+0000 7f16bb7fe700 1 -- 172.21.15.116:0/2810856342 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f16c00cf470 con 0x7f16c41014f0 2024-03-19T01:11:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.275+0000 7f16c9aad700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f16b00a29a0 0x7f16b00a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009750 tx=0x7f16b4009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:46.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.279+0000 7f16bb7fe700 1 -- 172.21.15.116:0/2810856342 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f16c0013030 con 0x7f16c41014f0 2024-03-19T01:11:46.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:46 smithi162 ceph-mon[67178]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-19T01:11:46.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:46 smithi116 ceph-mon[62701]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-19T01:11:46.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:46 smithi156 ceph-mon[66799]: pgmap v326: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-03-19T01:11:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.481+0000 7f16cc512700 1 -- 172.21.15.116:0/2810856342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f16c402dbb0 con 0x7f16c41014f0 2024-03-19T01:11:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.481+0000 7f16bb7fe700 1 -- 172.21.15.116:0/2810856342 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f16c00a0e20 con 0x7f16c41014f0 2024-03-19T01:11:46.482 INFO:teuthology.orchestra.run.smithi116.stdout:219043332144 2024-03-19T01:11:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.484+0000 7f16cc512700 1 -- 172.21.15.116:0/2810856342 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f16b00a29a0 msgr2=0x7f16b00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.484+0000 7f16cc512700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f16b00a29a0 0x7f16b00a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f16b4009750 tx=0x7f16b4009310 comp rx=0 tx=0).stop 2024-03-19T01:11:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.484+0000 7f16cc512700 1 -- 172.21.15.116:0/2810856342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16c41014f0 msgr2=0x7f16c41998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.484+0000 7f16cc512700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16c41014f0 0x7f16c41998c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009230 tx=0x7f16c000ff20 comp rx=0 tx=0).stop 2024-03-19T01:11:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.484+0000 7f16cc512700 1 -- 172.21.15.116:0/2810856342 shutdown_connections 2024-03-19T01:11:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.484+0000 7f16cc512700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f16c4107340 0x7f16c41a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.484+0000 7f16cc512700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16c41014f0 0x7f16c41998c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.484+0000 7f16cc512700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f16b00a29a0 0x7f16b00a4e60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.484+0000 7f16cc512700 1 --2- 172.21.15.116:0/2810856342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f16c4101eb0 0x7f16c4199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.484+0000 7f16cc512700 1 -- 172.21.15.116:0/2810856342 >> 172.21.15.116:0/2810856342 conn(0x7f16c40780d0 msgr2=0x7f16c4100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.484+0000 7f16cc512700 1 -- 172.21.15.116:0/2810856342 shutdown_connections 2024-03-19T01:11:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:46.484+0000 7f16cc512700 1 -- 172.21.15.116:0/2810856342 wait complete. 2024-03-19T01:11:47.005 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808864 got 287762808870 for osd.9 2024-03-19T01:11:47.005 DEBUG:teuthology.parallel:result is None 2024-03-19T01:11:47.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:47 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3102866287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-19T01:11:47.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:47 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2810856342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-19T01:11:47.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:47 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3102866287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-19T01:11:47.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:47 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2810856342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-19T01:11:47.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:47 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3102866287' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-19T01:11:47.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:47 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2810856342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-19T01:11:47.670 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332139 got 219043332144 for osd.7 2024-03-19T01:11:47.670 DEBUG:teuthology.parallel:result is None 2024-03-19T01:11:47.671 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-19T01:11:47.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-19T01:11:48.166 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:48 smithi116 ceph-mon[62701]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:11:48.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:48 smithi162 ceph-mon[67178]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:11:48.453 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:48.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:48 smithi156 ceph-mon[66799]: pgmap v327: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:11:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.170+0000 7f912ec95700 1 -- 172.21.15.116:0/4064642658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9128102d50 msgr2=0x7f912810f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.170+0000 7f912ec95700 1 --2- 172.21.15.116:0/4064642658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9128102d50 0x7f912810f1e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9110009990 tx=0x7f9110009ca0 comp rx=0 tx=0).stop 2024-03-19T01:11:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.170+0000 7f912ec95700 1 -- 172.21.15.116:0/4064642658 shutdown_connections 2024-03-19T01:11:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.170+0000 7f912ec95700 1 --2- 172.21.15.116:0/4064642658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9128102d50 0x7f912810f1e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.170+0000 7f912ec95700 1 --2- 172.21.15.116:0/4064642658 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91281083e0 0x7f91281087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:50.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.170+0000 7f912ec95700 1 --2- 172.21.15.116:0/4064642658 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9128102390 0x7f9128102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.170+0000 7f912ec95700 1 -- 172.21.15.116:0/4064642658 >> 172.21.15.116:0/4064642658 conn(0x7f9128079460 msgr2=0x7f9128079870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.170+0000 7f912ec95700 1 -- 172.21.15.116:0/4064642658 shutdown_connections 2024-03-19T01:11:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.170+0000 7f912ec95700 1 -- 172.21.15.116:0/4064642658 wait complete. 2024-03-19T01:11:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.171+0000 7f912ec95700 1 Processor -- start 2024-03-19T01:11:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.171+0000 7f912ec95700 1 -- start start 2024-03-19T01:11:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.171+0000 7f912ec95700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9128102390 0x7f912819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.171+0000 7f912ec95700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9128102d50 0x7f912819ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.171+0000 7f912ec95700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91281083e0 0x7f91281988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.171+0000 7f912ec95700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9128112620 con 0x7f91281083e0 2024-03-19T01:11:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.171+0000 7f912ec95700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f91281124a0 con 0x7f9128102d50 2024-03-19T01:11:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.171+0000 7f912ec95700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f91281127a0 con 0x7f9128102390 2024-03-19T01:11:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.171+0000 7f912ca31700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9128102390 0x7f912819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:50.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.171+0000 7f912ca31700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9128102390 0x7f912819e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:33966/0 (socket says 172.21.15.116:33966) 2024-03-19T01:11:50.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.171+0000 7f912d232700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91281083e0 0x7f91281988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:50.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.171+0000 7f912ca31700 1 -- 172.21.15.116:0/1802270488 learned_addr learned my addr 172.21.15.116:0/1802270488 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:50.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.171+0000 7f9127fff700 1 --2- 172.21.15.116:0/1802270488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9128102d50 0x7f912819ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:50.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.172+0000 7f912ca31700 1 -- 172.21.15.116:0/1802270488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9128102d50 msgr2=0x7f912819ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:50.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.172+0000 7f912ca31700 1 --2- 172.21.15.116:0/1802270488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9128102d50 0x7f912819ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:50.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.172+0000 7f912ca31700 1 -- 172.21.15.116:0/1802270488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91281083e0 msgr2=0x7f91281988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:50.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.172+0000 7f912ca31700 1 --2- 172.21.15.116:0/1802270488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91281083e0 0x7f91281988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:50.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.172+0000 7f912ca31700 1 -- 172.21.15.116:0/1802270488 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9128199020 con 0x7f9128102390 2024-03-19T01:11:50.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.172+0000 7f912ca31700 1 --2- 172.21.15.116:0/1802270488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9128102390 0x7f912819e6b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f911800e9a0 tx=0x7f911800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:50.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.172+0000 7f9125ffb700 1 -- 172.21.15.116:0/1802270488 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911800ca10 con 0x7f9128102390 2024-03-19T01:11:50.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.172+0000 7f912ec95700 1 -- 172.21.15.116:0/1802270488 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9128199310 con 0x7f9128102390 2024-03-19T01:11:50.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.172+0000 7f9125ffb700 1 -- 172.21.15.116:0/1802270488 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f911800cbb0 con 0x7f9128102390 2024-03-19T01:11:50.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.172+0000 7f9125ffb700 1 -- 172.21.15.116:0/1802270488 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9118018680 con 0x7f9128102390 2024-03-19T01:11:50.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.172+0000 7f912ec95700 1 -- 172.21.15.116:0/1802270488 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91281a5580 con 0x7f9128102390 2024-03-19T01:11:50.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.173+0000 7f912ec95700 1 -- 172.21.15.116:0/1802270488 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9128068fd0 con 0x7f9128102390 2024-03-19T01:11:50.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.174+0000 7f9125ffb700 1 -- 172.21.15.116:0/1802270488 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f9118010430 con 0x7f9128102390 2024-03-19T01:11:50.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.175+0000 7f9125ffb700 1 --2- 172.21.15.116:0/1802270488 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f91140a2920 0x7f91140a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:50.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.175+0000 7f9127fff700 1 --2- 172.21.15.116:0/1802270488 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f91140a2920 0x7f91140a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:50.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.175+0000 7f9125ffb700 1 -- 172.21.15.116:0/1802270488 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f91180cee30 con 0x7f9128102390 2024-03-19T01:11:50.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.175+0000 7f9127fff700 1 --2- 172.21.15.116:0/1802270488 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f91140a2920 0x7f91140a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9128079ad0 tx=0x7f912000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:50.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.184+0000 7f9125ffb700 1 -- 172.21.15.116:0/1802270488 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f911809bcf0 con 0x7f9128102390 2024-03-19T01:11:50.185 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:50 smithi116 ceph-mon[62701]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:11:50.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:50 smithi162 ceph-mon[67178]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:11:50.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.407+0000 7f912ec95700 1 -- 172.21.15.116:0/1802270488 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9128062750 con 0x7f91140a2920 2024-03-19T01:11:50.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.414+0000 7f9125ffb700 1 -- 172.21.15.116:0/1802270488 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259066 (secure 0 0 0) 0x7f9128062750 con 0x7f91140a2920 2024-03-19T01:11:50.419 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:11:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.417+0000 7f912ec95700 1 -- 172.21.15.116:0/1802270488 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f91140a2920 msgr2=0x7f91140a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.417+0000 7f912ec95700 1 --2- 172.21.15.116:0/1802270488 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f91140a2920 0x7f91140a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9128079ad0 tx=0x7f912000b040 comp rx=0 tx=0).stop 2024-03-19T01:11:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.417+0000 7f912ec95700 1 -- 172.21.15.116:0/1802270488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9128102390 msgr2=0x7f912819e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.417+0000 7f912ec95700 1 --2- 172.21.15.116:0/1802270488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9128102390 0x7f912819e6b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f911800e9a0 tx=0x7f911800ecb0 comp rx=0 tx=0).stop 2024-03-19T01:11:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.417+0000 7f912ec95700 1 -- 172.21.15.116:0/1802270488 shutdown_connections 2024-03-19T01:11:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.417+0000 7f912ec95700 1 --2- 172.21.15.116:0/1802270488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91281083e0 0x7f91281988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.417+0000 7f912ec95700 1 --2- 172.21.15.116:0/1802270488 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f91140a2920 0x7f91140a4de0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.417+0000 7f912ec95700 1 --2- 172.21.15.116:0/1802270488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9128102390 0x7f912819e6b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.417+0000 7f912ec95700 1 --2- 172.21.15.116:0/1802270488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9128102d50 0x7f912819ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.417+0000 7f912ec95700 1 -- 172.21.15.116:0/1802270488 >> 172.21.15.116:0/1802270488 conn(0x7f9128079460 msgr2=0x7f91281010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.417+0000 7f912ec95700 1 -- 172.21.15.116:0/1802270488 shutdown_connections 2024-03-19T01:11:50.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:50.417+0000 7f912ec95700 1 -- 172.21.15.116:0/1802270488 wait complete. 2024-03-19T01:11:50.423 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-19T01:11:50.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:50 smithi156 ceph-mon[66799]: pgmap v328: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:11:51.143 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-03-19T01:11:49.073851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658461,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":801,"num_read_kb":542,"num_write":421,"num_write_kb":3073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1182023,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381744,"kb_used_data":11700,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560080,"statfs":{"total":1151940427776,"available":1151549521920,"internally_reserved":0,"allocated":11980800,"data_stored":7445235,"data_compressed":183564,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002545"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.154655+0000","last_change":"2024-03-19T01:10:33.176468+0000","last_active":"2024-03-19T01:11:28.154655+0000","last_peered":"2024-03-19T01:11:28.154655+0000","last_clean":"2024-03-19T01:11:28.154655+0000","last_became_active":"2024-03-19T01:10:33.175995+0000","last_became_peered":"2024-03-19T01:10:33.175995+0000","last_unstale":"2024-03-19T01:11:28.154655+0000","last_undegraded":"2024-03-19T01:11:28.154655+0000","last_fullsized":"2024-03-19T01:11:28.154655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:40:02.320205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148022+0000","last_change":"2024-03-19T01:10:33.171253+0000","last_active":"2024-03-19T01:11:28.148022+0000","last_peered":"2024-03-19T01:11:28.148022+0000","last_clean":"2024-03-19T01:11:28.148022+0000","last_became_active":"2024-03-19T01:10:33.170330+0000","last_became_peered":"2024-03-19T01:10:33.170330+0000","last_unstale":"2024-03-19T01:11:28.148022+0000","last_undegraded":"2024-03-19T01:11:28.148022+0000","last_fullsized":"2024-03-19T01:11:28.148022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:38.905220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148298+0000","last_change":"2024-03-19T01:10:33.178006+0000","last_active":"2024-03-19T01:11:28.148298+0000","last_peered":"2024-03-19T01:11:28.148298+0000","last_clean":"2024-03-19T01:11:28.148298+0000","last_became_active":"2024-03-19T01:10:33.177662+0000","last_became_peered":"2024-03-19T01:10:33.177662+0000","last_unstale":"2024-03-19T01:11:28.148298+0000","last_undegraded":"2024-03-19T01:11:28.148298+0000","last_fullsized":"2024-03-19T01:11:28.148298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:08.980220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151084+0000","last_change":"2024-03-19T01:10:33.178805+0000","last_active":"2024-03-19T01:11:28.151084+0000","last_peered":"2024-03-19T01:11:28.151084+0000","last_clean":"2024-03-19T01:11:28.151084+0000","last_became_active":"2024-03-19T01:10:33.178385+0000","last_became_peered":"2024-03-19T01:10:33.178385+0000","last_unstale":"2024-03-19T01:11:28.151084+0000","last_undegraded":"2024-03-19T01:11:28.151084+0000","last_fullsized":"2024-03-19T01:11:28.151084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:06.454345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148248+0000","last_change":"2024-03-19T01:10:33.183110+0000","last_active":"2024-03-19T01:11:28.148248+0000","last_peered":"2024-03-19T01:11:28.148248+0000","last_clean":"2024-03-19T01:11:28.148248+0000","last_became_active":"2024-03-19T01:10:33.180036+0000","last_became_peered":"2024-03-19T01:10:33.180036+0000","last_unstale":"2024-03-19T01:11:28.148248+0000","last_undegraded":"2024-03-19T01:11:28.148248+0000","last_fullsized":"2024-03-19T01:11:28.148248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:57:51.397156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.154164+0000","last_change":"2024-03-19T01:11:28.154164+0000","last_active":"2024-03-19T01:11:28.154164+0000","last_peered":"2024-03-19T01:11:28.154164+0000","last_clean":"2024-03-19T01:11:28.154164+0000","last_became_active":"2024-03-19T01:11:28.153778+0000","last_became_peered":"2024-03-19T01:11:28.153778+0000","last_unstale":"2024-03-19T01:11:28.154164+0000","last_undegraded":"2024-03-19T01:11:28.154164+0000","last_fullsized":"2024-03-19T01:11:28.154164+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:50.083781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148162+0000","last_change":"2024-03-19T01:10:33.172946+0000","last_active":"2024-03-19T01:11:28.148162+0000","last_peered":"2024-03-19T01:11:28.148162+0000","last_clean":"2024-03-19T01:11:28.148162+0000","last_became_active":"2024-03-19T01:10:33.172617+0000","last_became_peered":"2024-03-19T01:10:33.172617+0000","last_unstale":"2024-03-19T01:11:28.148162+0000","last_undegraded":"2024-03-19T01:11:28.148162+0000","last_fullsized":"2024-03-19T01:11:28.148162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:18:20.311445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153044+0000","last_change":"2024-03-19T01:10:33.178162+0000","last_active":"2024-03-19T01:11:28.153044+0000","last_peered":"2024-03-19T01:11:28.153044+0000","last_clean":"2024-03-19T01:11:28.153044+0000","last_became_active":"2024-03-19T01:10:33.177752+0000","last_became_peered":"2024-03-19T01:10:33.177752+0000","last_unstale":"2024-03-19T01:11:28.153044+0000","last_undegraded":"2024-03-19T01:11:28.153044+0000","last_fullsized":"2024-03-19T01:11:28.153044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:01:35.681694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.216525+0000","last_change":"2024-03-19T01:10:33.175619+0000","last_active":"2024-03-19T01:11:28.216525+0000","last_peered":"2024-03-19T01:11:28.216525+0000","last_clean":"2024-03-19T01:11:28.216525+0000","last_became_active":"2024-03-19T01:10:33.175017+0000","last_became_peered":"2024-03-19T01:10:33.175017+0000","last_unstale":"2024-03-19T01:11:28.216525+0000","last_undegraded":"2024-03-19T01:11:28.216525+0000","last_fullsized":"2024-03-19T01:11:28.216525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:49:52.019461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151899+0000","last_change":"2024-03-19T01:10:33.173035+0000","last_active":"2024-03-19T01:11:28.151899+0000","last_peered":"2024-03-19T01:11:28.151899+0000","last_clean":"2024-03-19T01:11:28.151899+0000","last_became_active":"2024-03-19T01:10:33.172717+0000","last_became_peered":"2024-03-19T01:10:33.172717+0000","last_unstale":"2024-03-19T01:11:28.151899+0000","last_undegraded":"2024-03-19T01:11:28.151899+0000","last_fullsized":"2024-03-19T01:11:28.151899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:38.078147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148957+0000","last_change":"2024-03-19T01:10:33.182679+0000","last_active":"2024-03-19T01:11:28.148957+0000","last_peered":"2024-03-19T01:11:28.148957+0000","last_clean":"2024-03-19T01:11:28.148957+0000","last_became_active":"2024-03-19T01:10:33.182384+0000","last_became_peered":"2024-03-19T01:10:33.182384+0000","last_unstale":"2024-03-19T01:11:28.148957+0000","last_undegraded":"2024-03-19T01:11:28.148957+0000","last_fullsized":"2024-03-19T01:11:28.148957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:37:26.542393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150655+0000","last_change":"2024-03-19T01:10:33.178818+0000","last_active":"2024-03-19T01:11:28.150655+0000","last_peered":"2024-03-19T01:11:28.150655+0000","last_clean":"2024-03-19T01:11:28.150655+0000","last_became_active":"2024-03-19T01:10:33.178286+0000","last_became_peered":"2024-03-19T01:10:33.178286+0000","last_unstale":"2024-03-19T01:11:28.150655+0000","last_undegraded":"2024-03-19T01:11:28.150655+0000","last_fullsized":"2024-03-19T01:11:28.150655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:05:19.473738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.149581+0000","last_change":"2024-03-19T01:10:33.157042+0000","last_active":"2024-03-19T01:11:28.149581+0000","last_peered":"2024-03-19T01:11:28.149581+0000","last_clean":"2024-03-19T01:11:28.149581+0000","last_became_active":"2024-03-19T01:10:33.156576+0000","last_became_peered":"2024-03-19T01:10:33.156576+0000","last_unstale":"2024-03-19T01:11:28.149581+0000","last_undegraded":"2024-03-19T01:11:28.149581+0000","last_fullsized":"2024-03-19T01:11:28.149581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:51.625850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.147897+0000","last_change":"2024-03-19T01:10:33.177981+0000","last_active":"2024-03-19T01:11:28.147897+0000","last_peered":"2024-03-19T01:11:28.147897+0000","last_clean":"2024-03-19T01:11:28.147897+0000","last_became_active":"2024-03-19T01:10:33.177634+0000","last_became_peered":"2024-03-19T01:10:33.177634+0000","last_unstale":"2024-03-19T01:11:28.147897+0000","last_undegraded":"2024-03-19T01:11:28.147897+0000","last_fullsized":"2024-03-19T01:11:28.147897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:49.646938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.147763+0000","last_change":"2024-03-19T01:10:33.168982+0000","last_active":"2024-03-19T01:11:28.147763+0000","last_peered":"2024-03-19T01:11:28.147763+0000","last_clean":"2024-03-19T01:11:28.147763+0000","last_became_active":"2024-03-19T01:10:33.168569+0000","last_became_peered":"2024-03-19T01:10:33.168569+0000","last_unstale":"2024-03-19T01:11:28.147763+0000","last_undegraded":"2024-03-19T01:11:28.147763+0000","last_fullsized":"2024-03-19T01:11:28.147763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:43.389585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.215848+0000","last_change":"2024-03-19T01:10:33.184056+0000","last_active":"2024-03-19T01:11:28.215848+0000","last_peered":"2024-03-19T01:11:28.215848+0000","last_clean":"2024-03-19T01:11:28.215848+0000","last_became_active":"2024-03-19T01:10:33.183712+0000","last_became_peered":"2024-03-19T01:10:33.183712+0000","last_unstale":"2024-03-19T01:11:28.215848+0000","last_undegraded":"2024-03-19T01:11:28.215848+0000","last_fullsized":"2024-03-19T01:11:28.215848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:27.268859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.154753+0000","last_change":"2024-03-19T01:10:33.180114+0000","last_active":"2024-03-19T01:11:28.154753+0000","last_peered":"2024-03-19T01:11:28.154753+0000","last_clean":"2024-03-19T01:11:28.154753+0000","last_became_active":"2024-03-19T01:10:33.179796+0000","last_became_peered":"2024-03-19T01:10:33.179796+0000","last_unstale":"2024-03-19T01:11:28.154753+0000","last_undegraded":"2024-03-19T01:11:28.154753+0000","last_fullsized":"2024-03-19T01:11:28.154753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:55:23.009912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151331+0000","last_change":"2024-03-19T01:10:33.179888+0000","last_active":"2024-03-19T01:11:28.151331+0000","last_peered":"2024-03-19T01:11:28.151331+0000","last_clean":"2024-03-19T01:11:28.151331+0000","last_became_active":"2024-03-19T01:10:33.179562+0000","last_became_peered":"2024-03-19T01:10:33.179562+0000","last_unstale":"2024-03-19T01:11:28.151331+0000","last_undegraded":"2024-03-19T01:11:28.151331+0000","last_fullsized":"2024-03-19T01:11:28.151331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:53.346043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151054+0000","last_change":"2024-03-19T01:10:33.177619+0000","last_active":"2024-03-19T01:11:28.151054+0000","last_peered":"2024-03-19T01:11:28.151054+0000","last_clean":"2024-03-19T01:11:28.151054+0000","last_became_active":"2024-03-19T01:10:33.176002+0000","last_became_peered":"2024-03-19T01:10:33.176002+0000","last_unstale":"2024-03-19T01:11:28.151054+0000","last_undegraded":"2024-03-19T01:11:28.151054+0000","last_fullsized":"2024-03-19T01:11:28.151054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:50.637964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":102,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:46.066005+0000","last_change":"2024-03-19T01:10:32.151875+0000","last_active":"2024-03-19T01:11:46.066005+0000","last_peered":"2024-03-19T01:11:46.066005+0000","last_clean":"2024-03-19T01:11:46.066005+0000","last_became_active":"2024-03-19T01:10:32.151448+0000","last_became_peered":"2024-03-19T01:10:32.151448+0000","last_unstale":"2024-03-19T01:11:46.066005+0000","last_undegraded":"2024-03-19T01:11:46.066005+0000","last_fullsized":"2024-03-19T01:11:46.066005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:17.930078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.155370+0000","last_change":"2024-03-19T01:10:33.184258+0000","last_active":"2024-03-19T01:11:28.155370+0000","last_peered":"2024-03-19T01:11:28.155370+0000","last_clean":"2024-03-19T01:11:28.155370+0000","last_became_active":"2024-03-19T01:10:33.183899+0000","last_became_peered":"2024-03-19T01:10:33.183899+0000","last_unstale":"2024-03-19T01:11:28.155370+0000","last_undegraded":"2024-03-19T01:11:28.155370+0000","last_fullsized":"2024-03-19T01:11:28.155370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:47.220818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153044+0000","last_change":"2024-03-19T01:10:32.151998+0000","last_active":"2024-03-19T01:11:28.153044+0000","last_peered":"2024-03-19T01:11:28.153044+0000","last_clean":"2024-03-19T01:11:28.153044+0000","last_became_active":"2024-03-19T01:10:32.151644+0000","last_became_peered":"2024-03-19T01:10:32.151644+0000","last_unstale":"2024-03-19T01:11:28.153044+0000","last_undegraded":"2024-03-19T01:11:28.153044+0000","last_fullsized":"2024-03-19T01:11:28.153044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:20:28.660705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148188+0000","last_change":"2024-03-19T01:10:33.176759+0000","last_active":"2024-03-19T01:11:28.148188+0000","last_peered":"2024-03-19T01:11:28.148188+0000","last_clean":"2024-03-19T01:11:28.148188+0000","last_became_active":"2024-03-19T01:10:33.176409+0000","last_became_peered":"2024-03-19T01:10:33.176409+0000","last_unstale":"2024-03-19T01:11:28.148188+0000","last_undegraded":"2024-03-19T01:11:28.148188+0000","last_fullsized":"2024-03-19T01:11:28.148188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:21:08.513411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.166861+0000","last_change":"2024-03-19T01:11:28.166417+0000","last_active":"2024-03-19T01:11:28.166861+0000","last_peered":"2024-03-19T01:11:28.166861+0000","last_clean":"2024-03-19T01:11:28.166861+0000","last_became_active":"2024-03-19T01:11:28.155627+0000","last_became_peered":"2024-03-19T01:11:28.155627+0000","last_unstale":"2024-03-19T01:11:28.166861+0000","last_undegraded":"2024-03-19T01:11:28.166861+0000","last_fullsized":"2024-03-19T01:11:28.166861+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:14.701579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.216184+0000","last_change":"2024-03-19T01:10:33.174475+0000","last_active":"2024-03-19T01:11:28.216184+0000","last_peered":"2024-03-19T01:11:28.216184+0000","last_clean":"2024-03-19T01:11:28.216184+0000","last_became_active":"2024-03-19T01:10:33.174154+0000","last_became_peered":"2024-03-19T01:10:33.174154+0000","last_unstale":"2024-03-19T01:11:28.216184+0000","last_undegraded":"2024-03-19T01:11:28.216184+0000","last_fullsized":"2024-03-19T01:11:28.216184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:17.874609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148450+0000","last_change":"2024-03-19T01:10:32.151717+0000","last_active":"2024-03-19T01:11:28.148450+0000","last_peered":"2024-03-19T01:11:28.148450+0000","last_clean":"2024-03-19T01:11:28.148450+0000","last_became_active":"2024-03-19T01:10:32.151356+0000","last_became_peered":"2024-03-19T01:10:32.151356+0000","last_unstale":"2024-03-19T01:11:28.148450+0000","last_undegraded":"2024-03-19T01:11:28.148450+0000","last_fullsized":"2024-03-19T01:11:28.148450+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:02.614226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153048+0000","last_change":"2024-03-19T01:10:33.177202+0000","last_active":"2024-03-19T01:11:28.153048+0000","last_peered":"2024-03-19T01:11:28.153048+0000","last_clean":"2024-03-19T01:11:28.153048+0000","last_became_active":"2024-03-19T01:10:33.172004+0000","last_became_peered":"2024-03-19T01:10:33.172004+0000","last_unstale":"2024-03-19T01:11:28.153048+0000","last_undegraded":"2024-03-19T01:11:28.153048+0000","last_fullsized":"2024-03-19T01:11:28.153048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:11:36.610997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:47.062549+0000","last_change":"2024-03-19T01:10:32.151138+0000","last_active":"2024-03-19T01:11:47.062549+0000","last_peered":"2024-03-19T01:11:47.062549+0000","last_clean":"2024-03-19T01:11:47.062549+0000","last_became_active":"2024-03-19T01:10:32.150297+0000","last_became_peered":"2024-03-19T01:10:32.150297+0000","last_unstale":"2024-03-19T01:11:47.062549+0000","last_undegraded":"2024-03-19T01:11:47.062549+0000","last_fullsized":"2024-03-19T01:11:47.062549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:32:37.104986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148422+0000","last_change":"2024-03-19T01:10:33.178878+0000","last_active":"2024-03-19T01:11:28.148422+0000","last_peered":"2024-03-19T01:11:28.148422+0000","last_clean":"2024-03-19T01:11:28.148422+0000","last_became_active":"2024-03-19T01:10:33.178526+0000","last_became_peered":"2024-03-19T01:10:33.178526+0000","last_unstale":"2024-03-19T01:11:28.148422+0000","last_undegraded":"2024-03-19T01:11:28.148422+0000","last_fullsized":"2024-03-19T01:11:28.148422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:40.074065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151640+0000","last_change":"2024-03-19T01:10:32.153255+0000","last_active":"2024-03-19T01:11:28.151640+0000","last_peered":"2024-03-19T01:11:28.151640+0000","last_clean":"2024-03-19T01:11:28.151640+0000","last_became_active":"2024-03-19T01:10:32.152976+0000","last_became_peered":"2024-03-19T01:10:32.152976+0000","last_unstale":"2024-03-19T01:11:28.151640+0000","last_undegraded":"2024-03-19T01:11:28.151640+0000","last_fullsized":"2024-03-19T01:11:28.151640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:51:00.313688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.152041+0000","last_change":"2024-03-19T01:10:33.173710+0000","last_active":"2024-03-19T01:11:28.152041+0000","last_peered":"2024-03-19T01:11:28.152041+0000","last_clean":"2024-03-19T01:11:28.152041+0000","last_became_active":"2024-03-19T01:10:33.173325+0000","last_became_peered":"2024-03-19T01:10:33.173325+0000","last_unstale":"2024-03-19T01:11:28.152041+0000","last_undegraded":"2024-03-19T01:11:28.152041+0000","last_fullsized":"2024-03-19T01:11:28.152041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:21.755723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150859+0000","last_change":"2024-03-19T01:10:32.144286+0000","last_active":"2024-03-19T01:11:28.150859+0000","last_peered":"2024-03-19T01:11:28.150859+0000","last_clean":"2024-03-19T01:11:28.150859+0000","last_became_active":"2024-03-19T01:10:32.143959+0000","last_became_peered":"2024-03-19T01:10:32.143959+0000","last_unstale":"2024-03-19T01:11:28.150859+0000","last_undegraded":"2024-03-19T01:11:28.150859+0000","last_fullsized":"2024-03-19T01:11:28.150859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:46:15.750018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153267+0000","last_change":"2024-03-19T01:10:33.178122+0000","last_active":"2024-03-19T01:11:28.153267+0000","last_peered":"2024-03-19T01:11:28.153267+0000","last_clean":"2024-03-19T01:11:28.153267+0000","last_became_active":"2024-03-19T01:10:33.175491+0000","last_became_peered":"2024-03-19T01:10:33.175491+0000","last_unstale":"2024-03-19T01:11:28.153267+0000","last_undegraded":"2024-03-19T01:11:28.153267+0000","last_fullsized":"2024-03-19T01:11:28.153267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:34.418463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151674+0000","last_change":"2024-03-19T01:10:32.147561+0000","last_active":"2024-03-19T01:11:28.151674+0000","last_peered":"2024-03-19T01:11:28.151674+0000","last_clean":"2024-03-19T01:11:28.151674+0000","last_became_active":"2024-03-19T01:10:32.146825+0000","last_became_peered":"2024-03-19T01:10:32.146825+0000","last_unstale":"2024-03-19T01:11:28.151674+0000","last_undegraded":"2024-03-19T01:11:28.151674+0000","last_fullsized":"2024-03-19T01:11:28.151674+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:20:09.833960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150792+0000","last_change":"2024-03-19T01:10:33.170492+0000","last_active":"2024-03-19T01:11:28.150792+0000","last_peered":"2024-03-19T01:11:28.150792+0000","last_clean":"2024-03-19T01:11:28.150792+0000","last_became_active":"2024-03-19T01:10:33.170013+0000","last_became_peered":"2024-03-19T01:10:33.170013+0000","last_unstale":"2024-03-19T01:11:28.150792+0000","last_undegraded":"2024-03-19T01:11:28.150792+0000","last_fullsized":"2024-03-19T01:11:28.150792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:13.046663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":162,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:47.177766+0000","last_change":"2024-03-19T01:10:32.147060+0000","last_active":"2024-03-19T01:11:47.177766+0000","last_peered":"2024-03-19T01:11:47.177766+0000","last_clean":"2024-03-19T01:11:47.177766+0000","last_became_active":"2024-03-19T01:10:32.146721+0000","last_became_peered":"2024-03-19T01:10:32.146721+0000","last_unstale":"2024-03-19T01:11:47.177766+0000","last_undegraded":"2024-03-19T01:11:47.177766+0000","last_fullsized":"2024-03-19T01:11:47.177766+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:13:16.930746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":139,"num_read_kb":139,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.216405+0000","last_change":"2024-03-19T01:10:33.174431+0000","last_active":"2024-03-19T01:11:28.216405+0000","last_peered":"2024-03-19T01:11:28.216405+0000","last_clean":"2024-03-19T01:11:28.216405+0000","last_became_active":"2024-03-19T01:10:33.174040+0000","last_became_peered":"2024-03-19T01:10:33.174040+0000","last_unstale":"2024-03-19T01:11:28.216405+0000","last_undegraded":"2024-03-19T01:11:28.216405+0000","last_fullsized":"2024-03-19T01:11:28.216405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:53.850400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148335+0000","last_change":"2024-03-19T01:10:32.149194+0000","last_active":"2024-03-19T01:11:28.148335+0000","last_peered":"2024-03-19T01:11:28.148335+0000","last_clean":"2024-03-19T01:11:28.148335+0000","last_became_active":"2024-03-19T01:10:32.148835+0000","last_became_peered":"2024-03-19T01:10:32.148835+0000","last_unstale":"2024-03-19T01:11:28.148335+0000","last_undegraded":"2024-03-19T01:11:28.148335+0000","last_fullsized":"2024-03-19T01:11:28.148335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:55.697445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148317+0000","last_change":"2024-03-19T01:10:33.180203+0000","last_active":"2024-03-19T01:11:28.148317+0000","last_peered":"2024-03-19T01:11:28.148317+0000","last_clean":"2024-03-19T01:11:28.148317+0000","last_became_active":"2024-03-19T01:10:33.179812+0000","last_became_peered":"2024-03-19T01:10:33.179812+0000","last_unstale":"2024-03-19T01:11:28.148317+0000","last_undegraded":"2024-03-19T01:11:28.148317+0000","last_fullsized":"2024-03-19T01:11:28.148317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:38.371706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153151+0000","last_change":"2024-03-19T01:10:32.153852+0000","last_active":"2024-03-19T01:11:28.153151+0000","last_peered":"2024-03-19T01:11:28.153151+0000","last_clean":"2024-03-19T01:11:28.153151+0000","last_became_active":"2024-03-19T01:10:32.153537+0000","last_became_peered":"2024-03-19T01:10:32.153537+0000","last_unstale":"2024-03-19T01:11:28.153151+0000","last_undegraded":"2024-03-19T01:11:28.153151+0000","last_fullsized":"2024-03-19T01:11:28.153151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:48.544344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.216266+0000","last_change":"2024-03-19T01:10:33.175629+0000","last_active":"2024-03-19T01:11:28.216266+0000","last_peered":"2024-03-19T01:11:28.216266+0000","last_clean":"2024-03-19T01:11:28.216266+0000","last_became_active":"2024-03-19T01:10:33.175278+0000","last_became_peered":"2024-03-19T01:10:33.175278+0000","last_unstale":"2024-03-19T01:11:28.216266+0000","last_undegraded":"2024-03-19T01:11:28.216266+0000","last_fullsized":"2024-03-19T01:11:28.216266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:42:31.534216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'30","reported_seq":196,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:48.062921+0000","last_change":"2024-03-19T01:10:32.148290+0000","last_active":"2024-03-19T01:11:48.062921+0000","last_peered":"2024-03-19T01:11:48.062921+0000","last_clean":"2024-03-19T01:11:48.062921+0000","last_became_active":"2024-03-19T01:10:32.147940+0000","last_became_peered":"2024-03-19T01:10:32.147940+0000","last_unstale":"2024-03-19T01:11:48.062921+0000","last_undegraded":"2024-03-19T01:11:48.062921+0000","last_fullsized":"2024-03-19T01:11:48.062921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:40:58.125706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":147,"num_read_kb":133,"num_write":32,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148877+0000","last_change":"2024-03-19T01:10:33.174723+0000","last_active":"2024-03-19T01:11:28.148877+0000","last_peered":"2024-03-19T01:11:28.148877+0000","last_clean":"2024-03-19T01:11:28.148877+0000","last_became_active":"2024-03-19T01:10:33.174369+0000","last_became_peered":"2024-03-19T01:10:33.174369+0000","last_unstale":"2024-03-19T01:11:28.148877+0000","last_undegraded":"2024-03-19T01:11:28.148877+0000","last_fullsized":"2024-03-19T01:11:28.148877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:33.641050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150624+0000","last_change":"2024-03-19T01:10:32.146760+0000","last_active":"2024-03-19T01:11:28.150624+0000","last_peered":"2024-03-19T01:11:28.150624+0000","last_clean":"2024-03-19T01:11:28.150624+0000","last_became_active":"2024-03-19T01:10:32.145610+0000","last_became_peered":"2024-03-19T01:10:32.145610+0000","last_unstale":"2024-03-19T01:11:28.150624+0000","last_undegraded":"2024-03-19T01:11:28.150624+0000","last_fullsized":"2024-03-19T01:11:28.150624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:10:10.959609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.149541+0000","last_change":"2024-03-19T01:10:33.176948+0000","last_active":"2024-03-19T01:11:28.149541+0000","last_peered":"2024-03-19T01:11:28.149541+0000","last_clean":"2024-03-19T01:11:28.149541+0000","last_became_active":"2024-03-19T01:10:33.176604+0000","last_became_peered":"2024-03-19T01:10:33.176604+0000","last_unstale":"2024-03-19T01:11:28.149541+0000","last_undegraded":"2024-03-19T01:11:28.149541+0000","last_fullsized":"2024-03-19T01:11:28.149541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:45:45.032639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151964+0000","last_change":"2024-03-19T01:10:32.152397+0000","last_active":"2024-03-19T01:11:28.151964+0000","last_peered":"2024-03-19T01:11:28.151964+0000","last_clean":"2024-03-19T01:11:28.151964+0000","last_became_active":"2024-03-19T01:10:32.152026+0000","last_became_peered":"2024-03-19T01:10:32.152026+0000","last_unstale":"2024-03-19T01:11:28.151964+0000","last_undegraded":"2024-03-19T01:11:28.151964+0000","last_fullsized":"2024-03-19T01:11:28.151964+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:47.789504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148628+0000","last_change":"2024-03-19T01:10:33.168529+0000","last_active":"2024-03-19T01:11:28.148628+0000","last_peered":"2024-03-19T01:11:28.148628+0000","last_clean":"2024-03-19T01:11:28.148628+0000","last_became_active":"2024-03-19T01:10:33.168198+0000","last_became_peered":"2024-03-19T01:10:33.168198+0000","last_unstale":"2024-03-19T01:11:28.148628+0000","last_undegraded":"2024-03-19T01:11:28.148628+0000","last_fullsized":"2024-03-19T01:11:28.148628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:36.030791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'6","reported_seq":97,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:47.062503+0000","last_change":"2024-03-19T01:10:32.145565+0000","last_active":"2024-03-19T01:11:47.062503+0000","last_peered":"2024-03-19T01:11:47.062503+0000","last_clean":"2024-03-19T01:11:47.062503+0000","last_became_active":"2024-03-19T01:10:32.145234+0000","last_became_peered":"2024-03-19T01:10:32.145234+0000","last_unstale":"2024-03-19T01:11:47.062503+0000","last_undegraded":"2024-03-19T01:11:47.062503+0000","last_fullsized":"2024-03-19T01:11:47.062503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:35.563136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148122+0000","last_change":"2024-03-19T01:10:33.182001+0000","last_active":"2024-03-19T01:11:28.148122+0000","last_peered":"2024-03-19T01:11:28.148122+0000","last_clean":"2024-03-19T01:11:28.148122+0000","last_became_active":"2024-03-19T01:10:33.181638+0000","last_became_peered":"2024-03-19T01:10:33.181638+0000","last_unstale":"2024-03-19T01:11:28.148122+0000","last_undegraded":"2024-03-19T01:11:28.148122+0000","last_fullsized":"2024-03-19T01:11:28.148122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:59:23.155190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148157+0000","last_change":"2024-03-19T01:10:32.154159+0000","last_active":"2024-03-19T01:11:28.148157+0000","last_peered":"2024-03-19T01:11:28.148157+0000","last_clean":"2024-03-19T01:11:28.148157+0000","last_became_active":"2024-03-19T01:10:32.153853+0000","last_became_peered":"2024-03-19T01:10:32.153853+0000","last_unstale":"2024-03-19T01:11:28.148157+0000","last_undegraded":"2024-03-19T01:11:28.148157+0000","last_fullsized":"2024-03-19T01:11:28.148157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:41:13.727588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.217014+0000","last_change":"2024-03-19T01:11:28.217014+0000","last_active":"2024-03-19T01:11:28.217014+0000","last_peered":"2024-03-19T01:11:28.217014+0000","last_clean":"2024-03-19T01:11:28.217014+0000","last_became_active":"2024-03-19T01:11:28.216608+0000","last_became_peered":"2024-03-19T01:11:28.216608+0000","last_unstale":"2024-03-19T01:11:28.217014+0000","last_undegraded":"2024-03-19T01:11:28.217014+0000","last_fullsized":"2024-03-19T01:11:28.217014+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:47.600606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.149001+0000","last_change":"2024-03-19T01:10:32.144855+0000","last_active":"2024-03-19T01:11:28.149001+0000","last_peered":"2024-03-19T01:11:28.149001+0000","last_clean":"2024-03-19T01:11:28.149001+0000","last_became_active":"2024-03-19T01:10:32.144540+0000","last_became_peered":"2024-03-19T01:10:32.144540+0000","last_unstale":"2024-03-19T01:11:28.149001+0000","last_undegraded":"2024-03-19T01:11:28.149001+0000","last_fullsized":"2024-03-19T01:11:28.149001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:22.399954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153809+0000","last_change":"2024-03-19T01:10:33.172739+0000","last_active":"2024-03-19T01:11:28.153809+0000","last_peered":"2024-03-19T01:11:28.153809+0000","last_clean":"2024-03-19T01:11:28.153809+0000","last_became_active":"2024-03-19T01:10:33.172391+0000","last_became_peered":"2024-03-19T01:10:33.172391+0000","last_unstale":"2024-03-19T01:11:28.153809+0000","last_undegraded":"2024-03-19T01:11:28.153809+0000","last_fullsized":"2024-03-19T01:11:28.153809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:30.619013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151401+0000","last_change":"2024-03-19T01:10:32.144491+0000","last_active":"2024-03-19T01:11:28.151401+0000","last_peered":"2024-03-19T01:11:28.151401+0000","last_clean":"2024-03-19T01:11:28.151401+0000","last_became_active":"2024-03-19T01:10:32.144170+0000","last_became_peered":"2024-03-19T01:10:32.144170+0000","last_unstale":"2024-03-19T01:11:28.151401+0000","last_undegraded":"2024-03-19T01:11:28.151401+0000","last_fullsized":"2024-03-19T01:11:28.151401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:07.998818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151653+0000","last_change":"2024-03-19T01:10:33.175479+0000","last_active":"2024-03-19T01:11:28.151653+0000","last_peered":"2024-03-19T01:11:28.151653+0000","last_clean":"2024-03-19T01:11:28.151653+0000","last_became_active":"2024-03-19T01:10:33.172530+0000","last_became_peered":"2024-03-19T01:10:33.172530+0000","last_unstale":"2024-03-19T01:11:28.151653+0000","last_undegraded":"2024-03-19T01:11:28.151653+0000","last_fullsized":"2024-03-19T01:11:28.151653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:16.058826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":92,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:44.064452+0000","last_change":"2024-03-19T01:10:32.143629+0000","last_active":"2024-03-19T01:11:44.064452+0000","last_peered":"2024-03-19T01:11:44.064452+0000","last_clean":"2024-03-19T01:11:44.064452+0000","last_became_active":"2024-03-19T01:10:32.143321+0000","last_became_peered":"2024-03-19T01:10:32.143321+0000","last_unstale":"2024-03-19T01:11:44.064452+0000","last_undegraded":"2024-03-19T01:11:44.064452+0000","last_fullsized":"2024-03-19T01:11:44.064452+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:47.404245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151790+0000","last_change":"2024-03-19T01:10:33.171284+0000","last_active":"2024-03-19T01:11:28.151790+0000","last_peered":"2024-03-19T01:11:28.151790+0000","last_clean":"2024-03-19T01:11:28.151790+0000","last_became_active":"2024-03-19T01:10:33.168717+0000","last_became_peered":"2024-03-19T01:10:33.168717+0000","last_unstale":"2024-03-19T01:11:28.151790+0000","last_undegraded":"2024-03-19T01:11:28.151790+0000","last_fullsized":"2024-03-19T01:11:28.151790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:52.335303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.152064+0000","last_change":"2024-03-19T01:10:33.177261+0000","last_active":"2024-03-19T01:11:28.152064+0000","last_peered":"2024-03-19T01:11:28.152064+0000","last_clean":"2024-03-19T01:11:28.152064+0000","last_became_active":"2024-03-19T01:10:33.171836+0000","last_became_peered":"2024-03-19T01:10:33.171836+0000","last_unstale":"2024-03-19T01:11:28.152064+0000","last_undegraded":"2024-03-19T01:11:28.152064+0000","last_fullsized":"2024-03-19T01:11:28.152064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:49:14.577925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":91,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:43.058061+0000","last_change":"2024-03-19T01:10:32.141587+0000","last_active":"2024-03-19T01:11:43.058061+0000","last_peered":"2024-03-19T01:11:43.058061+0000","last_clean":"2024-03-19T01:11:43.058061+0000","last_became_active":"2024-03-19T01:10:32.141236+0000","last_became_peered":"2024-03-19T01:10:32.141236+0000","last_unstale":"2024-03-19T01:11:43.058061+0000","last_undegraded":"2024-03-19T01:11:43.058061+0000","last_fullsized":"2024-03-19T01:11:43.058061+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:07:40.685399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.216813+0000","last_change":"2024-03-19T01:11:28.216813+0000","last_active":"2024-03-19T01:11:28.216813+0000","last_peered":"2024-03-19T01:11:28.216813+0000","last_clean":"2024-03-19T01:11:28.216813+0000","last_became_active":"2024-03-19T01:11:28.216483+0000","last_became_peered":"2024-03-19T01:11:28.216483+0000","last_unstale":"2024-03-19T01:11:28.216813+0000","last_undegraded":"2024-03-19T01:11:28.216813+0000","last_fullsized":"2024-03-19T01:11:28.216813+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:38:49.974387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150951+0000","last_change":"2024-03-19T01:10:32.145816+0000","last_active":"2024-03-19T01:11:28.150951+0000","last_peered":"2024-03-19T01:11:28.150951+0000","last_clean":"2024-03-19T01:11:28.150951+0000","last_became_active":"2024-03-19T01:10:32.145055+0000","last_became_peered":"2024-03-19T01:10:32.145055+0000","last_unstale":"2024-03-19T01:11:28.150951+0000","last_undegraded":"2024-03-19T01:11:28.150951+0000","last_fullsized":"2024-03-19T01:11:28.150951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:56.428854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":342,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148180+0000","last_change":"2024-03-19T01:09:20.932686+0000","last_active":"2024-03-19T01:11:28.148180+0000","last_peered":"2024-03-19T01:11:28.148180+0000","last_clean":"2024-03-19T01:11:28.148180+0000","last_became_active":"2024-03-19T01:09:20.932170+0000","last_became_peered":"2024-03-19T01:09:20.932170+0000","last_unstale":"2024-03-19T01:11:28.148180+0000","last_undegraded":"2024-03-19T01:11:28.148180+0000","last_fullsized":"2024-03-19T01:11:28.148180+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:05:53.937682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:05:53.937682+0000","last_clean_scrub_stamp":"2024-03-19T01:05:53.937682+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:41.523200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150683+0000","last_change":"2024-03-19T01:10:33.173230+0000","last_active":"2024-03-19T01:11:28.150683+0000","last_peered":"2024-03-19T01:11:28.150683+0000","last_clean":"2024-03-19T01:11:28.150683+0000","last_became_active":"2024-03-19T01:10:33.172858+0000","last_became_peered":"2024-03-19T01:10:33.172858+0000","last_unstale":"2024-03-19T01:11:28.150683+0000","last_undegraded":"2024-03-19T01:11:28.150683+0000","last_fullsized":"2024-03-19T01:11:28.150683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:43.582211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148306+0000","last_change":"2024-03-19T01:10:32.147419+0000","last_active":"2024-03-19T01:11:28.148306+0000","last_peered":"2024-03-19T01:11:28.148306+0000","last_clean":"2024-03-19T01:11:28.148306+0000","last_became_active":"2024-03-19T01:10:32.147111+0000","last_became_peered":"2024-03-19T01:10:32.147111+0000","last_unstale":"2024-03-19T01:11:28.148306+0000","last_undegraded":"2024-03-19T01:11:28.148306+0000","last_fullsized":"2024-03-19T01:11:28.148306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:45:34.989763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148988+0000","last_change":"2024-03-19T01:10:32.154233+0000","last_active":"2024-03-19T01:11:28.148988+0000","last_peered":"2024-03-19T01:11:28.148988+0000","last_clean":"2024-03-19T01:11:28.148988+0000","last_became_active":"2024-03-19T01:10:32.153939+0000","last_became_peered":"2024-03-19T01:10:32.153939+0000","last_unstale":"2024-03-19T01:11:28.148988+0000","last_undegraded":"2024-03-19T01:11:28.148988+0000","last_fullsized":"2024-03-19T01:11:28.148988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:45.377161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151669+0000","last_change":"2024-03-19T01:10:33.168395+0000","last_active":"2024-03-19T01:11:28.151669+0000","last_peered":"2024-03-19T01:11:28.151669+0000","last_clean":"2024-03-19T01:11:28.151669+0000","last_became_active":"2024-03-19T01:10:33.168078+0000","last_became_peered":"2024-03-19T01:10:33.168078+0000","last_unstale":"2024-03-19T01:11:28.151669+0000","last_undegraded":"2024-03-19T01:11:28.151669+0000","last_fullsized":"2024-03-19T01:11:28.151669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:54:52.597591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'13","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.165836+0000","last_change":"2024-03-19T01:11:28.165836+0000","last_active":"2024-03-19T01:11:28.165836+0000","last_peered":"2024-03-19T01:11:28.165836+0000","last_clean":"2024-03-19T01:11:28.165836+0000","last_became_active":"2024-03-19T01:11:28.158005+0000","last_became_peered":"2024-03-19T01:11:28.158005+0000","last_unstale":"2024-03-19T01:11:28.165836+0000","last_undegraded":"2024-03-19T01:11:28.165836+0000","last_fullsized":"2024-03-19T01:11:28.165836+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:35:10.440755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.155960+0000","last_change":"2024-03-19T01:10:33.174635+0000","last_active":"2024-03-19T01:11:28.155960+0000","last_peered":"2024-03-19T01:11:28.155960+0000","last_clean":"2024-03-19T01:11:28.155960+0000","last_became_active":"2024-03-19T01:10:33.173970+0000","last_became_peered":"2024-03-19T01:10:33.173970+0000","last_unstale":"2024-03-19T01:11:28.155960+0000","last_undegraded":"2024-03-19T01:11:28.155960+0000","last_fullsized":"2024-03-19T01:11:28.155960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:16.174169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150856+0000","last_change":"2024-03-19T01:10:32.148225+0000","last_active":"2024-03-19T01:11:28.150856+0000","last_peered":"2024-03-19T01:11:28.150856+0000","last_clean":"2024-03-19T01:11:28.150856+0000","last_became_active":"2024-03-19T01:10:32.147889+0000","last_became_peered":"2024-03-19T01:10:32.147889+0000","last_unstale":"2024-03-19T01:11:28.150856+0000","last_undegraded":"2024-03-19T01:11:28.150856+0000","last_fullsized":"2024-03-19T01:11:28.150856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:56.037018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148136+0000","last_change":"2024-03-19T01:10:33.172925+0000","last_active":"2024-03-19T01:11:28.148136+0000","last_peered":"2024-03-19T01:11:28.148136+0000","last_clean":"2024-03-19T01:11:28.148136+0000","last_became_active":"2024-03-19T01:10:33.172572+0000","last_became_peered":"2024-03-19T01:10:33.172572+0000","last_unstale":"2024-03-19T01:11:28.148136+0000","last_undegraded":"2024-03-19T01:11:28.148136+0000","last_fullsized":"2024-03-19T01:11:28.148136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:02.853337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.215893+0000","last_change":"2024-03-19T01:10:32.146245+0000","last_active":"2024-03-19T01:11:28.215893+0000","last_peered":"2024-03-19T01:11:28.215893+0000","last_clean":"2024-03-19T01:11:28.215893+0000","last_became_active":"2024-03-19T01:10:32.145877+0000","last_became_peered":"2024-03-19T01:10:32.145877+0000","last_unstale":"2024-03-19T01:11:28.215893+0000","last_undegraded":"2024-03-19T01:11:28.215893+0000","last_fullsized":"2024-03-19T01:11:28.215893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:45.323155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148180+0000","last_change":"2024-03-19T01:10:33.178538+0000","last_active":"2024-03-19T01:11:28.148180+0000","last_peered":"2024-03-19T01:11:28.148180+0000","last_clean":"2024-03-19T01:11:28.148180+0000","last_became_active":"2024-03-19T01:10:33.178170+0000","last_became_peered":"2024-03-19T01:10:33.178170+0000","last_unstale":"2024-03-19T01:11:28.148180+0000","last_undegraded":"2024-03-19T01:11:28.148180+0000","last_fullsized":"2024-03-19T01:11:28.148180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:41.913715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153566+0000","last_change":"2024-03-19T01:10:32.147697+0000","last_active":"2024-03-19T01:11:28.153566+0000","last_peered":"2024-03-19T01:11:28.153566+0000","last_clean":"2024-03-19T01:11:28.153566+0000","last_became_active":"2024-03-19T01:10:32.147399+0000","last_became_peered":"2024-03-19T01:10:32.147399+0000","last_unstale":"2024-03-19T01:11:28.153566+0000","last_undegraded":"2024-03-19T01:11:28.153566+0000","last_fullsized":"2024-03-19T01:11:28.153566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:33:38.961646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151301+0000","last_change":"2024-03-19T01:10:33.175990+0000","last_active":"2024-03-19T01:11:28.151301+0000","last_peered":"2024-03-19T01:11:28.151301+0000","last_clean":"2024-03-19T01:11:28.151301+0000","last_became_active":"2024-03-19T01:10:33.175539+0000","last_became_peered":"2024-03-19T01:10:33.175539+0000","last_unstale":"2024-03-19T01:11:28.151301+0000","last_undegraded":"2024-03-19T01:11:28.151301+0000","last_fullsized":"2024-03-19T01:11:28.151301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:36.799131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150742+0000","last_change":"2024-03-19T01:10:32.151834+0000","last_active":"2024-03-19T01:11:28.150742+0000","last_peered":"2024-03-19T01:11:28.150742+0000","last_clean":"2024-03-19T01:11:28.150742+0000","last_became_active":"2024-03-19T01:10:32.151514+0000","last_became_peered":"2024-03-19T01:10:32.151514+0000","last_unstale":"2024-03-19T01:11:28.150742+0000","last_undegraded":"2024-03-19T01:11:28.150742+0000","last_fullsized":"2024-03-19T01:11:28.150742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:40:16.839334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.154035+0000","last_change":"2024-03-19T01:10:33.185206+0000","last_active":"2024-03-19T01:11:28.154035+0000","last_peered":"2024-03-19T01:11:28.154035+0000","last_clean":"2024-03-19T01:11:28.154035+0000","last_became_active":"2024-03-19T01:10:33.184901+0000","last_became_peered":"2024-03-19T01:10:33.184901+0000","last_unstale":"2024-03-19T01:11:28.154035+0000","last_undegraded":"2024-03-19T01:11:28.154035+0000","last_fullsized":"2024-03-19T01:11:28.154035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:54:47.266255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.220475+0000","last_change":"2024-03-19T01:11:28.220475+0000","last_active":"2024-03-19T01:11:28.220475+0000","last_peered":"2024-03-19T01:11:28.220475+0000","last_clean":"2024-03-19T01:11:28.220475+0000","last_became_active":"2024-03-19T01:11:28.217201+0000","last_became_peered":"2024-03-19T01:11:28.217201+0000","last_unstale":"2024-03-19T01:11:28.220475+0000","last_undegraded":"2024-03-19T01:11:28.220475+0000","last_fullsized":"2024-03-19T01:11:28.220475+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:14.366839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148764+0000","last_change":"2024-03-19T01:10:33.181232+0000","last_active":"2024-03-19T01:11:28.148764+0000","last_peered":"2024-03-19T01:11:28.148764+0000","last_clean":"2024-03-19T01:11:28.148764+0000","last_became_active":"2024-03-19T01:10:33.180719+0000","last_became_peered":"2024-03-19T01:10:33.180719+0000","last_unstale":"2024-03-19T01:11:28.148764+0000","last_undegraded":"2024-03-19T01:11:28.148764+0000","last_fullsized":"2024-03-19T01:11:28.148764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:43.201965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'5","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:47.178182+0000","last_change":"2024-03-19T01:11:28.160961+0000","last_active":"2024-03-19T01:11:47.178182+0000","last_peered":"2024-03-19T01:11:47.178182+0000","last_clean":"2024-03-19T01:11:47.178182+0000","last_became_active":"2024-03-19T01:11:28.155196+0000","last_became_peered":"2024-03-19T01:11:28.155196+0000","last_unstale":"2024-03-19T01:11:47.178182+0000","last_undegraded":"2024-03-19T01:11:47.178182+0000","last_fullsized":"2024-03-19T01:11:47.178182+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:38.586516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.149065+0000","last_change":"2024-03-19T01:10:33.178129+0000","last_active":"2024-03-19T01:11:28.149065+0000","last_peered":"2024-03-19T01:11:28.149065+0000","last_clean":"2024-03-19T01:11:28.149065+0000","last_became_active":"2024-03-19T01:10:33.177802+0000","last_became_peered":"2024-03-19T01:10:33.177802+0000","last_unstale":"2024-03-19T01:11:28.149065+0000","last_undegraded":"2024-03-19T01:11:28.149065+0000","last_fullsized":"2024-03-19T01:11:28.149065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:46:42.698865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.152934+0000","last_change":"2024-03-19T01:10:32.149613+0000","last_active":"2024-03-19T01:11:28.152934+0000","last_peered":"2024-03-19T01:11:28.152934+0000","last_clean":"2024-03-19T01:11:28.152934+0000","last_became_active":"2024-03-19T01:10:32.147651+0000","last_became_peered":"2024-03-19T01:10:32.147651+0000","last_unstale":"2024-03-19T01:11:28.152934+0000","last_undegraded":"2024-03-19T01:11:28.152934+0000","last_fullsized":"2024-03-19T01:11:28.152934+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:06:59.222944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.168072+0000","last_change":"2024-03-19T01:11:28.168072+0000","last_active":"2024-03-19T01:11:28.168072+0000","last_peered":"2024-03-19T01:11:28.168072+0000","last_clean":"2024-03-19T01:11:28.168072+0000","last_became_active":"2024-03-19T01:11:28.157973+0000","last_became_peered":"2024-03-19T01:11:28.157973+0000","last_unstale":"2024-03-19T01:11:28.168072+0000","last_undegraded":"2024-03-19T01:11:28.168072+0000","last_fullsized":"2024-03-19T01:11:28.168072+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:14.879683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150006+0000","last_change":"2024-03-19T01:10:32.149253+0000","last_active":"2024-03-19T01:11:28.150006+0000","last_peered":"2024-03-19T01:11:28.150006+0000","last_clean":"2024-03-19T01:11:28.150006+0000","last_became_active":"2024-03-19T01:10:32.148520+0000","last_became_peered":"2024-03-19T01:10:32.148520+0000","last_unstale":"2024-03-19T01:11:28.150006+0000","last_undegraded":"2024-03-19T01:11:28.150006+0000","last_fullsized":"2024-03-19T01:11:28.150006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:10:21.211510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148503+0000","last_change":"2024-03-19T01:10:33.182684+0000","last_active":"2024-03-19T01:11:28.148503+0000","last_peered":"2024-03-19T01:11:28.148503+0000","last_clean":"2024-03-19T01:11:28.148503+0000","last_became_active":"2024-03-19T01:10:33.182269+0000","last_became_peered":"2024-03-19T01:10:33.182269+0000","last_unstale":"2024-03-19T01:11:28.148503+0000","last_undegraded":"2024-03-19T01:11:28.148503+0000","last_fullsized":"2024-03-19T01:11:28.148503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:49:33.318462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151402+0000","last_change":"2024-03-19T01:10:33.172795+0000","last_active":"2024-03-19T01:11:28.151402+0000","last_peered":"2024-03-19T01:11:28.151402+0000","last_clean":"2024-03-19T01:11:28.151402+0000","last_became_active":"2024-03-19T01:10:33.172483+0000","last_became_peered":"2024-03-19T01:10:33.172483+0000","last_unstale":"2024-03-19T01:11:28.151402+0000","last_undegraded":"2024-03-19T01:11:28.151402+0000","last_fullsized":"2024-03-19T01:11:28.151402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:38:36.910222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.156200+0000","last_change":"2024-03-19T01:11:28.156200+0000","last_active":"2024-03-19T01:11:28.156200+0000","last_peered":"2024-03-19T01:11:28.156200+0000","last_clean":"2024-03-19T01:11:28.156200+0000","last_became_active":"2024-03-19T01:11:28.155824+0000","last_became_peered":"2024-03-19T01:11:28.155824+0000","last_unstale":"2024-03-19T01:11:28.156200+0000","last_undegraded":"2024-03-19T01:11:28.156200+0000","last_fullsized":"2024-03-19T01:11:28.156200+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:54:20.895403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150650+0000","last_change":"2024-03-19T01:10:33.178847+0000","last_active":"2024-03-19T01:11:28.150650+0000","last_peered":"2024-03-19T01:11:28.150650+0000","last_clean":"2024-03-19T01:11:28.150650+0000","last_became_active":"2024-03-19T01:10:33.178533+0000","last_became_peered":"2024-03-19T01:10:33.178533+0000","last_unstale":"2024-03-19T01:11:28.150650+0000","last_undegraded":"2024-03-19T01:11:28.150650+0000","last_fullsized":"2024-03-19T01:11:28.150650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:52:15.401797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148064+0000","last_change":"2024-03-19T01:10:33.169936+0000","last_active":"2024-03-19T01:11:28.148064+0000","last_peered":"2024-03-19T01:11:28.148064+0000","last_clean":"2024-03-19T01:11:28.148064+0000","last_became_active":"2024-03-19T01:10:33.169594+0000","last_became_peered":"2024-03-19T01:10:33.169594+0000","last_unstale":"2024-03-19T01:11:28.148064+0000","last_undegraded":"2024-03-19T01:11:28.148064+0000","last_fullsized":"2024-03-19T01:11:28.148064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:41.932575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.154178+0000","last_change":"2024-03-19T01:11:28.154178+0000","last_active":"2024-03-19T01:11:28.154178+0000","last_peered":"2024-03-19T01:11:28.154178+0000","last_clean":"2024-03-19T01:11:28.154178+0000","last_became_active":"2024-03-19T01:11:28.153739+0000","last_became_peered":"2024-03-19T01:11:28.153739+0000","last_unstale":"2024-03-19T01:11:28.154178+0000","last_undegraded":"2024-03-19T01:11:28.154178+0000","last_fullsized":"2024-03-19T01:11:28.154178+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:48:37.480130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.154271+0000","last_change":"2024-03-19T01:10:33.178699+0000","last_active":"2024-03-19T01:11:28.154271+0000","last_peered":"2024-03-19T01:11:28.154271+0000","last_clean":"2024-03-19T01:11:28.154271+0000","last_became_active":"2024-03-19T01:10:33.178380+0000","last_became_peered":"2024-03-19T01:10:33.178380+0000","last_unstale":"2024-03-19T01:11:28.154271+0000","last_undegraded":"2024-03-19T01:11:28.154271+0000","last_fullsized":"2024-03-19T01:11:28.154271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:46:29.502416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150752+0000","last_change":"2024-03-19T01:10:33.179908+0000","last_active":"2024-03-19T01:11:28.150752+0000","last_peered":"2024-03-19T01:11:28.150752+0000","last_clean":"2024-03-19T01:11:28.150752+0000","last_became_active":"2024-03-19T01:10:33.179578+0000","last_became_peered":"2024-03-19T01:10:33.179578+0000","last_unstale":"2024-03-19T01:11:28.150752+0000","last_undegraded":"2024-03-19T01:11:28.150752+0000","last_fullsized":"2024-03-19T01:11:28.150752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:30:53.918078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153979+0000","last_change":"2024-03-19T01:10:33.179788+0000","last_active":"2024-03-19T01:11:28.153979+0000","last_peered":"2024-03-19T01:11:28.153979+0000","last_clean":"2024-03-19T01:11:28.153979+0000","last_became_active":"2024-03-19T01:10:33.179473+0000","last_became_peered":"2024-03-19T01:10:33.179473+0000","last_unstale":"2024-03-19T01:11:28.153979+0000","last_undegraded":"2024-03-19T01:11:28.153979+0000","last_fullsized":"2024-03-19T01:11:28.153979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:36.284768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151690+0000","last_change":"2024-03-19T01:10:33.172698+0000","last_active":"2024-03-19T01:11:28.151690+0000","last_peered":"2024-03-19T01:11:28.151690+0000","last_clean":"2024-03-19T01:11:28.151690+0000","last_became_active":"2024-03-19T01:10:33.172369+0000","last_became_peered":"2024-03-19T01:10:33.172369+0000","last_unstale":"2024-03-19T01:11:28.151690+0000","last_undegraded":"2024-03-19T01:11:28.151690+0000","last_fullsized":"2024-03-19T01:11:28.151690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:09:18.961724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153491+0000","last_change":"2024-03-19T01:10:33.184350+0000","last_active":"2024-03-19T01:11:28.153491+0000","last_peered":"2024-03-19T01:11:28.153491+0000","last_clean":"2024-03-19T01:11:28.153491+0000","last_became_active":"2024-03-19T01:10:33.180533+0000","last_became_peered":"2024-03-19T01:10:33.180533+0000","last_unstale":"2024-03-19T01:11:28.153491+0000","last_undegraded":"2024-03-19T01:11:28.153491+0000","last_fullsized":"2024-03-19T01:11:28.153491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:28.003560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.216073+0000","last_change":"2024-03-19T01:10:33.180396+0000","last_active":"2024-03-19T01:11:28.216073+0000","last_peered":"2024-03-19T01:11:28.216073+0000","last_clean":"2024-03-19T01:11:28.216073+0000","last_became_active":"2024-03-19T01:10:33.180062+0000","last_became_peered":"2024-03-19T01:10:33.180062+0000","last_unstale":"2024-03-19T01:11:28.216073+0000","last_undegraded":"2024-03-19T01:11:28.216073+0000","last_fullsized":"2024-03-19T01:11:28.216073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:15.429623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.149893+0000","last_change":"2024-03-19T01:10:33.167261+0000","last_active":"2024-03-19T01:11:28.149893+0000","last_peered":"2024-03-19T01:11:28.149893+0000","last_clean":"2024-03-19T01:11:28.149893+0000","last_became_active":"2024-03-19T01:10:33.166943+0000","last_became_peered":"2024-03-19T01:10:33.166943+0000","last_unstale":"2024-03-19T01:11:28.149893+0000","last_undegraded":"2024-03-19T01:11:28.149893+0000","last_fullsized":"2024-03-19T01:11:28.149893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:01:44.451161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.152915+0000","last_change":"2024-03-19T01:10:33.175750+0000","last_active":"2024-03-19T01:11:28.152915+0000","last_peered":"2024-03-19T01:11:28.152915+0000","last_clean":"2024-03-19T01:11:28.152915+0000","last_became_active":"2024-03-19T01:10:33.175411+0000","last_became_peered":"2024-03-19T01:10:33.175411+0000","last_unstale":"2024-03-19T01:11:28.152915+0000","last_undegraded":"2024-03-19T01:11:28.152915+0000","last_fullsized":"2024-03-19T01:11:28.152915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:20:41.113626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284643,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67859,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":675,"num_read_kb":433,"num_write":238,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1170,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":203577,"data_compressed":5559,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383708,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":457013,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645346,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1101824,"data_stored":1109479,"data_compressed":43833,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808871,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":456943,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070508,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":457321,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332145,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":1049839,"data_compressed":41980,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495670,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":456777,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":518619,"data_compressed":7638,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":457999,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1028096,"data_stored":517669,"data_compressed":7638,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":456485,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920833,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":458082,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757308,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":1049009,"data_compressed":41980,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34556,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34556,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34556,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62798,"data_compressed":1853,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":61848,"data_compressed":1853,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":63173,"data_compressed":1853,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:11:51.148 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-19T01:11:51.441 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:51.707 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:51 smithi116 ceph-mon[62701]: from='client.24775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:11:51.707 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:51 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:51.707 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:51 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:51.707 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:51 smithi116 ceph-mon[62701]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T01:11:51.707 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:51 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:11:51.707 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:51 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:11:51.707 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:51 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:51.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:51 smithi156 ceph-mon[66799]: from='client.24775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:11:51.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:51 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:51 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:51 smithi156 ceph-mon[66799]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T01:11:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:51 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:11:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:51 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:11:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:51 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:51.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:51 smithi162 ceph-mon[67178]: from='client.24775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:11:51.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:51 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:51.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:51 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:51.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:51 smithi162 ceph-mon[67178]: pgmap v329: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T01:11:51.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:51 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:11:51.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:51 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:11:51.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:51 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:11:53.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.173+0000 7f702c638700 1 -- 172.21.15.116:0/3470753585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024108410 msgr2=0x7f70241087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:53.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.173+0000 7f702c638700 1 --2- 172.21.15.116:0/3470753585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024108410 0x7f70241087f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7020009990 tx=0x7f7020009ca0 comp rx=0 tx=0).stop 2024-03-19T01:11:53.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.173+0000 7f702c638700 1 -- 172.21.15.116:0/3470753585 shutdown_connections 2024-03-19T01:11:53.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.173+0000 7f702c638700 1 --2- 172.21.15.116:0/3470753585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7024102d20 0x7f702410f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:53.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.173+0000 7f702c638700 1 --2- 172.21.15.116:0/3470753585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024108410 0x7f70241087f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:53.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.173+0000 7f702c638700 1 --2- 172.21.15.116:0/3470753585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7024102360 0x7f70241027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:53.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.173+0000 7f702c638700 1 -- 172.21.15.116:0/3470753585 >> 172.21.15.116:0/3470753585 conn(0x7f7024079440 msgr2=0x7f7024079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:53.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.173+0000 7f702c638700 1 -- 172.21.15.116:0/3470753585 shutdown_connections 2024-03-19T01:11:53.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.173+0000 7f702c638700 1 -- 172.21.15.116:0/3470753585 wait complete. 2024-03-19T01:11:53.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.174+0000 7f702c638700 1 Processor -- start 2024-03-19T01:11:53.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.174+0000 7f702c638700 1 -- start start 2024-03-19T01:11:53.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.174+0000 7f702c638700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7024102360 0x7f702419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:53.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.174+0000 7f702c638700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7024102d20 0x7f702419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:53.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.174+0000 7f702c638700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024108410 0x7f70241988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:53.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.174+0000 7f702c638700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7024112650 con 0x7f7024108410 2024-03-19T01:11:53.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.174+0000 7f702c638700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f70241124d0 con 0x7f7024102360 2024-03-19T01:11:53.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.174+0000 7f702a3d4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7024102360 0x7f702419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:53.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.174+0000 7f702c638700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f70241127d0 con 0x7f7024102d20 2024-03-19T01:11:53.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.174+0000 7f702a3d4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7024102360 0x7f702419e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:34560/0 (socket says 172.21.15.116:34560) 2024-03-19T01:11:53.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.174+0000 7f702a3d4700 1 -- 172.21.15.116:0/3779766916 learned_addr learned my addr 172.21.15.116:0/3779766916 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:53.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.174+0000 7f702abd5700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024108410 0x7f70241988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:53.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.174+0000 7f7029bd3700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7024102d20 0x7f702419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:53.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.175+0000 7f702abd5700 1 -- 172.21.15.116:0/3779766916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7024102d20 msgr2=0x7f702419ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:53.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.175+0000 7f702abd5700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7024102d20 0x7f702419ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:53.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.175+0000 7f702abd5700 1 -- 172.21.15.116:0/3779766916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7024102360 msgr2=0x7f702419e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.175+0000 7f702abd5700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7024102360 0x7f702419e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.175+0000 7f702abd5700 1 -- 172.21.15.116:0/3779766916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7024199030 con 0x7f7024108410 2024-03-19T01:11:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.175+0000 7f702a3d4700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7024102360 0x7f702419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.175+0000 7f702abd5700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024108410 0x7f70241988d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f701c00efb0 tx=0x7f701c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.175+0000 7f701b7fe700 1 -- 172.21.15.116:0/3779766916 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701c003a00 con 0x7f7024108410 2024-03-19T01:11:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.175+0000 7f701b7fe700 1 -- 172.21.15.116:0/3779766916 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f701c003ba0 con 0x7f7024108410 2024-03-19T01:11:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.175+0000 7f702c638700 1 -- 172.21.15.116:0/3779766916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7024199320 con 0x7f7024108410 2024-03-19T01:11:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.175+0000 7f701b7fe700 1 -- 172.21.15.116:0/3779766916 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701c004850 con 0x7f7024108410 2024-03-19T01:11:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.175+0000 7f702c638700 1 -- 172.21.15.116:0/3779766916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70241a55e0 con 0x7f7024108410 2024-03-19T01:11:53.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.176+0000 7f702c638700 1 -- 172.21.15.116:0/3779766916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7024068fd0 con 0x7f7024108410 2024-03-19T01:11:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.178+0000 7f701b7fe700 1 -- 172.21.15.116:0/3779766916 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f701c012050 con 0x7f7024108410 2024-03-19T01:11:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.179+0000 7f701b7fe700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70100a29a0 0x7f70100a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.179+0000 7f702a3d4700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70100a29a0 0x7f70100a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.179+0000 7f701b7fe700 1 -- 172.21.15.116:0/3779766916 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f701c0ce420 con 0x7f7024108410 2024-03-19T01:11:53.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.179+0000 7f702a3d4700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70100a29a0 0x7f70100a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7020009750 tx=0x7f7020005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:53.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.185+0000 7f701b7fe700 1 -- 172.21.15.116:0/3779766916 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f701c022060 con 0x7f7024108410 2024-03-19T01:11:53.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.379+0000 7f702c638700 1 -- 172.21.15.116:0/3779766916 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7024062750 con 0x7f70100a29a0 2024-03-19T01:11:53.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.386+0000 7f701b7fe700 1 -- 172.21.15.116:0/3779766916 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259066 (secure 0 0 0) 0x7f7020067410 con 0x7f70100a29a0 2024-03-19T01:11:53.390 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:11:53.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.389+0000 7f702c638700 1 -- 172.21.15.116:0/3779766916 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70100a29a0 msgr2=0x7f70100a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:53.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.389+0000 7f702c638700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70100a29a0 0x7f70100a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7020009750 tx=0x7f7020005e80 comp rx=0 tx=0).stop 2024-03-19T01:11:53.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.389+0000 7f702c638700 1 -- 172.21.15.116:0/3779766916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024108410 msgr2=0x7f70241988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:53.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.389+0000 7f702c638700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024108410 0x7f70241988d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f701c00efb0 tx=0x7f701c00c3a0 comp rx=0 tx=0).stop 2024-03-19T01:11:53.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.389+0000 7f702c638700 1 -- 172.21.15.116:0/3779766916 shutdown_connections 2024-03-19T01:11:53.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.389+0000 7f702c638700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7024108410 0x7f70241988d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:53.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.389+0000 7f702c638700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7024102360 0x7f702419e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:53.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.389+0000 7f702c638700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70100a29a0 0x7f70100a4e60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:53.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.389+0000 7f702c638700 1 --2- 172.21.15.116:0/3779766916 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7024102d20 0x7f702419ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:53.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.389+0000 7f702c638700 1 -- 172.21.15.116:0/3779766916 >> 172.21.15.116:0/3779766916 conn(0x7f7024079440 msgr2=0x7f7024101080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:53.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.389+0000 7f702c638700 1 -- 172.21.15.116:0/3779766916 shutdown_connections 2024-03-19T01:11:53.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:53.389+0000 7f702c638700 1 -- 172.21.15.116:0/3779766916 wait complete. 2024-03-19T01:11:53.393 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-03-19T01:11:54.097 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-03-19T01:11:53.074707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658461,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":816,"num_read_kb":552,"num_write":421,"num_write_kb":3073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1182023,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":381744,"kb_used_data":11700,"kb_used_omap":18,"kb_used_meta":369837,"kb_avail":1124560080,"statfs":{"total":1151940427776,"available":1151549521920,"internally_reserved":0,"allocated":11980800,"data_stored":7445235,"data_compressed":183564,"data_compressed_allocated":245760,"data_compressed_original":1990656,"omap_allocated":19143,"internal_metadata":378713401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002543"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.154655+0000","last_change":"2024-03-19T01:10:33.176468+0000","last_active":"2024-03-19T01:11:28.154655+0000","last_peered":"2024-03-19T01:11:28.154655+0000","last_clean":"2024-03-19T01:11:28.154655+0000","last_became_active":"2024-03-19T01:10:33.175995+0000","last_became_peered":"2024-03-19T01:10:33.175995+0000","last_unstale":"2024-03-19T01:11:28.154655+0000","last_undegraded":"2024-03-19T01:11:28.154655+0000","last_fullsized":"2024-03-19T01:11:28.154655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:40:02.320205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148022+0000","last_change":"2024-03-19T01:10:33.171253+0000","last_active":"2024-03-19T01:11:28.148022+0000","last_peered":"2024-03-19T01:11:28.148022+0000","last_clean":"2024-03-19T01:11:28.148022+0000","last_became_active":"2024-03-19T01:10:33.170330+0000","last_became_peered":"2024-03-19T01:10:33.170330+0000","last_unstale":"2024-03-19T01:11:28.148022+0000","last_undegraded":"2024-03-19T01:11:28.148022+0000","last_fullsized":"2024-03-19T01:11:28.148022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:38.905220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148298+0000","last_change":"2024-03-19T01:10:33.178006+0000","last_active":"2024-03-19T01:11:28.148298+0000","last_peered":"2024-03-19T01:11:28.148298+0000","last_clean":"2024-03-19T01:11:28.148298+0000","last_became_active":"2024-03-19T01:10:33.177662+0000","last_became_peered":"2024-03-19T01:10:33.177662+0000","last_unstale":"2024-03-19T01:11:28.148298+0000","last_undegraded":"2024-03-19T01:11:28.148298+0000","last_fullsized":"2024-03-19T01:11:28.148298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:16:08.980220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151084+0000","last_change":"2024-03-19T01:10:33.178805+0000","last_active":"2024-03-19T01:11:28.151084+0000","last_peered":"2024-03-19T01:11:28.151084+0000","last_clean":"2024-03-19T01:11:28.151084+0000","last_became_active":"2024-03-19T01:10:33.178385+0000","last_became_peered":"2024-03-19T01:10:33.178385+0000","last_unstale":"2024-03-19T01:11:28.151084+0000","last_undegraded":"2024-03-19T01:11:28.151084+0000","last_fullsized":"2024-03-19T01:11:28.151084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:59:06.454345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148248+0000","last_change":"2024-03-19T01:10:33.183110+0000","last_active":"2024-03-19T01:11:28.148248+0000","last_peered":"2024-03-19T01:11:28.148248+0000","last_clean":"2024-03-19T01:11:28.148248+0000","last_became_active":"2024-03-19T01:10:33.180036+0000","last_became_peered":"2024-03-19T01:10:33.180036+0000","last_unstale":"2024-03-19T01:11:28.148248+0000","last_undegraded":"2024-03-19T01:11:28.148248+0000","last_fullsized":"2024-03-19T01:11:28.148248+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:57:51.397156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.154164+0000","last_change":"2024-03-19T01:11:28.154164+0000","last_active":"2024-03-19T01:11:28.154164+0000","last_peered":"2024-03-19T01:11:28.154164+0000","last_clean":"2024-03-19T01:11:28.154164+0000","last_became_active":"2024-03-19T01:11:28.153778+0000","last_became_peered":"2024-03-19T01:11:28.153778+0000","last_unstale":"2024-03-19T01:11:28.154164+0000","last_undegraded":"2024-03-19T01:11:28.154164+0000","last_fullsized":"2024-03-19T01:11:28.154164+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:47:50.083781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,7],"acting":[10,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148162+0000","last_change":"2024-03-19T01:10:33.172946+0000","last_active":"2024-03-19T01:11:28.148162+0000","last_peered":"2024-03-19T01:11:28.148162+0000","last_clean":"2024-03-19T01:11:28.148162+0000","last_became_active":"2024-03-19T01:10:33.172617+0000","last_became_peered":"2024-03-19T01:10:33.172617+0000","last_unstale":"2024-03-19T01:11:28.148162+0000","last_undegraded":"2024-03-19T01:11:28.148162+0000","last_fullsized":"2024-03-19T01:11:28.148162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:18:20.311445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153044+0000","last_change":"2024-03-19T01:10:33.178162+0000","last_active":"2024-03-19T01:11:28.153044+0000","last_peered":"2024-03-19T01:11:28.153044+0000","last_clean":"2024-03-19T01:11:28.153044+0000","last_became_active":"2024-03-19T01:10:33.177752+0000","last_became_peered":"2024-03-19T01:10:33.177752+0000","last_unstale":"2024-03-19T01:11:28.153044+0000","last_undegraded":"2024-03-19T01:11:28.153044+0000","last_fullsized":"2024-03-19T01:11:28.153044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:01:35.681694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.216525+0000","last_change":"2024-03-19T01:10:33.175619+0000","last_active":"2024-03-19T01:11:28.216525+0000","last_peered":"2024-03-19T01:11:28.216525+0000","last_clean":"2024-03-19T01:11:28.216525+0000","last_became_active":"2024-03-19T01:10:33.175017+0000","last_became_peered":"2024-03-19T01:10:33.175017+0000","last_unstale":"2024-03-19T01:11:28.216525+0000","last_undegraded":"2024-03-19T01:11:28.216525+0000","last_fullsized":"2024-03-19T01:11:28.216525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:49:52.019461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151899+0000","last_change":"2024-03-19T01:10:33.173035+0000","last_active":"2024-03-19T01:11:28.151899+0000","last_peered":"2024-03-19T01:11:28.151899+0000","last_clean":"2024-03-19T01:11:28.151899+0000","last_became_active":"2024-03-19T01:10:33.172717+0000","last_became_peered":"2024-03-19T01:10:33.172717+0000","last_unstale":"2024-03-19T01:11:28.151899+0000","last_undegraded":"2024-03-19T01:11:28.151899+0000","last_fullsized":"2024-03-19T01:11:28.151899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:09:38.078147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148957+0000","last_change":"2024-03-19T01:10:33.182679+0000","last_active":"2024-03-19T01:11:28.148957+0000","last_peered":"2024-03-19T01:11:28.148957+0000","last_clean":"2024-03-19T01:11:28.148957+0000","last_became_active":"2024-03-19T01:10:33.182384+0000","last_became_peered":"2024-03-19T01:10:33.182384+0000","last_unstale":"2024-03-19T01:11:28.148957+0000","last_undegraded":"2024-03-19T01:11:28.148957+0000","last_fullsized":"2024-03-19T01:11:28.148957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:37:26.542393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150655+0000","last_change":"2024-03-19T01:10:33.178818+0000","last_active":"2024-03-19T01:11:28.150655+0000","last_peered":"2024-03-19T01:11:28.150655+0000","last_clean":"2024-03-19T01:11:28.150655+0000","last_became_active":"2024-03-19T01:10:33.178286+0000","last_became_peered":"2024-03-19T01:10:33.178286+0000","last_unstale":"2024-03-19T01:11:28.150655+0000","last_undegraded":"2024-03-19T01:11:28.150655+0000","last_fullsized":"2024-03-19T01:11:28.150655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:05:19.473738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.149581+0000","last_change":"2024-03-19T01:10:33.157042+0000","last_active":"2024-03-19T01:11:28.149581+0000","last_peered":"2024-03-19T01:11:28.149581+0000","last_clean":"2024-03-19T01:11:28.149581+0000","last_became_active":"2024-03-19T01:10:33.156576+0000","last_became_peered":"2024-03-19T01:10:33.156576+0000","last_unstale":"2024-03-19T01:11:28.149581+0000","last_undegraded":"2024-03-19T01:11:28.149581+0000","last_fullsized":"2024-03-19T01:11:28.149581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:57:51.625850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.147897+0000","last_change":"2024-03-19T01:10:33.177981+0000","last_active":"2024-03-19T01:11:28.147897+0000","last_peered":"2024-03-19T01:11:28.147897+0000","last_clean":"2024-03-19T01:11:28.147897+0000","last_became_active":"2024-03-19T01:10:33.177634+0000","last_became_peered":"2024-03-19T01:10:33.177634+0000","last_unstale":"2024-03-19T01:11:28.147897+0000","last_undegraded":"2024-03-19T01:11:28.147897+0000","last_fullsized":"2024-03-19T01:11:28.147897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:29:49.646938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.147763+0000","last_change":"2024-03-19T01:10:33.168982+0000","last_active":"2024-03-19T01:11:28.147763+0000","last_peered":"2024-03-19T01:11:28.147763+0000","last_clean":"2024-03-19T01:11:28.147763+0000","last_became_active":"2024-03-19T01:10:33.168569+0000","last_became_peered":"2024-03-19T01:10:33.168569+0000","last_unstale":"2024-03-19T01:11:28.147763+0000","last_undegraded":"2024-03-19T01:11:28.147763+0000","last_fullsized":"2024-03-19T01:11:28.147763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:45:43.389585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.215848+0000","last_change":"2024-03-19T01:10:33.184056+0000","last_active":"2024-03-19T01:11:28.215848+0000","last_peered":"2024-03-19T01:11:28.215848+0000","last_clean":"2024-03-19T01:11:28.215848+0000","last_became_active":"2024-03-19T01:10:33.183712+0000","last_became_peered":"2024-03-19T01:10:33.183712+0000","last_unstale":"2024-03-19T01:11:28.215848+0000","last_undegraded":"2024-03-19T01:11:28.215848+0000","last_fullsized":"2024-03-19T01:11:28.215848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:27.268859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.154753+0000","last_change":"2024-03-19T01:10:33.180114+0000","last_active":"2024-03-19T01:11:28.154753+0000","last_peered":"2024-03-19T01:11:28.154753+0000","last_clean":"2024-03-19T01:11:28.154753+0000","last_became_active":"2024-03-19T01:10:33.179796+0000","last_became_peered":"2024-03-19T01:10:33.179796+0000","last_unstale":"2024-03-19T01:11:28.154753+0000","last_undegraded":"2024-03-19T01:11:28.154753+0000","last_fullsized":"2024-03-19T01:11:28.154753+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:55:23.009912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151331+0000","last_change":"2024-03-19T01:10:33.179888+0000","last_active":"2024-03-19T01:11:28.151331+0000","last_peered":"2024-03-19T01:11:28.151331+0000","last_clean":"2024-03-19T01:11:28.151331+0000","last_became_active":"2024-03-19T01:10:33.179562+0000","last_became_peered":"2024-03-19T01:10:33.179562+0000","last_unstale":"2024-03-19T01:11:28.151331+0000","last_undegraded":"2024-03-19T01:11:28.151331+0000","last_fullsized":"2024-03-19T01:11:28.151331+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:58:53.346043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151054+0000","last_change":"2024-03-19T01:10:33.177619+0000","last_active":"2024-03-19T01:11:28.151054+0000","last_peered":"2024-03-19T01:11:28.151054+0000","last_clean":"2024-03-19T01:11:28.151054+0000","last_became_active":"2024-03-19T01:10:33.176002+0000","last_became_peered":"2024-03-19T01:10:33.176002+0000","last_unstale":"2024-03-19T01:11:28.151054+0000","last_undegraded":"2024-03-19T01:11:28.151054+0000","last_fullsized":"2024-03-19T01:11:28.151054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:23:50.637964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":107,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:51.071032+0000","last_change":"2024-03-19T01:10:32.151875+0000","last_active":"2024-03-19T01:11:51.071032+0000","last_peered":"2024-03-19T01:11:51.071032+0000","last_clean":"2024-03-19T01:11:51.071032+0000","last_became_active":"2024-03-19T01:10:32.151448+0000","last_became_peered":"2024-03-19T01:10:32.151448+0000","last_unstale":"2024-03-19T01:11:51.071032+0000","last_undegraded":"2024-03-19T01:11:51.071032+0000","last_fullsized":"2024-03-19T01:11:51.071032+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:59:17.930078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.155370+0000","last_change":"2024-03-19T01:10:33.184258+0000","last_active":"2024-03-19T01:11:28.155370+0000","last_peered":"2024-03-19T01:11:28.155370+0000","last_clean":"2024-03-19T01:11:28.155370+0000","last_became_active":"2024-03-19T01:10:33.183899+0000","last_became_peered":"2024-03-19T01:10:33.183899+0000","last_unstale":"2024-03-19T01:11:28.155370+0000","last_undegraded":"2024-03-19T01:11:28.155370+0000","last_fullsized":"2024-03-19T01:11:28.155370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:07:47.220818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153044+0000","last_change":"2024-03-19T01:10:32.151998+0000","last_active":"2024-03-19T01:11:28.153044+0000","last_peered":"2024-03-19T01:11:28.153044+0000","last_clean":"2024-03-19T01:11:28.153044+0000","last_became_active":"2024-03-19T01:10:32.151644+0000","last_became_peered":"2024-03-19T01:10:32.151644+0000","last_unstale":"2024-03-19T01:11:28.153044+0000","last_undegraded":"2024-03-19T01:11:28.153044+0000","last_fullsized":"2024-03-19T01:11:28.153044+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:20:28.660705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148188+0000","last_change":"2024-03-19T01:10:33.176759+0000","last_active":"2024-03-19T01:11:28.148188+0000","last_peered":"2024-03-19T01:11:28.148188+0000","last_clean":"2024-03-19T01:11:28.148188+0000","last_became_active":"2024-03-19T01:10:33.176409+0000","last_became_peered":"2024-03-19T01:10:33.176409+0000","last_unstale":"2024-03-19T01:11:28.148188+0000","last_undegraded":"2024-03-19T01:11:28.148188+0000","last_fullsized":"2024-03-19T01:11:28.148188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:21:08.513411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.166861+0000","last_change":"2024-03-19T01:11:28.166417+0000","last_active":"2024-03-19T01:11:28.166861+0000","last_peered":"2024-03-19T01:11:28.166861+0000","last_clean":"2024-03-19T01:11:28.166861+0000","last_became_active":"2024-03-19T01:11:28.155627+0000","last_became_peered":"2024-03-19T01:11:28.155627+0000","last_unstale":"2024-03-19T01:11:28.166861+0000","last_undegraded":"2024-03-19T01:11:28.166861+0000","last_fullsized":"2024-03-19T01:11:28.166861+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:14.701579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.216184+0000","last_change":"2024-03-19T01:10:33.174475+0000","last_active":"2024-03-19T01:11:28.216184+0000","last_peered":"2024-03-19T01:11:28.216184+0000","last_clean":"2024-03-19T01:11:28.216184+0000","last_became_active":"2024-03-19T01:10:33.174154+0000","last_became_peered":"2024-03-19T01:10:33.174154+0000","last_unstale":"2024-03-19T01:11:28.216184+0000","last_undegraded":"2024-03-19T01:11:28.216184+0000","last_fullsized":"2024-03-19T01:11:28.216184+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:44:17.874609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148450+0000","last_change":"2024-03-19T01:10:32.151717+0000","last_active":"2024-03-19T01:11:28.148450+0000","last_peered":"2024-03-19T01:11:28.148450+0000","last_clean":"2024-03-19T01:11:28.148450+0000","last_became_active":"2024-03-19T01:10:32.151356+0000","last_became_peered":"2024-03-19T01:10:32.151356+0000","last_unstale":"2024-03-19T01:11:28.148450+0000","last_undegraded":"2024-03-19T01:11:28.148450+0000","last_fullsized":"2024-03-19T01:11:28.148450+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:02.614226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153048+0000","last_change":"2024-03-19T01:10:33.177202+0000","last_active":"2024-03-19T01:11:28.153048+0000","last_peered":"2024-03-19T01:11:28.153048+0000","last_clean":"2024-03-19T01:11:28.153048+0000","last_became_active":"2024-03-19T01:10:33.172004+0000","last_became_peered":"2024-03-19T01:10:33.172004+0000","last_unstale":"2024-03-19T01:11:28.153048+0000","last_undegraded":"2024-03-19T01:11:28.153048+0000","last_fullsized":"2024-03-19T01:11:28.153048+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:11:36.610997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'5","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:47.062549+0000","last_change":"2024-03-19T01:10:32.151138+0000","last_active":"2024-03-19T01:11:47.062549+0000","last_peered":"2024-03-19T01:11:47.062549+0000","last_clean":"2024-03-19T01:11:47.062549+0000","last_became_active":"2024-03-19T01:10:32.150297+0000","last_became_peered":"2024-03-19T01:10:32.150297+0000","last_unstale":"2024-03-19T01:11:47.062549+0000","last_undegraded":"2024-03-19T01:11:47.062549+0000","last_fullsized":"2024-03-19T01:11:47.062549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:32:37.104986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148422+0000","last_change":"2024-03-19T01:10:33.178878+0000","last_active":"2024-03-19T01:11:28.148422+0000","last_peered":"2024-03-19T01:11:28.148422+0000","last_clean":"2024-03-19T01:11:28.148422+0000","last_became_active":"2024-03-19T01:10:33.178526+0000","last_became_peered":"2024-03-19T01:10:33.178526+0000","last_unstale":"2024-03-19T01:11:28.148422+0000","last_undegraded":"2024-03-19T01:11:28.148422+0000","last_fullsized":"2024-03-19T01:11:28.148422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:40.074065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151640+0000","last_change":"2024-03-19T01:10:32.153255+0000","last_active":"2024-03-19T01:11:28.151640+0000","last_peered":"2024-03-19T01:11:28.151640+0000","last_clean":"2024-03-19T01:11:28.151640+0000","last_became_active":"2024-03-19T01:10:32.152976+0000","last_became_peered":"2024-03-19T01:10:32.152976+0000","last_unstale":"2024-03-19T01:11:28.151640+0000","last_undegraded":"2024-03-19T01:11:28.151640+0000","last_fullsized":"2024-03-19T01:11:28.151640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:51:00.313688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.152041+0000","last_change":"2024-03-19T01:10:33.173710+0000","last_active":"2024-03-19T01:11:28.152041+0000","last_peered":"2024-03-19T01:11:28.152041+0000","last_clean":"2024-03-19T01:11:28.152041+0000","last_became_active":"2024-03-19T01:10:33.173325+0000","last_became_peered":"2024-03-19T01:10:33.173325+0000","last_unstale":"2024-03-19T01:11:28.152041+0000","last_undegraded":"2024-03-19T01:11:28.152041+0000","last_fullsized":"2024-03-19T01:11:28.152041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:05:21.755723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150859+0000","last_change":"2024-03-19T01:10:32.144286+0000","last_active":"2024-03-19T01:11:28.150859+0000","last_peered":"2024-03-19T01:11:28.150859+0000","last_clean":"2024-03-19T01:11:28.150859+0000","last_became_active":"2024-03-19T01:10:32.143959+0000","last_became_peered":"2024-03-19T01:10:32.143959+0000","last_unstale":"2024-03-19T01:11:28.150859+0000","last_undegraded":"2024-03-19T01:11:28.150859+0000","last_fullsized":"2024-03-19T01:11:28.150859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:46:15.750018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153267+0000","last_change":"2024-03-19T01:10:33.178122+0000","last_active":"2024-03-19T01:11:28.153267+0000","last_peered":"2024-03-19T01:11:28.153267+0000","last_clean":"2024-03-19T01:11:28.153267+0000","last_became_active":"2024-03-19T01:10:33.175491+0000","last_became_peered":"2024-03-19T01:10:33.175491+0000","last_unstale":"2024-03-19T01:11:28.153267+0000","last_undegraded":"2024-03-19T01:11:28.153267+0000","last_fullsized":"2024-03-19T01:11:28.153267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:34.418463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151674+0000","last_change":"2024-03-19T01:10:32.147561+0000","last_active":"2024-03-19T01:11:28.151674+0000","last_peered":"2024-03-19T01:11:28.151674+0000","last_clean":"2024-03-19T01:11:28.151674+0000","last_became_active":"2024-03-19T01:10:32.146825+0000","last_became_peered":"2024-03-19T01:10:32.146825+0000","last_unstale":"2024-03-19T01:11:28.151674+0000","last_undegraded":"2024-03-19T01:11:28.151674+0000","last_fullsized":"2024-03-19T01:11:28.151674+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:20:09.833960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150792+0000","last_change":"2024-03-19T01:10:33.170492+0000","last_active":"2024-03-19T01:11:28.150792+0000","last_peered":"2024-03-19T01:11:28.150792+0000","last_clean":"2024-03-19T01:11:28.150792+0000","last_became_active":"2024-03-19T01:10:33.170013+0000","last_became_peered":"2024-03-19T01:10:33.170013+0000","last_unstale":"2024-03-19T01:11:28.150792+0000","last_undegraded":"2024-03-19T01:11:28.150792+0000","last_fullsized":"2024-03-19T01:11:28.150792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:19:13.046663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":162,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:47.177766+0000","last_change":"2024-03-19T01:10:32.147060+0000","last_active":"2024-03-19T01:11:47.177766+0000","last_peered":"2024-03-19T01:11:47.177766+0000","last_clean":"2024-03-19T01:11:47.177766+0000","last_became_active":"2024-03-19T01:10:32.146721+0000","last_became_peered":"2024-03-19T01:10:32.146721+0000","last_unstale":"2024-03-19T01:11:47.177766+0000","last_undegraded":"2024-03-19T01:11:47.177766+0000","last_fullsized":"2024-03-19T01:11:47.177766+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:13:16.930746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":139,"num_read_kb":139,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.216405+0000","last_change":"2024-03-19T01:10:33.174431+0000","last_active":"2024-03-19T01:11:28.216405+0000","last_peered":"2024-03-19T01:11:28.216405+0000","last_clean":"2024-03-19T01:11:28.216405+0000","last_became_active":"2024-03-19T01:10:33.174040+0000","last_became_peered":"2024-03-19T01:10:33.174040+0000","last_unstale":"2024-03-19T01:11:28.216405+0000","last_undegraded":"2024-03-19T01:11:28.216405+0000","last_fullsized":"2024-03-19T01:11:28.216405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:50:53.850400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148335+0000","last_change":"2024-03-19T01:10:32.149194+0000","last_active":"2024-03-19T01:11:28.148335+0000","last_peered":"2024-03-19T01:11:28.148335+0000","last_clean":"2024-03-19T01:11:28.148335+0000","last_became_active":"2024-03-19T01:10:32.148835+0000","last_became_peered":"2024-03-19T01:10:32.148835+0000","last_unstale":"2024-03-19T01:11:28.148335+0000","last_undegraded":"2024-03-19T01:11:28.148335+0000","last_fullsized":"2024-03-19T01:11:28.148335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:07:55.697445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148317+0000","last_change":"2024-03-19T01:10:33.180203+0000","last_active":"2024-03-19T01:11:28.148317+0000","last_peered":"2024-03-19T01:11:28.148317+0000","last_clean":"2024-03-19T01:11:28.148317+0000","last_became_active":"2024-03-19T01:10:33.179812+0000","last_became_peered":"2024-03-19T01:10:33.179812+0000","last_unstale":"2024-03-19T01:11:28.148317+0000","last_undegraded":"2024-03-19T01:11:28.148317+0000","last_fullsized":"2024-03-19T01:11:28.148317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:37:38.371706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153151+0000","last_change":"2024-03-19T01:10:32.153852+0000","last_active":"2024-03-19T01:11:28.153151+0000","last_peered":"2024-03-19T01:11:28.153151+0000","last_clean":"2024-03-19T01:11:28.153151+0000","last_became_active":"2024-03-19T01:10:32.153537+0000","last_became_peered":"2024-03-19T01:10:32.153537+0000","last_unstale":"2024-03-19T01:11:28.153151+0000","last_undegraded":"2024-03-19T01:11:28.153151+0000","last_fullsized":"2024-03-19T01:11:28.153151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:21:48.544344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.216266+0000","last_change":"2024-03-19T01:10:33.175629+0000","last_active":"2024-03-19T01:11:28.216266+0000","last_peered":"2024-03-19T01:11:28.216266+0000","last_clean":"2024-03-19T01:11:28.216266+0000","last_became_active":"2024-03-19T01:10:33.175278+0000","last_became_peered":"2024-03-19T01:10:33.175278+0000","last_unstale":"2024-03-19T01:11:28.216266+0000","last_undegraded":"2024-03-19T01:11:28.216266+0000","last_fullsized":"2024-03-19T01:11:28.216266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:42:31.534216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"85'30","reported_seq":196,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:48.062921+0000","last_change":"2024-03-19T01:10:32.148290+0000","last_active":"2024-03-19T01:11:48.062921+0000","last_peered":"2024-03-19T01:11:48.062921+0000","last_clean":"2024-03-19T01:11:48.062921+0000","last_became_active":"2024-03-19T01:10:32.147940+0000","last_became_peered":"2024-03-19T01:10:32.147940+0000","last_unstale":"2024-03-19T01:11:48.062921+0000","last_undegraded":"2024-03-19T01:11:48.062921+0000","last_fullsized":"2024-03-19T01:11:48.062921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:40:58.125706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61046,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":147,"num_read_kb":133,"num_write":32,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148877+0000","last_change":"2024-03-19T01:10:33.174723+0000","last_active":"2024-03-19T01:11:28.148877+0000","last_peered":"2024-03-19T01:11:28.148877+0000","last_clean":"2024-03-19T01:11:28.148877+0000","last_became_active":"2024-03-19T01:10:33.174369+0000","last_became_peered":"2024-03-19T01:10:33.174369+0000","last_unstale":"2024-03-19T01:11:28.148877+0000","last_undegraded":"2024-03-19T01:11:28.148877+0000","last_fullsized":"2024-03-19T01:11:28.148877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:47:33.641050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150624+0000","last_change":"2024-03-19T01:10:32.146760+0000","last_active":"2024-03-19T01:11:28.150624+0000","last_peered":"2024-03-19T01:11:28.150624+0000","last_clean":"2024-03-19T01:11:28.150624+0000","last_became_active":"2024-03-19T01:10:32.145610+0000","last_became_peered":"2024-03-19T01:10:32.145610+0000","last_unstale":"2024-03-19T01:11:28.150624+0000","last_undegraded":"2024-03-19T01:11:28.150624+0000","last_fullsized":"2024-03-19T01:11:28.150624+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:10:10.959609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.149541+0000","last_change":"2024-03-19T01:10:33.176948+0000","last_active":"2024-03-19T01:11:28.149541+0000","last_peered":"2024-03-19T01:11:28.149541+0000","last_clean":"2024-03-19T01:11:28.149541+0000","last_became_active":"2024-03-19T01:10:33.176604+0000","last_became_peered":"2024-03-19T01:10:33.176604+0000","last_unstale":"2024-03-19T01:11:28.149541+0000","last_undegraded":"2024-03-19T01:11:28.149541+0000","last_fullsized":"2024-03-19T01:11:28.149541+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:45:45.032639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151964+0000","last_change":"2024-03-19T01:10:32.152397+0000","last_active":"2024-03-19T01:11:28.151964+0000","last_peered":"2024-03-19T01:11:28.151964+0000","last_clean":"2024-03-19T01:11:28.151964+0000","last_became_active":"2024-03-19T01:10:32.152026+0000","last_became_peered":"2024-03-19T01:10:32.152026+0000","last_unstale":"2024-03-19T01:11:28.151964+0000","last_undegraded":"2024-03-19T01:11:28.151964+0000","last_fullsized":"2024-03-19T01:11:28.151964+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:18:47.789504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148628+0000","last_change":"2024-03-19T01:10:33.168529+0000","last_active":"2024-03-19T01:11:28.148628+0000","last_peered":"2024-03-19T01:11:28.148628+0000","last_clean":"2024-03-19T01:11:28.148628+0000","last_became_active":"2024-03-19T01:10:33.168198+0000","last_became_peered":"2024-03-19T01:10:33.168198+0000","last_unstale":"2024-03-19T01:11:28.148628+0000","last_undegraded":"2024-03-19T01:11:28.148628+0000","last_fullsized":"2024-03-19T01:11:28.148628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:34:36.030791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"85'6","reported_seq":97,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:47.062503+0000","last_change":"2024-03-19T01:10:32.145565+0000","last_active":"2024-03-19T01:11:47.062503+0000","last_peered":"2024-03-19T01:11:47.062503+0000","last_clean":"2024-03-19T01:11:47.062503+0000","last_became_active":"2024-03-19T01:10:32.145234+0000","last_became_peered":"2024-03-19T01:10:32.145234+0000","last_unstale":"2024-03-19T01:11:47.062503+0000","last_undegraded":"2024-03-19T01:11:47.062503+0000","last_fullsized":"2024-03-19T01:11:47.062503+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:58:35.563136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":73,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148122+0000","last_change":"2024-03-19T01:10:33.182001+0000","last_active":"2024-03-19T01:11:28.148122+0000","last_peered":"2024-03-19T01:11:28.148122+0000","last_clean":"2024-03-19T01:11:28.148122+0000","last_became_active":"2024-03-19T01:10:33.181638+0000","last_became_peered":"2024-03-19T01:10:33.181638+0000","last_unstale":"2024-03-19T01:11:28.148122+0000","last_undegraded":"2024-03-19T01:11:28.148122+0000","last_fullsized":"2024-03-19T01:11:28.148122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:59:23.155190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148157+0000","last_change":"2024-03-19T01:10:32.154159+0000","last_active":"2024-03-19T01:11:28.148157+0000","last_peered":"2024-03-19T01:11:28.148157+0000","last_clean":"2024-03-19T01:11:28.148157+0000","last_became_active":"2024-03-19T01:10:32.153853+0000","last_became_peered":"2024-03-19T01:10:32.153853+0000","last_unstale":"2024-03-19T01:11:28.148157+0000","last_undegraded":"2024-03-19T01:11:28.148157+0000","last_fullsized":"2024-03-19T01:11:28.148157+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:41:13.727588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.217014+0000","last_change":"2024-03-19T01:11:28.217014+0000","last_active":"2024-03-19T01:11:28.217014+0000","last_peered":"2024-03-19T01:11:28.217014+0000","last_clean":"2024-03-19T01:11:28.217014+0000","last_became_active":"2024-03-19T01:11:28.216608+0000","last_became_peered":"2024-03-19T01:11:28.216608+0000","last_unstale":"2024-03-19T01:11:28.217014+0000","last_undegraded":"2024-03-19T01:11:28.217014+0000","last_fullsized":"2024-03-19T01:11:28.217014+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:47.600606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.149001+0000","last_change":"2024-03-19T01:10:32.144855+0000","last_active":"2024-03-19T01:11:28.149001+0000","last_peered":"2024-03-19T01:11:28.149001+0000","last_clean":"2024-03-19T01:11:28.149001+0000","last_became_active":"2024-03-19T01:10:32.144540+0000","last_became_peered":"2024-03-19T01:10:32.144540+0000","last_unstale":"2024-03-19T01:11:28.149001+0000","last_undegraded":"2024-03-19T01:11:28.149001+0000","last_fullsized":"2024-03-19T01:11:28.149001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:23:22.399954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153809+0000","last_change":"2024-03-19T01:10:33.172739+0000","last_active":"2024-03-19T01:11:28.153809+0000","last_peered":"2024-03-19T01:11:28.153809+0000","last_clean":"2024-03-19T01:11:28.153809+0000","last_became_active":"2024-03-19T01:10:33.172391+0000","last_became_peered":"2024-03-19T01:10:33.172391+0000","last_unstale":"2024-03-19T01:11:28.153809+0000","last_undegraded":"2024-03-19T01:11:28.153809+0000","last_fullsized":"2024-03-19T01:11:28.153809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:30.619013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151401+0000","last_change":"2024-03-19T01:10:32.144491+0000","last_active":"2024-03-19T01:11:28.151401+0000","last_peered":"2024-03-19T01:11:28.151401+0000","last_clean":"2024-03-19T01:11:28.151401+0000","last_became_active":"2024-03-19T01:10:32.144170+0000","last_became_peered":"2024-03-19T01:10:32.144170+0000","last_unstale":"2024-03-19T01:11:28.151401+0000","last_undegraded":"2024-03-19T01:11:28.151401+0000","last_fullsized":"2024-03-19T01:11:28.151401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:07.998818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151653+0000","last_change":"2024-03-19T01:10:33.175479+0000","last_active":"2024-03-19T01:11:28.151653+0000","last_peered":"2024-03-19T01:11:28.151653+0000","last_clean":"2024-03-19T01:11:28.151653+0000","last_became_active":"2024-03-19T01:10:33.172530+0000","last_became_peered":"2024-03-19T01:10:33.172530+0000","last_unstale":"2024-03-19T01:11:28.151653+0000","last_undegraded":"2024-03-19T01:11:28.151653+0000","last_fullsized":"2024-03-19T01:11:28.151653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:16.058826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":97,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:49.069507+0000","last_change":"2024-03-19T01:10:32.143629+0000","last_active":"2024-03-19T01:11:49.069507+0000","last_peered":"2024-03-19T01:11:49.069507+0000","last_clean":"2024-03-19T01:11:49.069507+0000","last_became_active":"2024-03-19T01:10:32.143321+0000","last_became_peered":"2024-03-19T01:10:32.143321+0000","last_unstale":"2024-03-19T01:11:49.069507+0000","last_undegraded":"2024-03-19T01:11:49.069507+0000","last_fullsized":"2024-03-19T01:11:49.069507+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:47.404245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151790+0000","last_change":"2024-03-19T01:10:33.171284+0000","last_active":"2024-03-19T01:11:28.151790+0000","last_peered":"2024-03-19T01:11:28.151790+0000","last_clean":"2024-03-19T01:11:28.151790+0000","last_became_active":"2024-03-19T01:10:33.168717+0000","last_became_peered":"2024-03-19T01:10:33.168717+0000","last_unstale":"2024-03-19T01:11:28.151790+0000","last_undegraded":"2024-03-19T01:11:28.151790+0000","last_fullsized":"2024-03-19T01:11:28.151790+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:38:52.335303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.152064+0000","last_change":"2024-03-19T01:10:33.177261+0000","last_active":"2024-03-19T01:11:28.152064+0000","last_peered":"2024-03-19T01:11:28.152064+0000","last_clean":"2024-03-19T01:11:28.152064+0000","last_became_active":"2024-03-19T01:10:33.171836+0000","last_became_peered":"2024-03-19T01:10:33.171836+0000","last_unstale":"2024-03-19T01:11:28.152064+0000","last_undegraded":"2024-03-19T01:11:28.152064+0000","last_fullsized":"2024-03-19T01:11:28.152064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:49:14.577925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:48.062974+0000","last_change":"2024-03-19T01:10:32.141587+0000","last_active":"2024-03-19T01:11:48.062974+0000","last_peered":"2024-03-19T01:11:48.062974+0000","last_clean":"2024-03-19T01:11:48.062974+0000","last_became_active":"2024-03-19T01:10:32.141236+0000","last_became_peered":"2024-03-19T01:10:32.141236+0000","last_unstale":"2024-03-19T01:11:48.062974+0000","last_undegraded":"2024-03-19T01:11:48.062974+0000","last_fullsized":"2024-03-19T01:11:48.062974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T05:07:40.685399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.216813+0000","last_change":"2024-03-19T01:11:28.216813+0000","last_active":"2024-03-19T01:11:28.216813+0000","last_peered":"2024-03-19T01:11:28.216813+0000","last_clean":"2024-03-19T01:11:28.216813+0000","last_became_active":"2024-03-19T01:11:28.216483+0000","last_became_peered":"2024-03-19T01:11:28.216483+0000","last_unstale":"2024-03-19T01:11:28.216813+0000","last_undegraded":"2024-03-19T01:11:28.216813+0000","last_fullsized":"2024-03-19T01:11:28.216813+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:38:49.974387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150951+0000","last_change":"2024-03-19T01:10:32.145816+0000","last_active":"2024-03-19T01:11:28.150951+0000","last_peered":"2024-03-19T01:11:28.150951+0000","last_clean":"2024-03-19T01:11:28.150951+0000","last_became_active":"2024-03-19T01:10:32.145055+0000","last_became_peered":"2024-03-19T01:10:32.145055+0000","last_unstale":"2024-03-19T01:11:28.150951+0000","last_undegraded":"2024-03-19T01:11:28.150951+0000","last_fullsized":"2024-03-19T01:11:28.150951+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:26:56.428854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":342,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148180+0000","last_change":"2024-03-19T01:09:20.932686+0000","last_active":"2024-03-19T01:11:28.148180+0000","last_peered":"2024-03-19T01:11:28.148180+0000","last_clean":"2024-03-19T01:11:28.148180+0000","last_became_active":"2024-03-19T01:09:20.932170+0000","last_became_peered":"2024-03-19T01:09:20.932170+0000","last_unstale":"2024-03-19T01:11:28.148180+0000","last_undegraded":"2024-03-19T01:11:28.148180+0000","last_fullsized":"2024-03-19T01:11:28.148180+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:05:53.937682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:05:53.937682+0000","last_clean_scrub_stamp":"2024-03-19T01:05:53.937682+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:48:41.523200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150683+0000","last_change":"2024-03-19T01:10:33.173230+0000","last_active":"2024-03-19T01:11:28.150683+0000","last_peered":"2024-03-19T01:11:28.150683+0000","last_clean":"2024-03-19T01:11:28.150683+0000","last_became_active":"2024-03-19T01:10:33.172858+0000","last_became_peered":"2024-03-19T01:10:33.172858+0000","last_unstale":"2024-03-19T01:11:28.150683+0000","last_undegraded":"2024-03-19T01:11:28.150683+0000","last_fullsized":"2024-03-19T01:11:28.150683+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:28:43.582211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148306+0000","last_change":"2024-03-19T01:10:32.147419+0000","last_active":"2024-03-19T01:11:28.148306+0000","last_peered":"2024-03-19T01:11:28.148306+0000","last_clean":"2024-03-19T01:11:28.148306+0000","last_became_active":"2024-03-19T01:10:32.147111+0000","last_became_peered":"2024-03-19T01:10:32.147111+0000","last_unstale":"2024-03-19T01:11:28.148306+0000","last_undegraded":"2024-03-19T01:11:28.148306+0000","last_fullsized":"2024-03-19T01:11:28.148306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:45:34.989763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148988+0000","last_change":"2024-03-19T01:10:32.154233+0000","last_active":"2024-03-19T01:11:28.148988+0000","last_peered":"2024-03-19T01:11:28.148988+0000","last_clean":"2024-03-19T01:11:28.148988+0000","last_became_active":"2024-03-19T01:10:32.153939+0000","last_became_peered":"2024-03-19T01:10:32.153939+0000","last_unstale":"2024-03-19T01:11:28.148988+0000","last_undegraded":"2024-03-19T01:11:28.148988+0000","last_fullsized":"2024-03-19T01:11:28.148988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:26:45.377161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151669+0000","last_change":"2024-03-19T01:10:33.168395+0000","last_active":"2024-03-19T01:11:28.151669+0000","last_peered":"2024-03-19T01:11:28.151669+0000","last_clean":"2024-03-19T01:11:28.151669+0000","last_became_active":"2024-03-19T01:10:33.168078+0000","last_became_peered":"2024-03-19T01:10:33.168078+0000","last_unstale":"2024-03-19T01:11:28.151669+0000","last_undegraded":"2024-03-19T01:11:28.151669+0000","last_fullsized":"2024-03-19T01:11:28.151669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:54:52.597591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'13","reported_seq":48,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.165836+0000","last_change":"2024-03-19T01:11:28.165836+0000","last_active":"2024-03-19T01:11:28.165836+0000","last_peered":"2024-03-19T01:11:28.165836+0000","last_clean":"2024-03-19T01:11:28.165836+0000","last_became_active":"2024-03-19T01:11:28.158005+0000","last_became_peered":"2024-03-19T01:11:28.158005+0000","last_unstale":"2024-03-19T01:11:28.165836+0000","last_undegraded":"2024-03-19T01:11:28.165836+0000","last_fullsized":"2024-03-19T01:11:28.165836+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:35:10.440755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.155960+0000","last_change":"2024-03-19T01:10:33.174635+0000","last_active":"2024-03-19T01:11:28.155960+0000","last_peered":"2024-03-19T01:11:28.155960+0000","last_clean":"2024-03-19T01:11:28.155960+0000","last_became_active":"2024-03-19T01:10:33.173970+0000","last_became_peered":"2024-03-19T01:10:33.173970+0000","last_unstale":"2024-03-19T01:11:28.155960+0000","last_undegraded":"2024-03-19T01:11:28.155960+0000","last_fullsized":"2024-03-19T01:11:28.155960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:16.174169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150856+0000","last_change":"2024-03-19T01:10:32.148225+0000","last_active":"2024-03-19T01:11:28.150856+0000","last_peered":"2024-03-19T01:11:28.150856+0000","last_clean":"2024-03-19T01:11:28.150856+0000","last_became_active":"2024-03-19T01:10:32.147889+0000","last_became_peered":"2024-03-19T01:10:32.147889+0000","last_unstale":"2024-03-19T01:11:28.150856+0000","last_undegraded":"2024-03-19T01:11:28.150856+0000","last_fullsized":"2024-03-19T01:11:28.150856+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:30:56.037018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148136+0000","last_change":"2024-03-19T01:10:33.172925+0000","last_active":"2024-03-19T01:11:28.148136+0000","last_peered":"2024-03-19T01:11:28.148136+0000","last_clean":"2024-03-19T01:11:28.148136+0000","last_became_active":"2024-03-19T01:10:33.172572+0000","last_became_peered":"2024-03-19T01:10:33.172572+0000","last_unstale":"2024-03-19T01:11:28.148136+0000","last_undegraded":"2024-03-19T01:11:28.148136+0000","last_fullsized":"2024-03-19T01:11:28.148136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:17:02.853337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.215893+0000","last_change":"2024-03-19T01:10:32.146245+0000","last_active":"2024-03-19T01:11:28.215893+0000","last_peered":"2024-03-19T01:11:28.215893+0000","last_clean":"2024-03-19T01:11:28.215893+0000","last_became_active":"2024-03-19T01:10:32.145877+0000","last_became_peered":"2024-03-19T01:10:32.145877+0000","last_unstale":"2024-03-19T01:11:28.215893+0000","last_undegraded":"2024-03-19T01:11:28.215893+0000","last_fullsized":"2024-03-19T01:11:28.215893+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:24:45.323155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148180+0000","last_change":"2024-03-19T01:10:33.178538+0000","last_active":"2024-03-19T01:11:28.148180+0000","last_peered":"2024-03-19T01:11:28.148180+0000","last_clean":"2024-03-19T01:11:28.148180+0000","last_became_active":"2024-03-19T01:10:33.178170+0000","last_became_peered":"2024-03-19T01:10:33.178170+0000","last_unstale":"2024-03-19T01:11:28.148180+0000","last_undegraded":"2024-03-19T01:11:28.148180+0000","last_fullsized":"2024-03-19T01:11:28.148180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:49:41.913715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153566+0000","last_change":"2024-03-19T01:10:32.147697+0000","last_active":"2024-03-19T01:11:28.153566+0000","last_peered":"2024-03-19T01:11:28.153566+0000","last_clean":"2024-03-19T01:11:28.153566+0000","last_became_active":"2024-03-19T01:10:32.147399+0000","last_became_peered":"2024-03-19T01:10:32.147399+0000","last_unstale":"2024-03-19T01:11:28.153566+0000","last_undegraded":"2024-03-19T01:11:28.153566+0000","last_fullsized":"2024-03-19T01:11:28.153566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:33:38.961646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151301+0000","last_change":"2024-03-19T01:10:33.175990+0000","last_active":"2024-03-19T01:11:28.151301+0000","last_peered":"2024-03-19T01:11:28.151301+0000","last_clean":"2024-03-19T01:11:28.151301+0000","last_became_active":"2024-03-19T01:10:33.175539+0000","last_became_peered":"2024-03-19T01:10:33.175539+0000","last_unstale":"2024-03-19T01:11:28.151301+0000","last_undegraded":"2024-03-19T01:11:28.151301+0000","last_fullsized":"2024-03-19T01:11:28.151301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:00:36.799131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150742+0000","last_change":"2024-03-19T01:10:32.151834+0000","last_active":"2024-03-19T01:11:28.150742+0000","last_peered":"2024-03-19T01:11:28.150742+0000","last_clean":"2024-03-19T01:11:28.150742+0000","last_became_active":"2024-03-19T01:10:32.151514+0000","last_became_peered":"2024-03-19T01:10:32.151514+0000","last_unstale":"2024-03-19T01:11:28.150742+0000","last_undegraded":"2024-03-19T01:11:28.150742+0000","last_fullsized":"2024-03-19T01:11:28.150742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T11:40:16.839334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.154035+0000","last_change":"2024-03-19T01:10:33.185206+0000","last_active":"2024-03-19T01:11:28.154035+0000","last_peered":"2024-03-19T01:11:28.154035+0000","last_clean":"2024-03-19T01:11:28.154035+0000","last_became_active":"2024-03-19T01:10:33.184901+0000","last_became_peered":"2024-03-19T01:10:33.184901+0000","last_unstale":"2024-03-19T01:11:28.154035+0000","last_undegraded":"2024-03-19T01:11:28.154035+0000","last_fullsized":"2024-03-19T01:11:28.154035+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:54:47.266255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.220475+0000","last_change":"2024-03-19T01:11:28.220475+0000","last_active":"2024-03-19T01:11:28.220475+0000","last_peered":"2024-03-19T01:11:28.220475+0000","last_clean":"2024-03-19T01:11:28.220475+0000","last_became_active":"2024-03-19T01:11:28.217201+0000","last_became_peered":"2024-03-19T01:11:28.217201+0000","last_unstale":"2024-03-19T01:11:28.220475+0000","last_undegraded":"2024-03-19T01:11:28.220475+0000","last_fullsized":"2024-03-19T01:11:28.220475+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:39:14.366839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,7],"acting":[8,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148764+0000","last_change":"2024-03-19T01:10:33.181232+0000","last_active":"2024-03-19T01:11:28.148764+0000","last_peered":"2024-03-19T01:11:28.148764+0000","last_clean":"2024-03-19T01:11:28.148764+0000","last_became_active":"2024-03-19T01:10:33.180719+0000","last_became_peered":"2024-03-19T01:10:33.180719+0000","last_unstale":"2024-03-19T01:11:28.148764+0000","last_undegraded":"2024-03-19T01:11:28.148764+0000","last_fullsized":"2024-03-19T01:11:28.148764+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:33:43.201965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"85'5","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:47.178182+0000","last_change":"2024-03-19T01:11:28.160961+0000","last_active":"2024-03-19T01:11:47.178182+0000","last_peered":"2024-03-19T01:11:47.178182+0000","last_clean":"2024-03-19T01:11:47.178182+0000","last_became_active":"2024-03-19T01:11:28.155196+0000","last_became_peered":"2024-03-19T01:11:28.155196+0000","last_unstale":"2024-03-19T01:11:47.178182+0000","last_undegraded":"2024-03-19T01:11:47.178182+0000","last_fullsized":"2024-03-19T01:11:47.178182+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:46:38.586516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.149065+0000","last_change":"2024-03-19T01:10:33.178129+0000","last_active":"2024-03-19T01:11:28.149065+0000","last_peered":"2024-03-19T01:11:28.149065+0000","last_clean":"2024-03-19T01:11:28.149065+0000","last_became_active":"2024-03-19T01:10:33.177802+0000","last_became_peered":"2024-03-19T01:10:33.177802+0000","last_unstale":"2024-03-19T01:11:28.149065+0000","last_undegraded":"2024-03-19T01:11:28.149065+0000","last_fullsized":"2024-03-19T01:11:28.149065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T09:46:42.698865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.152934+0000","last_change":"2024-03-19T01:10:32.149613+0000","last_active":"2024-03-19T01:11:28.152934+0000","last_peered":"2024-03-19T01:11:28.152934+0000","last_clean":"2024-03-19T01:11:28.152934+0000","last_became_active":"2024-03-19T01:10:32.147651+0000","last_became_peered":"2024-03-19T01:10:32.147651+0000","last_unstale":"2024-03-19T01:11:28.152934+0000","last_undegraded":"2024-03-19T01:11:28.152934+0000","last_fullsized":"2024-03-19T01:11:28.152934+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:06:59.222944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.168072+0000","last_change":"2024-03-19T01:11:28.168072+0000","last_active":"2024-03-19T01:11:28.168072+0000","last_peered":"2024-03-19T01:11:28.168072+0000","last_clean":"2024-03-19T01:11:28.168072+0000","last_became_active":"2024-03-19T01:11:28.157973+0000","last_became_peered":"2024-03-19T01:11:28.157973+0000","last_unstale":"2024-03-19T01:11:28.168072+0000","last_undegraded":"2024-03-19T01:11:28.168072+0000","last_fullsized":"2024-03-19T01:11:28.168072+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:24:14.879683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150006+0000","last_change":"2024-03-19T01:10:32.149253+0000","last_active":"2024-03-19T01:11:28.150006+0000","last_peered":"2024-03-19T01:11:28.150006+0000","last_clean":"2024-03-19T01:11:28.150006+0000","last_became_active":"2024-03-19T01:10:32.148520+0000","last_became_peered":"2024-03-19T01:10:32.148520+0000","last_unstale":"2024-03-19T01:11:28.150006+0000","last_undegraded":"2024-03-19T01:11:28.150006+0000","last_fullsized":"2024-03-19T01:11:28.150006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:31.120189+0000","last_clean_scrub_stamp":"2024-03-19T01:10:31.120189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:10:21.211510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148503+0000","last_change":"2024-03-19T01:10:33.182684+0000","last_active":"2024-03-19T01:11:28.148503+0000","last_peered":"2024-03-19T01:11:28.148503+0000","last_clean":"2024-03-19T01:11:28.148503+0000","last_became_active":"2024-03-19T01:10:33.182269+0000","last_became_peered":"2024-03-19T01:10:33.182269+0000","last_unstale":"2024-03-19T01:11:28.148503+0000","last_undegraded":"2024-03-19T01:11:28.148503+0000","last_fullsized":"2024-03-19T01:11:28.148503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:49:33.318462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151402+0000","last_change":"2024-03-19T01:10:33.172795+0000","last_active":"2024-03-19T01:11:28.151402+0000","last_peered":"2024-03-19T01:11:28.151402+0000","last_clean":"2024-03-19T01:11:28.151402+0000","last_became_active":"2024-03-19T01:10:33.172483+0000","last_became_peered":"2024-03-19T01:10:33.172483+0000","last_unstale":"2024-03-19T01:11:28.151402+0000","last_undegraded":"2024-03-19T01:11:28.151402+0000","last_fullsized":"2024-03-19T01:11:28.151402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:38:36.910222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.156200+0000","last_change":"2024-03-19T01:11:28.156200+0000","last_active":"2024-03-19T01:11:28.156200+0000","last_peered":"2024-03-19T01:11:28.156200+0000","last_clean":"2024-03-19T01:11:28.156200+0000","last_became_active":"2024-03-19T01:11:28.155824+0000","last_became_peered":"2024-03-19T01:11:28.155824+0000","last_unstale":"2024-03-19T01:11:28.156200+0000","last_undegraded":"2024-03-19T01:11:28.156200+0000","last_fullsized":"2024-03-19T01:11:28.156200+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:54:20.895403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150650+0000","last_change":"2024-03-19T01:10:33.178847+0000","last_active":"2024-03-19T01:11:28.150650+0000","last_peered":"2024-03-19T01:11:28.150650+0000","last_clean":"2024-03-19T01:11:28.150650+0000","last_became_active":"2024-03-19T01:10:33.178533+0000","last_became_peered":"2024-03-19T01:10:33.178533+0000","last_unstale":"2024-03-19T01:11:28.150650+0000","last_undegraded":"2024-03-19T01:11:28.150650+0000","last_fullsized":"2024-03-19T01:11:28.150650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T10:52:15.401797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.148064+0000","last_change":"2024-03-19T01:10:33.169936+0000","last_active":"2024-03-19T01:11:28.148064+0000","last_peered":"2024-03-19T01:11:28.148064+0000","last_clean":"2024-03-19T01:11:28.148064+0000","last_became_active":"2024-03-19T01:10:33.169594+0000","last_became_peered":"2024-03-19T01:10:33.169594+0000","last_unstale":"2024-03-19T01:11:28.148064+0000","last_undegraded":"2024-03-19T01:11:28.148064+0000","last_fullsized":"2024-03-19T01:11:28.148064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:41.932575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":23,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.154178+0000","last_change":"2024-03-19T01:11:28.154178+0000","last_active":"2024-03-19T01:11:28.154178+0000","last_peered":"2024-03-19T01:11:28.154178+0000","last_clean":"2024-03-19T01:11:28.154178+0000","last_became_active":"2024-03-19T01:11:28.153739+0000","last_became_peered":"2024-03-19T01:11:28.153739+0000","last_unstale":"2024-03-19T01:11:28.154178+0000","last_undegraded":"2024-03-19T01:11:28.154178+0000","last_fullsized":"2024-03-19T01:11:28.154178+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:48:37.480130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.154271+0000","last_change":"2024-03-19T01:10:33.178699+0000","last_active":"2024-03-19T01:11:28.154271+0000","last_peered":"2024-03-19T01:11:28.154271+0000","last_clean":"2024-03-19T01:11:28.154271+0000","last_became_active":"2024-03-19T01:10:33.178380+0000","last_became_peered":"2024-03-19T01:10:33.178380+0000","last_unstale":"2024-03-19T01:11:28.154271+0000","last_undegraded":"2024-03-19T01:11:28.154271+0000","last_fullsized":"2024-03-19T01:11:28.154271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T04:46:29.502416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.150752+0000","last_change":"2024-03-19T01:10:33.179908+0000","last_active":"2024-03-19T01:11:28.150752+0000","last_peered":"2024-03-19T01:11:28.150752+0000","last_clean":"2024-03-19T01:11:28.150752+0000","last_became_active":"2024-03-19T01:10:33.179578+0000","last_became_peered":"2024-03-19T01:10:33.179578+0000","last_unstale":"2024-03-19T01:11:28.150752+0000","last_undegraded":"2024-03-19T01:11:28.150752+0000","last_fullsized":"2024-03-19T01:11:28.150752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T12:30:53.918078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153979+0000","last_change":"2024-03-19T01:10:33.179788+0000","last_active":"2024-03-19T01:11:28.153979+0000","last_peered":"2024-03-19T01:11:28.153979+0000","last_clean":"2024-03-19T01:11:28.153979+0000","last_became_active":"2024-03-19T01:10:33.179473+0000","last_became_peered":"2024-03-19T01:10:33.179473+0000","last_unstale":"2024-03-19T01:11:28.153979+0000","last_undegraded":"2024-03-19T01:11:28.153979+0000","last_fullsized":"2024-03-19T01:11:28.153979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T08:24:36.284768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.151690+0000","last_change":"2024-03-19T01:10:33.172698+0000","last_active":"2024-03-19T01:11:28.151690+0000","last_peered":"2024-03-19T01:11:28.151690+0000","last_clean":"2024-03-19T01:11:28.151690+0000","last_became_active":"2024-03-19T01:10:33.172369+0000","last_became_peered":"2024-03-19T01:10:33.172369+0000","last_unstale":"2024-03-19T01:11:28.151690+0000","last_undegraded":"2024-03-19T01:11:28.151690+0000","last_fullsized":"2024-03-19T01:11:28.151690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:09:18.961724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.153491+0000","last_change":"2024-03-19T01:10:33.184350+0000","last_active":"2024-03-19T01:11:28.153491+0000","last_peered":"2024-03-19T01:11:28.153491+0000","last_clean":"2024-03-19T01:11:28.153491+0000","last_became_active":"2024-03-19T01:10:33.180533+0000","last_became_peered":"2024-03-19T01:10:33.180533+0000","last_unstale":"2024-03-19T01:11:28.153491+0000","last_undegraded":"2024-03-19T01:11:28.153491+0000","last_fullsized":"2024-03-19T01:11:28.153491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:41:28.003560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.216073+0000","last_change":"2024-03-19T01:10:33.180396+0000","last_active":"2024-03-19T01:11:28.216073+0000","last_peered":"2024-03-19T01:11:28.216073+0000","last_clean":"2024-03-19T01:11:28.216073+0000","last_became_active":"2024-03-19T01:10:33.180062+0000","last_became_peered":"2024-03-19T01:10:33.180062+0000","last_unstale":"2024-03-19T01:11:28.216073+0000","last_undegraded":"2024-03-19T01:11:28.216073+0000","last_fullsized":"2024-03-19T01:11:28.216073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:43:15.429623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.149893+0000","last_change":"2024-03-19T01:10:33.167261+0000","last_active":"2024-03-19T01:11:28.149893+0000","last_peered":"2024-03-19T01:11:28.149893+0000","last_clean":"2024-03-19T01:11:28.149893+0000","last_became_active":"2024-03-19T01:10:33.166943+0000","last_became_peered":"2024-03-19T01:10:33.166943+0000","last_unstale":"2024-03-19T01:11:28.149893+0000","last_undegraded":"2024-03-19T01:11:28.149893+0000","last_fullsized":"2024-03-19T01:11:28.149893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T07:01:44.451161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-19T01:11:28.152915+0000","last_change":"2024-03-19T01:10:33.175750+0000","last_active":"2024-03-19T01:11:28.152915+0000","last_peered":"2024-03-19T01:11:28.152915+0000","last_clean":"2024-03-19T01:11:28.152915+0000","last_became_active":"2024-03-19T01:10:33.175411+0000","last_became_peered":"2024-03-19T01:10:33.175411+0000","last_unstale":"2024-03-19T01:11:28.152915+0000","last_undegraded":"2024-03-19T01:11:28.152915+0000","last_fullsized":"2024-03-19T01:11:28.152915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-19T01:10:32.128012+0000","last_clean_scrub_stamp":"2024-03-19T01:10:32.128012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T06:20:41.113626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284643,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67859,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":690,"num_read_kb":443,"num_write":238,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1170,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":203577,"data_compressed":5559,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383709,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":950272,"data_stored":457013,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645346,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1101824,"data_stored":1109479,"data_compressed":43833,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808872,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":456943,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070509,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":457321,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332146,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36344,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708808,"statfs":{"total":95995035648,"available":95957819392,"internally_reserved":0,"allocated":1024000,"data_stored":1049839,"data_compressed":41980,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495671,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":456777,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":518619,"data_compressed":7638,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":457999,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1028096,"data_stored":517669,"data_compressed":7638,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":974848,"data_stored":456485,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920833,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":458082,"data_compressed":5785,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757308,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":1049009,"data_compressed":41980,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34556,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34556,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34556,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36195,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":62798,"data_compressed":1853,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":61848,"data_compressed":1853,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":63173,"data_compressed":1853,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-19T01:11:54.101 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-19T01:11:54.101 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-19T01:11:54.101 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-19T01:11:54.102 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-19T01:11:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:11:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:54 smithi162 ceph-mon[67178]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:11:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:54 smithi116 ceph-mon[62701]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:11:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:54 smithi156 ceph-mon[66799]: pgmap v330: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:11:55.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:55 smithi162 ceph-mon[67178]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:11:55.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:55 smithi116 ceph-mon[62701]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:11:55.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:55 smithi156 ceph-mon[66799]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-19T01:11:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.038+0000 7f1c1dae7700 1 -- 172.21.15.116:0/2035931143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c18101bb0 msgr2=0x7f1c1810e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.038+0000 7f1c1dae7700 1 --2- 172.21.15.116:0/2035931143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c18101bb0 0x7f1c1810e090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009990 tx=0x7f1c08009ca0 comp rx=0 tx=0).stop 2024-03-19T01:11:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.038+0000 7f1c1dae7700 1 -- 172.21.15.116:0/2035931143 shutdown_connections 2024-03-19T01:11:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.038+0000 7f1c1dae7700 1 --2- 172.21.15.116:0/2035931143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c18101bb0 0x7f1c1810e090 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:56.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.038+0000 7f1c1dae7700 1 --2- 172.21.15.116:0/2035931143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c18107250 0x7f1c18107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:56.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.038+0000 7f1c1dae7700 1 --2- 172.21.15.116:0/2035931143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c18101210 0x7f1c18101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:56.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.038+0000 7f1c1dae7700 1 -- 172.21.15.116:0/2035931143 >> 172.21.15.116:0/2035931143 conn(0x7f1c18078140 msgr2=0x7f1c18078550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:56.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.038+0000 7f1c1dae7700 1 -- 172.21.15.116:0/2035931143 shutdown_connections 2024-03-19T01:11:56.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.038+0000 7f1c1dae7700 1 -- 172.21.15.116:0/2035931143 wait complete. 2024-03-19T01:11:56.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.039+0000 7f1c1dae7700 1 Processor -- start 2024-03-19T01:11:56.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.039+0000 7f1c1dae7700 1 -- start start 2024-03-19T01:11:56.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.039+0000 7f1c1dae7700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c18101210 0x7f1c18102ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:56.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.039+0000 7f1c1dae7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c18101bb0 0x7f1c18103400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:56.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.039+0000 7f1c1dae7700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c18107250 0x7f1c18103940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:56.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.039+0000 7f1c1dae7700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c18111520 con 0x7f1c18107250 2024-03-19T01:11:56.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.039+0000 7f1c1dae7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1c181113a0 con 0x7f1c18101bb0 2024-03-19T01:11:56.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.039+0000 7f1c1dae7700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1c181116a0 con 0x7f1c18101210 2024-03-19T01:11:56.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c16ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c18101bb0 0x7f1c18103400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c16ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c18101bb0 0x7f1c18103400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:34570/0 (socket says 172.21.15.116:34570) 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c16ffd700 1 -- 172.21.15.116:0/14176190 learned_addr learned my addr 172.21.15.116:0/14176190 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c177fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c18101210 0x7f1c18102ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c17fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c18107250 0x7f1c18103940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c16ffd700 1 -- 172.21.15.116:0/14176190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c18101210 msgr2=0x7f1c18102ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c16ffd700 1 --2- 172.21.15.116:0/14176190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c18101210 0x7f1c18102ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c16ffd700 1 -- 172.21.15.116:0/14176190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c18107250 msgr2=0x7f1c18103940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c16ffd700 1 --2- 172.21.15.116:0/14176190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c18107250 0x7f1c18103940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c16ffd700 1 -- 172.21.15.116:0/14176190 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c181a6fa0 con 0x7f1c18101bb0 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c17fff700 1 --2- 172.21.15.116:0/14176190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c18107250 0x7f1c18103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c16ffd700 1 --2- 172.21.15.116:0/14176190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c18101bb0 0x7f1c18103400 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00ef10 tx=0x7f1c0c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c14ff9700 1 -- 172.21.15.116:0/14176190 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c00c9c0 con 0x7f1c18101bb0 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c1dae7700 1 -- 172.21.15.116:0/14176190 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c181a7230 con 0x7f1c18101bb0 2024-03-19T01:11:56.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c14ff9700 1 -- 172.21.15.116:0/14176190 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c0c019070 con 0x7f1c18101bb0 2024-03-19T01:11:56.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.040+0000 7f1c14ff9700 1 -- 172.21.15.116:0/14176190 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c007e60 con 0x7f1c18101bb0 2024-03-19T01:11:56.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.041+0000 7f1c1dae7700 1 -- 172.21.15.116:0/14176190 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c181a77c0 con 0x7f1c18101bb0 2024-03-19T01:11:56.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.041+0000 7f1c1dae7700 1 -- 172.21.15.116:0/14176190 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c18068fd0 con 0x7f1c18101bb0 2024-03-19T01:11:56.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.042+0000 7f1c14ff9700 1 -- 172.21.15.116:0/14176190 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f1c0c012050 con 0x7f1c18101bb0 2024-03-19T01:11:56.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.043+0000 7f1c14ff9700 1 --2- 172.21.15.116:0/14176190 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1c000a29a0 0x7f1c000a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:56.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.043+0000 7f1c177fe700 1 --2- 172.21.15.116:0/14176190 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1c000a29a0 0x7f1c000a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:56.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.043+0000 7f1c14ff9700 1 -- 172.21.15.116:0/14176190 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1c0c0cebf0 con 0x7f1c18101bb0 2024-03-19T01:11:56.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.043+0000 7f1c177fe700 1 --2- 172.21.15.116:0/14176190 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1c000a29a0 0x7f1c000a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1c10009be0 tx=0x7f1c10009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:56.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.051+0000 7f1c14ff9700 1 -- 172.21.15.116:0/14176190 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1c0c010040 con 0x7f1c18101bb0 2024-03-19T01:11:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.331+0000 7f1c1dae7700 1 -- 172.21.15.116:0/14176190 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f1c181043f0 con 0x7f1c18101bb0 2024-03-19T01:11:56.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.331+0000 7f1c14ff9700 1 -- 172.21.15.116:0/14176190 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f1c181043f0 con 0x7f1c18101bb0 2024-03-19T01:11:56.332 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:11:56.332 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-19T01:11:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.333+0000 7f1c1dae7700 1 -- 172.21.15.116:0/14176190 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1c000a29a0 msgr2=0x7f1c000a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.333+0000 7f1c1dae7700 1 --2- 172.21.15.116:0/14176190 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1c000a29a0 0x7f1c000a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1c10009be0 tx=0x7f1c10009260 comp rx=0 tx=0).stop 2024-03-19T01:11:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.333+0000 7f1c1dae7700 1 -- 172.21.15.116:0/14176190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c18101bb0 msgr2=0x7f1c18103400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.333+0000 7f1c1dae7700 1 --2- 172.21.15.116:0/14176190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c18101bb0 0x7f1c18103400 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00ef10 tx=0x7f1c0c00c3a0 comp rx=0 tx=0).stop 2024-03-19T01:11:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.333+0000 7f1c1dae7700 1 -- 172.21.15.116:0/14176190 shutdown_connections 2024-03-19T01:11:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.333+0000 7f1c1dae7700 1 --2- 172.21.15.116:0/14176190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c18107250 0x7f1c18103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.333+0000 7f1c1dae7700 1 --2- 172.21.15.116:0/14176190 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1c000a29a0 0x7f1c000a4e60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:56.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.333+0000 7f1c1dae7700 1 --2- 172.21.15.116:0/14176190 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1c18101210 0x7f1c18102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.333+0000 7f1c1dae7700 1 --2- 172.21.15.116:0/14176190 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c18101bb0 0x7f1c18103400 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.333+0000 7f1c1dae7700 1 -- 172.21.15.116:0/14176190 >> 172.21.15.116:0/14176190 conn(0x7f1c18078140 msgr2=0x7f1c18100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.333+0000 7f1c1dae7700 1 -- 172.21.15.116:0/14176190 shutdown_connections 2024-03-19T01:11:56.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:56.333+0000 7f1c1dae7700 1 -- 172.21.15.116:0/14176190 wait complete. 2024-03-19T01:11:56.398 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:56 smithi162 ceph-mon[67178]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:11:56.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:56 smithi116 ceph-mon[62701]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:11:56.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:56 smithi156 ceph-mon[66799]: pgmap v331: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:11:57.002 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-19T01:11:57.003 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:11:57.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.133+0000 7f247d3ff640 1 -- 172.21.15.116:0/278852700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24781016a0 msgr2=0x7f2478101ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.133+0000 7f247d3ff640 1 --2- 172.21.15.116:0/278852700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24781016a0 0x7f2478101ac0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2468009900 tx=0x7f246802bc30 comp rx=0 tx=0).stop 2024-03-19T01:11:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.133+0000 7f247d3ff640 1 -- 172.21.15.116:0/278852700 shutdown_connections 2024-03-19T01:11:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.133+0000 7f247d3ff640 1 --2- 172.21.15.116:0/278852700 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478103250 0x7f2478109ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.133+0000 7f247d3ff640 1 --2- 172.21.15.116:0/278852700 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2478102890 0x7f2478102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.133+0000 7f247d3ff640 1 --2- 172.21.15.116:0/278852700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24781016a0 0x7f2478101ac0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.133+0000 7f247d3ff640 1 -- 172.21.15.116:0/278852700 >> 172.21.15.116:0/278852700 conn(0x7f24780fcc30 msgr2=0x7f24780ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.133+0000 7f247d3ff640 1 -- 172.21.15.116:0/278852700 shutdown_connections 2024-03-19T01:11:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.133+0000 7f247d3ff640 1 -- 172.21.15.116:0/278852700 wait complete. 2024-03-19T01:11:57.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.134+0000 7f247d3ff640 1 Processor -- start 2024-03-19T01:11:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.134+0000 7f247d3ff640 1 -- start start 2024-03-19T01:11:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.134+0000 7f247d3ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24781016a0 0x7f24781979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f247d3ff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2478102890 0x7f2478197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f247d3ff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478103250 0x7f247819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f247d3ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f247810d2f0 con 0x7f24781016a0 2024-03-19T01:11:57.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f247d3ff640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f247810d170 con 0x7f2478103250 2024-03-19T01:11:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f247d3ff640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f247810d470 con 0x7f2478102890 2024-03-19T01:11:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f2476ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24781016a0 0x7f24781979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f2476ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24781016a0 0x7f24781979c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59578/0 (socket says 172.21.15.116:59578) 2024-03-19T01:11:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f2476ffd640 1 -- 172.21.15.116:0/4212587636 learned_addr learned my addr 172.21.15.116:0/4212587636 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f246e7fc640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2478102890 0x7f2478197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f24777fe640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478103250 0x7f247819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f2476ffd640 1 -- 172.21.15.116:0/4212587636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2478102890 msgr2=0x7f2478197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f2476ffd640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2478102890 0x7f2478197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f2476ffd640 1 -- 172.21.15.116:0/4212587636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478103250 msgr2=0x7f247819ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f2476ffd640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478103250 0x7f247819ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f2476ffd640 1 -- 172.21.15.116:0/4212587636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247819f5e0 con 0x7f24781016a0 2024-03-19T01:11:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f24777fe640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478103250 0x7f247819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:57.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.135+0000 7f2476ffd640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24781016a0 0x7f24781979c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2468000c00 tx=0x7f2468005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.136+0000 7f2474ff9640 1 -- 172.21.15.116:0/4212587636 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2468041070 con 0x7f24781016a0 2024-03-19T01:11:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.136+0000 7f2474ff9640 1 -- 172.21.15.116:0/4212587636 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24680044d0 con 0x7f24781016a0 2024-03-19T01:11:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.136+0000 7f247d3ff640 1 -- 172.21.15.116:0/4212587636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f247819f870 con 0x7f24781016a0 2024-03-19T01:11:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.136+0000 7f2474ff9640 1 -- 172.21.15.116:0/4212587636 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2468034440 con 0x7f24781016a0 2024-03-19T01:11:57.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.136+0000 7f247d3ff640 1 -- 172.21.15.116:0/4212587636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f247819fdb0 con 0x7f24781016a0 2024-03-19T01:11:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.137+0000 7f247d3ff640 1 -- 172.21.15.116:0/4212587636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2478108a60 con 0x7f24781016a0 2024-03-19T01:11:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.138+0000 7f2474ff9640 1 -- 172.21.15.116:0/4212587636 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f2468044070 con 0x7f24781016a0 2024-03-19T01:11:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.138+0000 7f2474ff9640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f24640a2910 0x7f24640a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.138+0000 7f246e7fc640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f24640a2910 0x7f24640a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.138+0000 7f2474ff9640 1 -- 172.21.15.116:0/4212587636 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f24680f0570 con 0x7f24781016a0 2024-03-19T01:11:57.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.139+0000 7f246e7fc640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f24640a2910 0x7f24640a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2478198ed0 tx=0x7f246000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:57.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.143+0000 7f2474ff9640 1 -- 172.21.15.116:0/4212587636 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f24680bcf50 con 0x7f24781016a0 2024-03-19T01:11:57.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.384+0000 7f247d3ff640 1 -- 172.21.15.116:0/4212587636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f247802d3a0 con 0x7f24781016a0 2024-03-19T01:11:57.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.385+0000 7f2474ff9640 1 -- 172.21.15.116:0/4212587636 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22804 (secure 0 0 0) 0x7f24680c0ff0 con 0x7f24781016a0 2024-03-19T01:11:57.386 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:11:57.387 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:11:28.144841+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:11:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.387+0000 7f247d3ff640 1 -- 172.21.15.116:0/4212587636 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f24640a2910 msgr2=0x7f24640a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.387+0000 7f247d3ff640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f24640a2910 0x7f24640a4dd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2478198ed0 tx=0x7f246000b410 comp rx=0 tx=0).stop 2024-03-19T01:11:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.387+0000 7f247d3ff640 1 -- 172.21.15.116:0/4212587636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24781016a0 msgr2=0x7f24781979c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.387+0000 7f247d3ff640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24781016a0 0x7f24781979c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2468000c00 tx=0x7f2468005d50 comp rx=0 tx=0).stop 2024-03-19T01:11:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.387+0000 7f247d3ff640 1 -- 172.21.15.116:0/4212587636 shutdown_connections 2024-03-19T01:11:57.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.387+0000 7f247d3ff640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f24640a2910 0x7f24640a4dd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.387+0000 7f247d3ff640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2478103250 0x7f247819ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.387+0000 7f247d3ff640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2478102890 0x7f2478197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.387+0000 7f247d3ff640 1 --2- 172.21.15.116:0/4212587636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24781016a0 0x7f24781979c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.387+0000 7f247d3ff640 1 -- 172.21.15.116:0/4212587636 >> 172.21.15.116:0/4212587636 conn(0x7f24780fcc30 msgr2=0x7f2478105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.387+0000 7f247d3ff640 1 -- 172.21.15.116:0/4212587636 shutdown_connections 2024-03-19T01:11:57.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.387+0000 7f247d3ff640 1 -- 172.21.15.116:0/4212587636 wait complete. 2024-03-19T01:11:57.398 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:57 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/14176190' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T01:11:57.400 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:57 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/14176190' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T01:11:57.401 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:05:53.301933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:10:31.028304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:10:31.704469+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:11:57.402 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:11:57.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:57 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/14176190' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-19T01:11:57.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.541+0000 7f013da80640 1 -- 172.21.15.116:0/2419699197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0138104bd0 msgr2=0x7f0138106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:57.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.541+0000 7f013da80640 1 --2- 172.21.15.116:0/2419699197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0138104bd0 0x7f0138106fc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0128009900 tx=0x7f012802bc30 comp rx=0 tx=0).stop 2024-03-19T01:11:57.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.541+0000 7f013da80640 1 -- 172.21.15.116:0/2419699197 shutdown_connections 2024-03-19T01:11:57.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.541+0000 7f013da80640 1 --2- 172.21.15.116:0/2419699197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0138107500 0x7f0138109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.541+0000 7f013da80640 1 --2- 172.21.15.116:0/2419699197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0138104bd0 0x7f0138106fc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.541+0000 7f013da80640 1 --2- 172.21.15.116:0/2419699197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01381022a0 0x7f0138104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.541+0000 7f013da80640 1 -- 172.21.15.116:0/2419699197 >> 172.21.15.116:0/2419699197 conn(0x7f01380fbcb0 msgr2=0x7f01380fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:57.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.541+0000 7f013da80640 1 -- 172.21.15.116:0/2419699197 shutdown_connections 2024-03-19T01:11:57.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.541+0000 7f013da80640 1 -- 172.21.15.116:0/2419699197 wait complete. 2024-03-19T01:11:57.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.541+0000 7f013da80640 1 Processor -- start 2024-03-19T01:11:57.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.541+0000 7f013da80640 1 -- start start 2024-03-19T01:11:57.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f013da80640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01381022a0 0x7f0138197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:57.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f013da80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0138104bd0 0x7f01381980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:57.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f013da80640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0138107500 0x7f013819f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:57.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f013da80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f013810d4a0 con 0x7f0138104bd0 2024-03-19T01:11:57.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f013da80640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f013810d320 con 0x7f01381022a0 2024-03-19T01:11:57.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f013da80640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f013810d620 con 0x7f0138107500 2024-03-19T01:11:57.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f0136ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0138104bd0 0x7f01381980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:57.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f0136ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0138104bd0 0x7f01381980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59596/0 (socket says 172.21.15.116:59596) 2024-03-19T01:11:57.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f0136ffd640 1 -- 172.21.15.116:0/3979367802 learned_addr learned my addr 172.21.15.116:0/3979367802 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:57.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f01377fe640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01381022a0 0x7f0138197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:57.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f0137fff640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0138107500 0x7f013819f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:57.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f0136ffd640 1 -- 172.21.15.116:0/3979367802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0138107500 msgr2=0x7f013819f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:57.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f0136ffd640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0138107500 0x7f013819f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f0136ffd640 1 -- 172.21.15.116:0/3979367802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01381022a0 msgr2=0x7f0138197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:57.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f0136ffd640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01381022a0 0x7f0138197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f0136ffd640 1 -- 172.21.15.116:0/3979367802 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013819f7c0 con 0x7f0138104bd0 2024-03-19T01:11:57.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f0136ffd640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0138104bd0 0x7f01381980e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0128000c00 tx=0x7f012802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:57.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f0134ff9640 1 -- 172.21.15.116:0/3979367802 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128041070 con 0x7f0138104bd0 2024-03-19T01:11:57.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f0134ff9640 1 -- 172.21.15.116:0/3979367802 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0128004450 con 0x7f0138104bd0 2024-03-19T01:11:57.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f013da80640 1 -- 172.21.15.116:0/3979367802 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013819fa50 con 0x7f0138104bd0 2024-03-19T01:11:57.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f0134ff9640 1 -- 172.21.15.116:0/3979367802 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128034440 con 0x7f0138104bd0 2024-03-19T01:11:57.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.542+0000 7f013da80640 1 -- 172.21.15.116:0/3979367802 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f013819ff40 con 0x7f0138104bd0 2024-03-19T01:11:57.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.543+0000 7f013da80640 1 -- 172.21.15.116:0/3979367802 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01380fd7a0 con 0x7f0138104bd0 2024-03-19T01:11:57.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.544+0000 7f0134ff9640 1 -- 172.21.15.116:0/3979367802 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f0128044070 con 0x7f0138104bd0 2024-03-19T01:11:57.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.545+0000 7f0134ff9640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f01200a2890 0x7f01200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:57.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.545+0000 7f01377fe640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f01200a2890 0x7f01200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:57.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.545+0000 7f0134ff9640 1 -- 172.21.15.116:0/3979367802 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f01280f07d0 con 0x7f0138104bd0 2024-03-19T01:11:57.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.545+0000 7f01377fe640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f01200a2890 0x7f01200a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f012c009bd0 tx=0x7f012c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:57.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.549+0000 7f0134ff9640 1 -- 172.21.15.116:0/3979367802 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f01280bd240 con 0x7f0138104bd0 2024-03-19T01:11:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.742+0000 7f013da80640 1 -- 172.21.15.116:0/3979367802 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f013802d1f0 con 0x7f0138104bd0 2024-03-19T01:11:57.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.742+0000 7f0134ff9640 1 -- 172.21.15.116:0/3979367802 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f01280c12e0 con 0x7f0138104bd0 2024-03-19T01:11:57.743 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T01:11:57.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.744+0000 7f013da80640 1 -- 172.21.15.116:0/3979367802 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f01200a2890 msgr2=0x7f01200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:57.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.744+0000 7f013da80640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f01200a2890 0x7f01200a4d50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f012c009bd0 tx=0x7f012c005f90 comp rx=0 tx=0).stop 2024-03-19T01:11:57.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.744+0000 7f013da80640 1 -- 172.21.15.116:0/3979367802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0138104bd0 msgr2=0x7f01381980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:57.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.744+0000 7f013da80640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0138104bd0 0x7f01381980e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0128000c00 tx=0x7f012802c640 comp rx=0 tx=0).stop 2024-03-19T01:11:57.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.744+0000 7f013da80640 1 -- 172.21.15.116:0/3979367802 shutdown_connections 2024-03-19T01:11:57.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.744+0000 7f013da80640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0138107500 0x7f013819f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.744+0000 7f013da80640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0138104bd0 0x7f01381980e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.744+0000 7f013da80640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f01200a2890 0x7f01200a4d50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.744+0000 7f013da80640 1 --2- 172.21.15.116:0/3979367802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f01381022a0 0x7f0138197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.744+0000 7f013da80640 1 -- 172.21.15.116:0/3979367802 >> 172.21.15.116:0/3979367802 conn(0x7f01380fbcb0 msgr2=0x7f0138102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:57.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.745+0000 7f013da80640 1 -- 172.21.15.116:0/3979367802 shutdown_connections 2024-03-19T01:11:57.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.745+0000 7f013da80640 1 -- 172.21.15.116:0/3979367802 wait complete. 2024-03-19T01:11:57.756 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:11:57.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.865+0000 7f820d8f8640 1 -- 172.21.15.116:0/92244294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208101a30 msgr2=0x7f8208108350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:57.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.865+0000 7f820d8f8640 1 --2- 172.21.15.116:0/92244294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208101a30 0x7f8208108350 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8200009ca0 tx=0x7f820002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.865+0000 7f820d8f8640 1 -- 172.21.15.116:0/92244294 shutdown_connections 2024-03-19T01:11:57.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.865+0000 7f820d8f8640 1 --2- 172.21.15.116:0/92244294 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208101a30 0x7f8208108350 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.865+0000 7f820d8f8640 1 --2- 172.21.15.116:0/92244294 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8208101070 0x7f82081014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.865+0000 7f820d8f8640 1 --2- 172.21.15.116:0/92244294 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82080ffe90 0x7f82081002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.865+0000 7f820d8f8640 1 -- 172.21.15.116:0/92244294 >> 172.21.15.116:0/92244294 conn(0x7f82080fb450 msgr2=0x7f82080fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.866+0000 7f820d8f8640 1 -- 172.21.15.116:0/92244294 shutdown_connections 2024-03-19T01:11:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.866+0000 7f820d8f8640 1 -- 172.21.15.116:0/92244294 wait complete. 2024-03-19T01:11:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.866+0000 7f820d8f8640 1 Processor -- start 2024-03-19T01:11:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.866+0000 7f820d8f8640 1 -- start start 2024-03-19T01:11:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.866+0000 7f820d8f8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82080ffe90 0x7f8208109580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.866+0000 7f820d8f8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8208101070 0x7f820819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.866+0000 7f820d8f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208101a30 0x7f82081a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.866+0000 7f820d8f8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f820810c400 con 0x7f8208101a30 2024-03-19T01:11:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.866+0000 7f820d8f8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f820810c280 con 0x7f8208101070 2024-03-19T01:11:57.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.866+0000 7f820d8f8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f820810c580 con 0x7f82080ffe90 2024-03-19T01:11:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.866+0000 7f82077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208101a30 0x7f82081a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f82077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208101a30 0x7f82081a11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59614/0 (socket says 172.21.15.116:59614) 2024-03-19T01:11:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f82077fe640 1 -- 172.21.15.116:0/373625963 learned_addr learned my addr 172.21.15.116:0/373625963 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f82067fc640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8208101070 0x7f820819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f8206ffd640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82080ffe90 0x7f8208109580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f82077fe640 1 -- 172.21.15.116:0/373625963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82080ffe90 msgr2=0x7f8208109580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f82077fe640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82080ffe90 0x7f8208109580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f82077fe640 1 -- 172.21.15.116:0/373625963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8208101070 msgr2=0x7f820819ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f82077fe640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8208101070 0x7f820819ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f82077fe640 1 -- 172.21.15.116:0/373625963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82081a19d0 con 0x7f8208101a30 2024-03-19T01:11:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f82077fe640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208101a30 0x7f82081a11e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8200037040 tx=0x7f8200032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f820c8f6640 1 -- 172.21.15.116:0/373625963 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8200041020 con 0x7f8208101a30 2024-03-19T01:11:57.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f820d8f8640 1 -- 172.21.15.116:0/373625963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82081a1c00 con 0x7f8208101a30 2024-03-19T01:11:57.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f820c8f6640 1 -- 172.21.15.116:0/373625963 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8200034cd0 con 0x7f8208101a30 2024-03-19T01:11:57.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f820c8f6640 1 -- 172.21.15.116:0/373625963 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8200039040 con 0x7f8208101a30 2024-03-19T01:11:57.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.867+0000 7f820d8f8640 1 -- 172.21.15.116:0/373625963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82081a20f0 con 0x7f8208101a30 2024-03-19T01:11:57.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.868+0000 7f820d8f8640 1 -- 172.21.15.116:0/373625963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8208107230 con 0x7f8208101a30 2024-03-19T01:11:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.869+0000 7f820c8f6640 1 -- 172.21.15.116:0/373625963 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 17) ==== 137342+0+0 (secure 0 0 0) 0x7f8200044020 con 0x7f8208101a30 2024-03-19T01:11:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.870+0000 7f820c8f6640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f81f00a28c0 0x7f81f00a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.870+0000 7f8206ffd640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f81f00a28c0 0x7f81f00a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.870+0000 7f820c8f6640 1 -- 172.21.15.116:0/373625963 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f82000f08e0 con 0x7f8208101a30 2024-03-19T01:11:57.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.870+0000 7f8206ffd640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f81f00a28c0 0x7f81f00a4d80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8208100be0 tx=0x7f81f8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:57.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:57.874+0000 7f820c8f6640 1 -- 172.21.15.116:0/373625963 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f820004b020 con 0x7f8208101a30 2024-03-19T01:11:58.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.103+0000 7f820d8f8640 1 -- 172.21.15.116:0/373625963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f820802d3a0 con 0x7f8208101a30 2024-03-19T01:11:58.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.104+0000 7f820c8f6640 1 -- 172.21.15.116:0/373625963 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f82000bd9a0 con 0x7f8208101a30 2024-03-19T01:11:58.105 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T01:11:58.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.106+0000 7f820d8f8640 1 -- 172.21.15.116:0/373625963 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f81f00a28c0 msgr2=0x7f81f00a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.106+0000 7f820d8f8640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f81f00a28c0 0x7f81f00a4d80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8208100be0 tx=0x7f81f8005f90 comp rx=0 tx=0).stop 2024-03-19T01:11:58.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.106+0000 7f820d8f8640 1 -- 172.21.15.116:0/373625963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208101a30 msgr2=0x7f82081a11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.106+0000 7f820d8f8640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208101a30 0x7f82081a11e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8200037040 tx=0x7f8200032f40 comp rx=0 tx=0).stop 2024-03-19T01:11:58.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.106+0000 7f820d8f8640 1 -- 172.21.15.116:0/373625963 shutdown_connections 2024-03-19T01:11:58.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.106+0000 7f820d8f8640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8208101a30 0x7f82081a11e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.106+0000 7f820d8f8640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f81f00a28c0 0x7f81f00a4d80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.106+0000 7f820d8f8640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8208101070 0x7f820819ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.106+0000 7f820d8f8640 1 --2- 172.21.15.116:0/373625963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f82080ffe90 0x7f8208109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.106+0000 7f820d8f8640 1 -- 172.21.15.116:0/373625963 >> 172.21.15.116:0/373625963 conn(0x7f82080fb450 msgr2=0x7f8208104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:58.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.106+0000 7f820d8f8640 1 -- 172.21.15.116:0/373625963 shutdown_connections 2024-03-19T01:11:58.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.106+0000 7f820d8f8640 1 -- 172.21.15.116:0/373625963 wait complete. 2024-03-19T01:11:58.121 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:11:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.231+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/1031740488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac0104c90 msgr2=0x7f1ac0107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.231+0000 7f1ac4e8a640 1 --2- 172.21.15.116:0/1031740488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac0104c90 0x7f1ac0107080 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0009900 tx=0x7f1ab002bc30 comp rx=0 tx=0).stop 2024-03-19T01:11:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.231+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/1031740488 shutdown_connections 2024-03-19T01:11:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.231+0000 7f1ac4e8a640 1 --2- 172.21.15.116:0/1031740488 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ac01075c0 0x7f1ac01099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.231+0000 7f1ac4e8a640 1 --2- 172.21.15.116:0/1031740488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac0104c90 0x7f1ac0107080 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.231+0000 7f1ac4e8a640 1 --2- 172.21.15.116:0/1031740488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0102360 0x7f1ac0104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.231+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/1031740488 >> 172.21.15.116:0/1031740488 conn(0x7f1ac00fbcb0 msgr2=0x7f1ac00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.231+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/1031740488 shutdown_connections 2024-03-19T01:11:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.231+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/1031740488 wait complete. 2024-03-19T01:11:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1ac4e8a640 1 Processor -- start 2024-03-19T01:11:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1ac4e8a640 1 -- start start 2024-03-19T01:11:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1ac4e8a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0102360 0x7f1ac0197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1ac4e8a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ac0104c90 0x7f1ac0198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1ac4e8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac01075c0 0x7f1ac019f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1ac4e8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac010d4b0 con 0x7f1ac01075c0 2024-03-19T01:11:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1ac4e8a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac010d330 con 0x7f1ac0102360 2024-03-19T01:11:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1ac4e8a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac010d630 con 0x7f1ac0104c90 2024-03-19T01:11:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1abed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac01075c0 0x7f1ac019f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1abed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac01075c0 0x7f1ac019f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59630/0 (socket says 172.21.15.116:59630) 2024-03-19T01:11:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1abe59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0102360 0x7f1ac0197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1abed9d640 1 -- 172.21.15.116:0/690286947 learned_addr learned my addr 172.21.15.116:0/690286947 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1abdd9b640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ac0104c90 0x7f1ac0198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1abed9d640 1 -- 172.21.15.116:0/690286947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ac0104c90 msgr2=0x7f1ac0198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1abed9d640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ac0104c90 0x7f1ac0198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1abed9d640 1 -- 172.21.15.116:0/690286947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0102360 msgr2=0x7f1ac0197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1abed9d640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0102360 0x7f1ac0197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1abed9d640 1 -- 172.21.15.116:0/690286947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac019f7e0 con 0x7f1ac01075c0 2024-03-19T01:11:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1abe59c640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0102360 0x7f1ac0197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1abed9d640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac01075c0 0x7f1ac019f170 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1ab800eb10 tx=0x7f1ab800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.232+0000 7f1ab77fe640 1 -- 172.21.15.116:0/690286947 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8004b20 con 0x7f1ac01075c0 2024-03-19T01:11:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.233+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/690286947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ac019fad0 con 0x7f1ac01075c0 2024-03-19T01:11:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.233+0000 7f1ab77fe640 1 -- 172.21.15.116:0/690286947 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ab8004cc0 con 0x7f1ac01075c0 2024-03-19T01:11:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.233+0000 7f1ab77fe640 1 -- 172.21.15.116:0/690286947 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8010760 con 0x7f1ac01075c0 2024-03-19T01:11:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.233+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/690286947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ac019fff0 con 0x7f1ac01075c0 2024-03-19T01:11:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.234+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/690286947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ac00fd7a0 con 0x7f1ac01075c0 2024-03-19T01:11:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.235+0000 7f1ab77fe640 1 -- 172.21.15.116:0/690286947 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f1ab8005020 con 0x7f1ac01075c0 2024-03-19T01:11:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.235+0000 7f1ab77fe640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1aac0a2800 0x7f1aac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.236+0000 7f1abe59c640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1aac0a2800 0x7f1aac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.236+0000 7f1ab77fe640 1 -- 172.21.15.116:0/690286947 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1ab80cf160 con 0x7f1ac01075c0 2024-03-19T01:11:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.236+0000 7f1abe59c640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1aac0a2800 0x7f1aac0a4cc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009fc0 tx=0x7f1aa8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:58.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.240+0000 7f1ab77fe640 1 -- 172.21.15.116:0/690286947 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1ab809c1e0 con 0x7f1ac01075c0 2024-03-19T01:11:58.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:58 smithi162 ceph-mon[67178]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:11:58.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:58 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/4212587636' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:11:58.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:58 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3979367802' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:11:58.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:58 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/373625963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:11:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:58 smithi116 ceph-mon[62701]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:11:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:58 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4212587636' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:11:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:58 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3979367802' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:11:58.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:58 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/373625963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:11:58.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.418+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/690286947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1ac002d240 con 0x7f1ac01075c0 2024-03-19T01:11:58.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.418+0000 7f1ab77fe640 1 -- 172.21.15.116:0/690286947 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f1ab809bae0 con 0x7f1ac01075c0 2024-03-19T01:11:58.419 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T01:11:58.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.420+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/690286947 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1aac0a2800 msgr2=0x7f1aac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.420+0000 7f1ac4e8a640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1aac0a2800 0x7f1aac0a4cc0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1aa8009fc0 tx=0x7f1aa8005f90 comp rx=0 tx=0).stop 2024-03-19T01:11:58.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.420+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/690286947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac01075c0 msgr2=0x7f1ac019f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.420+0000 7f1ac4e8a640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac01075c0 0x7f1ac019f170 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1ab800eb10 tx=0x7f1ab800c2d0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.420+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/690286947 shutdown_connections 2024-03-19T01:11:58.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.420+0000 7f1ac4e8a640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ac01075c0 0x7f1ac019f170 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.420+0000 7f1ac4e8a640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1ac0104c90 0x7f1ac0198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.420+0000 7f1ac4e8a640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1aac0a2800 0x7f1aac0a4cc0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.420+0000 7f1ac4e8a640 1 --2- 172.21.15.116:0/690286947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ac0102360 0x7f1ac0197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.420+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/690286947 >> 172.21.15.116:0/690286947 conn(0x7f1ac00fbcb0 msgr2=0x7f1ac0102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:58.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.420+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/690286947 shutdown_connections 2024-03-19T01:11:58.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.420+0000 7f1ac4e8a640 1 -- 172.21.15.116:0/690286947 wait complete. 2024-03-19T01:11:58.432 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:11:58.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:58 smithi156 ceph-mon[66799]: pgmap v332: 97 pgs: 97 active+clean; 643 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:11:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:58 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/4212587636' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:11:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:58 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3979367802' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:11:58.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:58 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/373625963' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:11:58.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.535+0000 7fb6b8309640 1 -- 172.21.15.116:0/865978638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6b0101930 msgr2=0x7fb6b0101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.535+0000 7fb6b8309640 1 --2- 172.21.15.116:0/865978638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6b0101930 0x7fb6b0101db0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0009900 tx=0x7fb6a002bc30 comp rx=0 tx=0).stop 2024-03-19T01:11:58.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.535+0000 7fb6b8309640 1 -- 172.21.15.116:0/865978638 shutdown_connections 2024-03-19T01:11:58.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.535+0000 7fb6b8309640 1 --2- 172.21.15.116:0/865978638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb6b01022f0 0x7fb6b0108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.535+0000 7fb6b8309640 1 --2- 172.21.15.116:0/865978638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6b0101930 0x7fb6b0101db0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.535+0000 7fb6b8309640 1 --2- 172.21.15.116:0/865978638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6b0100740 0x7fb6b0100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.535+0000 7fb6b8309640 1 -- 172.21.15.116:0/865978638 >> 172.21.15.116:0/865978638 conn(0x7fb6b00fbcb0 msgr2=0x7fb6b00fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:58.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.535+0000 7fb6b8309640 1 -- 172.21.15.116:0/865978638 shutdown_connections 2024-03-19T01:11:58.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.535+0000 7fb6b8309640 1 -- 172.21.15.116:0/865978638 wait complete. 2024-03-19T01:11:58.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b8309640 1 Processor -- start 2024-03-19T01:11:58.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b8309640 1 -- start start 2024-03-19T01:11:58.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b8309640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6b0100740 0x7fb6b0197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:58.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b8309640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb6b0101930 0x7fb6b0198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:58.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b8309640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6b01022f0 0x7fb6b019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:58.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b8309640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b010c3c0 con 0x7fb6b0100740 2024-03-19T01:11:58.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b8309640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b010c240 con 0x7fb6b0101930 2024-03-19T01:11:58.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b8309640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb6b010c540 con 0x7fb6b01022f0 2024-03-19T01:11:58.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b60a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6b0100740 0x7fb6b0197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:58.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b60a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6b0100740 0x7fb6b0197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59640/0 (socket says 172.21.15.116:59640) 2024-03-19T01:11:58.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b60a5640 1 -- 172.21.15.116:0/52400166 learned_addr learned my addr 172.21.15.116:0/52400166 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:58.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b58a4640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb6b0101930 0x7fb6b0198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:58.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b68a6640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6b01022f0 0x7fb6b019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:58.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b60a5640 1 -- 172.21.15.116:0/52400166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6b01022f0 msgr2=0x7fb6b019f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b60a5640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6b01022f0 0x7fb6b019f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b60a5640 1 -- 172.21.15.116:0/52400166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb6b0101930 msgr2=0x7fb6b0198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b60a5640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb6b0101930 0x7fb6b0198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b60a5640 1 -- 172.21.15.116:0/52400166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6b019f7c0 con 0x7fb6b0100740 2024-03-19T01:11:58.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6b60a5640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6b0100740 0x7fb6b0197c10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac00e9c0 tx=0x7fb6ac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:58.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.536+0000 7fb6a77fe640 1 -- 172.21.15.116:0/52400166 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac00cd80 con 0x7fb6b0100740 2024-03-19T01:11:58.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.537+0000 7fb6a77fe640 1 -- 172.21.15.116:0/52400166 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6ac0035f0 con 0x7fb6b0100740 2024-03-19T01:11:58.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.537+0000 7fb6b8309640 1 -- 172.21.15.116:0/52400166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6b019fab0 con 0x7fb6b0100740 2024-03-19T01:11:58.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.537+0000 7fb6a77fe640 1 -- 172.21.15.116:0/52400166 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac010690 con 0x7fb6b0100740 2024-03-19T01:11:58.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.537+0000 7fb6b8309640 1 -- 172.21.15.116:0/52400166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6b019ff80 con 0x7fb6b0100740 2024-03-19T01:11:58.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.537+0000 7fb6b8309640 1 -- 172.21.15.116:0/52400166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6b01a0210 con 0x7fb6b0100740 2024-03-19T01:11:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.538+0000 7fb6a77fe640 1 -- 172.21.15.116:0/52400166 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fb6ac0037a0 con 0x7fb6b0100740 2024-03-19T01:11:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.539+0000 7fb6a77fe640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb69c0a2800 0x7fb69c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.539+0000 7fb6b58a4640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb69c0a2800 0x7fb69c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:58.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.539+0000 7fb6a77fe640 1 -- 172.21.15.116:0/52400166 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb6ac0cede0 con 0x7fb6b0100740 2024-03-19T01:11:58.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.540+0000 7fb6b58a4640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb69c0a2800 0x7fb69c0a4cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb6a002c7b0 tx=0x7fb6a0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:58.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.543+0000 7fb6a77fe640 1 -- 172.21.15.116:0/52400166 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb6ac09be60 con 0x7fb6b0100740 2024-03-19T01:11:58.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.761+0000 7fb6b8309640 1 -- 172.21.15.116:0/52400166 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6b01a09a0 con 0x7fb6b0100740 2024-03-19T01:11:58.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.761+0000 7fb6a77fe640 1 -- 172.21.15.116:0/52400166 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11938 (secure 0 0 0) 0x7fb6ac09b760 con 0x7fb6b0100740 2024-03-19T01:11:58.762 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:11:58.762 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T01:10:32.487607+0000","modified":"2024-03-19T01:10:38.982004+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24488,"mds_2":24473,"mds_3":24460,"mds_4":24445},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1242639081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1242639081},{"type":"v1","addr":"172.21.15.116:6839","nonce":1242639081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24394":{"gid":24394,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":3,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.162:6839/441486392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":441486392},{"type":"v1","addr":"172.21.15.162:6839","nonce":441486392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"a","rank":1,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6841/3343188504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3343188504},{"type":"v1","addr":"172.21.15.116:6841","nonce":3343188504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1601537427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1601537427},{"type":"v1","addr":"172.21.15.156:6839","nonce":1601537427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24473,24445,24500,24460,24488]},"id":1}]} 2024-03-19T01:11:58.763 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2024-03-19T01:11:58.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.763+0000 7fb6b8309640 1 -- 172.21.15.116:0/52400166 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb69c0a2800 msgr2=0x7fb69c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.763+0000 7fb6b8309640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb69c0a2800 0x7fb69c0a4cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb6a002c7b0 tx=0x7fb6a0005fd0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.763+0000 7fb6b8309640 1 -- 172.21.15.116:0/52400166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6b0100740 msgr2=0x7fb6b0197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.763+0000 7fb6b8309640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6b0100740 0x7fb6b0197c10 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac00e9c0 tx=0x7fb6ac00ee80 comp rx=0 tx=0).stop 2024-03-19T01:11:58.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.763+0000 7fb6b8309640 1 -- 172.21.15.116:0/52400166 shutdown_connections 2024-03-19T01:11:58.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.763+0000 7fb6b8309640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb6b01022f0 0x7fb6b019f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.763+0000 7fb6b8309640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb69c0a2800 0x7fb69c0a4cc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.763+0000 7fb6b8309640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb6b0101930 0x7fb6b0198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.763+0000 7fb6b8309640 1 --2- 172.21.15.116:0/52400166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb6b0100740 0x7fb6b0197c10 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.763+0000 7fb6b8309640 1 -- 172.21.15.116:0/52400166 >> 172.21.15.116:0/52400166 conn(0x7fb6b00fbcb0 msgr2=0x7fb6b00fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:58.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.763+0000 7fb6b8309640 1 -- 172.21.15.116:0/52400166 shutdown_connections 2024-03-19T01:11:58.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.763+0000 7fb6b8309640 1 -- 172.21.15.116:0/52400166 wait complete. 2024-03-19T01:11:58.776 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:11:58.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.877+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/2485311548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e481022f0 msgr2=0x7f6e48108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.877+0000 7f6e4f2bf640 1 --2- 172.21.15.116:0/2485311548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e481022f0 0x7f6e48108b80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f6e40009ca0 tx=0x7f6e4002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.877+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/2485311548 shutdown_connections 2024-03-19T01:11:58.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.877+0000 7f6e4f2bf640 1 --2- 172.21.15.116:0/2485311548 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e481022f0 0x7f6e48108b80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.877+0000 7f6e4f2bf640 1 --2- 172.21.15.116:0/2485311548 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e48101930 0x7f6e48101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.877+0000 7f6e4f2bf640 1 --2- 172.21.15.116:0/2485311548 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e48100740 0x7f6e48100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.877+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/2485311548 >> 172.21.15.116:0/2485311548 conn(0x7f6e480fbcb0 msgr2=0x7f6e480fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:58.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.877+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/2485311548 shutdown_connections 2024-03-19T01:11:58.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.877+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/2485311548 wait complete. 2024-03-19T01:11:58.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4f2bf640 1 Processor -- start 2024-03-19T01:11:58.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4f2bf640 1 -- start start 2024-03-19T01:11:58.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4f2bf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e48100740 0x7f6e48197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:58.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4f2bf640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e48101930 0x7f6e48198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:58.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4f2bf640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e481022f0 0x7f6e4819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:58.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4f2bf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4810d4e0 con 0x7f6e48100740 2024-03-19T01:11:58.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4f2bf640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4810d360 con 0x7f6e48101930 2024-03-19T01:11:58.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4f2bf640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4810d660 con 0x7f6e481022f0 2024-03-19T01:11:58.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4d05b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e48100740 0x7f6e48197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4d05b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e48100740 0x7f6e48197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59662/0 (socket says 172.21.15.116:59662) 2024-03-19T01:11:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4d05b640 1 -- 172.21.15.116:0/1390489136 learned_addr learned my addr 172.21.15.116:0/1390489136 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4d85c640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e481022f0 0x7f6e4819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4c85a640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e48101930 0x7f6e48198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4d05b640 1 -- 172.21.15.116:0/1390489136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e481022f0 msgr2=0x7f6e4819f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4d05b640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e481022f0 0x7f6e4819f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4d05b640 1 -- 172.21.15.116:0/1390489136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e48101930 msgr2=0x7f6e48198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4d05b640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e48101930 0x7f6e48198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4d05b640 1 -- 172.21.15.116:0/1390489136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4819f730 con 0x7f6e48100740 2024-03-19T01:11:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4d85c640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e481022f0 0x7f6e4819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.878+0000 7f6e4d05b640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e48100740 0x7f6e48197b10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400e9c0 tx=0x7f6e4400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.879+0000 7f6e3e7fc640 1 -- 172.21.15.116:0/1390489136 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4400cd80 con 0x7f6e48100740 2024-03-19T01:11:58.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.879+0000 7f6e3e7fc640 1 -- 172.21.15.116:0/1390489136 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e440035f0 con 0x7f6e48100740 2024-03-19T01:11:58.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.879+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/1390489136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e4819fa20 con 0x7f6e48100740 2024-03-19T01:11:58.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.879+0000 7f6e3e7fc640 1 -- 172.21.15.116:0/1390489136 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e44010690 con 0x7f6e48100740 2024-03-19T01:11:58.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.879+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/1390489136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e4819ff70 con 0x7f6e48100740 2024-03-19T01:11:58.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.880+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/1390489136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e48107a60 con 0x7f6e48100740 2024-03-19T01:11:58.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.881+0000 7f6e3e7fc640 1 -- 172.21.15.116:0/1390489136 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6e440037a0 con 0x7f6e48100740 2024-03-19T01:11:58.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.881+0000 7f6e3e7fc640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6e340a2880 0x7f6e340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:58.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.881+0000 7f6e4c85a640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6e340a2880 0x7f6e340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:58.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.882+0000 7f6e3e7fc640 1 -- 172.21.15.116:0/1390489136 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6e440ced50 con 0x7f6e48100740 2024-03-19T01:11:58.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.882+0000 7f6e4c85a640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6e340a2880 0x7f6e340a4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6e380095f0 tx=0x7f6e38005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:58.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:58.886+0000 7f6e3e7fc640 1 -- 172.21.15.116:0/1390489136 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6e440d2030 con 0x7f6e48100740 2024-03-19T01:11:59.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.069+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/1390489136 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e48062420 con 0x7f6e48100740 2024-03-19T01:11:59.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.071+0000 7f6e3e7fc640 1 -- 172.21.15.116:0/1390489136 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22804 (secure 0 0 0) 0x7f6e4409bdd0 con 0x7f6e48100740 2024-03-19T01:11:59.072 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:11:59.072 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:11:28.144841+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:11:59.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.073+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/1390489136 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6e340a2880 msgr2=0x7f6e340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.073+0000 7f6e4f2bf640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6e340a2880 0x7f6e340a4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6e380095f0 tx=0x7f6e38005f90 comp rx=0 tx=0).stop 2024-03-19T01:11:59.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.073+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/1390489136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e48100740 msgr2=0x7f6e48197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.073+0000 7f6e4f2bf640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e48100740 0x7f6e48197b10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400e9c0 tx=0x7f6e4400ee80 comp rx=0 tx=0).stop 2024-03-19T01:11:59.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.073+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/1390489136 shutdown_connections 2024-03-19T01:11:59.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.073+0000 7f6e4f2bf640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6e481022f0 0x7f6e4819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.073+0000 7f6e4f2bf640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6e340a2880 0x7f6e340a4d40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.073+0000 7f6e4f2bf640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6e48101930 0x7f6e48198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.073+0000 7f6e4f2bf640 1 --2- 172.21.15.116:0/1390489136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e48100740 0x7f6e48197b10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.073+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/1390489136 >> 172.21.15.116:0/1390489136 conn(0x7f6e480fbcb0 msgr2=0x7f6e48104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:59.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.073+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/1390489136 shutdown_connections 2024-03-19T01:11:59.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.073+0000 7f6e4f2bf640 1 -- 172.21.15.116:0/1390489136 wait complete. 2024-03-19T01:11:59.085 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:05:53.301933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:10:31.028304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:10:31.704469+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:11:59.085 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:11:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.197+0000 7fbab0c20640 1 -- 172.21.15.116:0/540489565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbaac1022f0 msgr2=0x7fbaac108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.197+0000 7fbab0c20640 1 --2- 172.21.15.116:0/540489565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbaac1022f0 0x7fbaac108b80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fba9c009900 tx=0x7fba9c02bc30 comp rx=0 tx=0).stop 2024-03-19T01:11:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.197+0000 7fbab0c20640 1 -- 172.21.15.116:0/540489565 shutdown_connections 2024-03-19T01:11:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.197+0000 7fbab0c20640 1 --2- 172.21.15.116:0/540489565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbaac1022f0 0x7fbaac108b80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.197+0000 7fbab0c20640 1 --2- 172.21.15.116:0/540489565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaac101930 0x7fbaac101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.197+0000 7fbab0c20640 1 --2- 172.21.15.116:0/540489565 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbaac100740 0x7fbaac100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.197+0000 7fbab0c20640 1 -- 172.21.15.116:0/540489565 >> 172.21.15.116:0/540489565 conn(0x7fbaac0fbcd0 msgr2=0x7fbaac0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.197+0000 7fbab0c20640 1 -- 172.21.15.116:0/540489565 shutdown_connections 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.197+0000 7fbab0c20640 1 -- 172.21.15.116:0/540489565 wait complete. 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbab0c20640 1 Processor -- start 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbab0c20640 1 -- start start 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbab0c20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbaac100740 0x7fbaac06a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbab0c20640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaac101930 0x7fbaac19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbab0c20640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbaac1022f0 0x7fbaac19ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbab0c20640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac10c3c0 con 0x7fbaac100740 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbab0c20640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac10c240 con 0x7fbaac101930 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbab0c20640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac10c540 con 0x7fbaac1022f0 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbaaa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbaac100740 0x7fbaac06a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbaaa59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbaac100740 0x7fbaac06a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59682/0 (socket says 172.21.15.116:59682) 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbaaa59c640 1 -- 172.21.15.116:0/2855338483 learned_addr learned my addr 172.21.15.116:0/2855338483 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:59.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbaa9d9b640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaac101930 0x7fbaac19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbaaad9d640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbaac1022f0 0x7fbaac19ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbaaa59c640 1 -- 172.21.15.116:0/2855338483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbaac1022f0 msgr2=0x7fbaac19ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbaaa59c640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbaac1022f0 0x7fbaac19ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbaaa59c640 1 -- 172.21.15.116:0/2855338483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaac101930 msgr2=0x7fbaac19c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbaaa59c640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaac101930 0x7fbaac19c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbaaa59c640 1 -- 172.21.15.116:0/2855338483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaac19f700 con 0x7fbaac100740 2024-03-19T01:11:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.198+0000 7fbaaa59c640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbaac100740 0x7fbaac06a9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbaa400e9c0 tx=0x7fbaa400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.199+0000 7fba9b7fe640 1 -- 172.21.15.116:0/2855338483 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa400cd80 con 0x7fbaac100740 2024-03-19T01:11:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.199+0000 7fbab0c20640 1 -- 172.21.15.116:0/2855338483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaac19f9f0 con 0x7fbaac100740 2024-03-19T01:11:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.199+0000 7fba9b7fe640 1 -- 172.21.15.116:0/2855338483 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbaa40035f0 con 0x7fbaac100740 2024-03-19T01:11:59.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.199+0000 7fba9b7fe640 1 -- 172.21.15.116:0/2855338483 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4010690 con 0x7fbaac100740 2024-03-19T01:11:59.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.199+0000 7fbab0c20640 1 -- 172.21.15.116:0/2855338483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaac19fef0 con 0x7fbaac100740 2024-03-19T01:11:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.200+0000 7fbab0c20640 1 -- 172.21.15.116:0/2855338483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaac107ae0 con 0x7fbaac100740 2024-03-19T01:11:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.201+0000 7fba9b7fe640 1 -- 172.21.15.116:0/2855338483 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fbaa40037a0 con 0x7fbaac100740 2024-03-19T01:11:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.202+0000 7fba9b7fe640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fba940a2800 0x7fba940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.202+0000 7fbaa9d9b640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fba940a2800 0x7fba940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.202+0000 7fba9b7fe640 1 -- 172.21.15.116:0/2855338483 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbaa40cede0 con 0x7fbaac100740 2024-03-19T01:11:59.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.202+0000 7fbaa9d9b640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fba940a2800 0x7fba940a4cc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbaac06a730 tx=0x7fbaa0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:59.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.206+0000 7fba9b7fe640 1 -- 172.21.15.116:0/2855338483 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbaa409be60 con 0x7fbaac100740 2024-03-19T01:11:59.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.386+0000 7fbab0c20640 1 -- 172.21.15.116:0/2855338483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbaac02d3d0 con 0x7fbaac100740 2024-03-19T01:11:59.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.386+0000 7fba9b7fe640 1 -- 172.21.15.116:0/2855338483 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fbaa409b760 con 0x7fbaac100740 2024-03-19T01:11:59.387 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T01:11:59.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.389+0000 7fbab0c20640 1 -- 172.21.15.116:0/2855338483 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fba940a2800 msgr2=0x7fba940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.389+0000 7fbab0c20640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fba940a2800 0x7fba940a4cc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbaac06a730 tx=0x7fbaa0009500 comp rx=0 tx=0).stop 2024-03-19T01:11:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.389+0000 7fbab0c20640 1 -- 172.21.15.116:0/2855338483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbaac100740 msgr2=0x7fbaac06a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.389+0000 7fbab0c20640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbaac100740 0x7fbaac06a9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbaa400e9c0 tx=0x7fbaa400ee80 comp rx=0 tx=0).stop 2024-03-19T01:11:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.389+0000 7fbab0c20640 1 -- 172.21.15.116:0/2855338483 shutdown_connections 2024-03-19T01:11:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.389+0000 7fbab0c20640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbaac1022f0 0x7fbaac19ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.389+0000 7fbab0c20640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fba940a2800 0x7fba940a4cc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.389+0000 7fbab0c20640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbaac101930 0x7fbaac19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.389+0000 7fbab0c20640 1 --2- 172.21.15.116:0/2855338483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbaac100740 0x7fbaac06a9c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.389+0000 7fbab0c20640 1 -- 172.21.15.116:0/2855338483 >> 172.21.15.116:0/2855338483 conn(0x7fbaac0fbcd0 msgr2=0x7fbaac104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.389+0000 7fbab0c20640 1 -- 172.21.15.116:0/2855338483 shutdown_connections 2024-03-19T01:11:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.389+0000 7fbab0c20640 1 -- 172.21.15.116:0/2855338483 wait complete. 2024-03-19T01:11:59.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:59 smithi116 ceph-mon[62701]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-19T01:11:59.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:59 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/690286947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:11:59.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:59 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/52400166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:11:59.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:11:59 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1390489136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:11:59.400 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:11:59.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:59 smithi156 ceph-mon[66799]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-19T01:11:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:59 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/690286947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:11:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:59 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/52400166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:11:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:11:59 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1390489136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:11:59.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.509+0000 7fe259aea640 1 -- 172.21.15.116:0/1956318323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe254100740 msgr2=0x7fe254100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.509+0000 7fe259aea640 1 --2- 172.21.15.116:0/1956318323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe254100740 0x7fe254100b60 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe248009900 tx=0x7fe24802bc30 comp rx=0 tx=0).stop 2024-03-19T01:11:59.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.509+0000 7fe259aea640 1 -- 172.21.15.116:0/1956318323 shutdown_connections 2024-03-19T01:11:59.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.509+0000 7fe259aea640 1 --2- 172.21.15.116:0/1956318323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2541022f0 0x7fe254108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.509+0000 7fe259aea640 1 --2- 172.21.15.116:0/1956318323 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe254101930 0x7fe254101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.509+0000 7fe259aea640 1 --2- 172.21.15.116:0/1956318323 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe254100740 0x7fe254100b60 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.509+0000 7fe259aea640 1 -- 172.21.15.116:0/1956318323 >> 172.21.15.116:0/1956318323 conn(0x7fe2540fbcd0 msgr2=0x7fe2540fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:59.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.509+0000 7fe259aea640 1 -- 172.21.15.116:0/1956318323 shutdown_connections 2024-03-19T01:11:59.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.509+0000 7fe259aea640 1 -- 172.21.15.116:0/1956318323 wait complete. 2024-03-19T01:11:59.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe259aea640 1 Processor -- start 2024-03-19T01:11:59.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe259aea640 1 -- start start 2024-03-19T01:11:59.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe259aea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe254100740 0x7fe254197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:59.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe259aea640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe254101930 0x7fe254198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:59.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe259aea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2541022f0 0x7fe25419f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:59.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe259aea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe25410c3b0 con 0x7fe2541022f0 2024-03-19T01:11:59.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe259aea640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe25410c230 con 0x7fe254101930 2024-03-19T01:11:59.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe259aea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe25410c530 con 0x7fe254100740 2024-03-19T01:11:59.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe253fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2541022f0 0x7fe25419f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:59.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe2537fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe254100740 0x7fe254197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:59.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe2537fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe254100740 0x7fe254197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59510/0 (socket says 172.21.15.116:59510) 2024-03-19T01:11:59.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe252ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe254101930 0x7fe254198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:59.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe2537fe640 1 -- 172.21.15.116:0/2959196560 learned_addr learned my addr 172.21.15.116:0/2959196560 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:59.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.510+0000 7fe253fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2541022f0 0x7fe25419f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59696/0 (socket says 172.21.15.116:59696) 2024-03-19T01:11:59.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.511+0000 7fe2537fe640 1 -- 172.21.15.116:0/2959196560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe254101930 msgr2=0x7fe254198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.511+0000 7fe2537fe640 1 --2- 172.21.15.116:0/2959196560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe254101930 0x7fe254198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.511+0000 7fe2537fe640 1 -- 172.21.15.116:0/2959196560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2541022f0 msgr2=0x7fe25419f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.511+0000 7fe2537fe640 1 --2- 172.21.15.116:0/2959196560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2541022f0 0x7fe25419f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.511+0000 7fe2537fe640 1 -- 172.21.15.116:0/2959196560 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe25419f7f0 con 0x7fe254100740 2024-03-19T01:11:59.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.511+0000 7fe252ffd640 1 --2- 172.21.15.116:0/2959196560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe254101930 0x7fe254198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:59.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.511+0000 7fe2537fe640 1 --2- 172.21.15.116:0/2959196560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe254100740 0x7fe254197bd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe248000c00 tx=0x7fe24802c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:59.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.511+0000 7fe250ff9640 1 -- 172.21.15.116:0/2959196560 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe248041070 con 0x7fe254100740 2024-03-19T01:11:59.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.511+0000 7fe250ff9640 1 -- 172.21.15.116:0/2959196560 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe248004450 con 0x7fe254100740 2024-03-19T01:11:59.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.511+0000 7fe259aea640 1 -- 172.21.15.116:0/2959196560 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe25419fa80 con 0x7fe254100740 2024-03-19T01:11:59.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.511+0000 7fe250ff9640 1 -- 172.21.15.116:0/2959196560 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe248034440 con 0x7fe254100740 2024-03-19T01:11:59.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.511+0000 7fe259aea640 1 -- 172.21.15.116:0/2959196560 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe25419ff70 con 0x7fe254100740 2024-03-19T01:11:59.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.512+0000 7fe259aea640 1 -- 172.21.15.116:0/2959196560 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2541a0140 con 0x7fe254100740 2024-03-19T01:11:59.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.513+0000 7fe250ff9640 1 -- 172.21.15.116:0/2959196560 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fe248044070 con 0x7fe254100740 2024-03-19T01:11:59.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.513+0000 7fe250ff9640 1 --2- 172.21.15.116:0/2959196560 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe23c0a2800 0x7fe23c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:59.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.513+0000 7fe252ffd640 1 --2- 172.21.15.116:0/2959196560 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe23c0a2800 0x7fe23c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:59.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.514+0000 7fe250ff9640 1 -- 172.21.15.116:0/2959196560 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe2480f0540 con 0x7fe254100740 2024-03-19T01:11:59.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.514+0000 7fe252ffd640 1 --2- 172.21.15.116:0/2959196560 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe23c0a2800 0x7fe23c0a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe2541990e0 tx=0x7fe244005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:59.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.518+0000 7fe250ff9640 1 -- 172.21.15.116:0/2959196560 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe2480bd000 con 0x7fe254100740 2024-03-19T01:11:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:59 smithi162 ceph-mon[67178]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-19T01:11:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:59 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/690286947' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:11:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:59 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/52400166' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:11:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:11:59 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1390489136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:11:59.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.695+0000 7fe259aea640 1 -- 172.21.15.116:0/2959196560 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe2541a04d0 con 0x7fe254100740 2024-03-19T01:11:59.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.696+0000 7fe250ff9640 1 -- 172.21.15.116:0/2959196560 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7fe2480c10a0 con 0x7fe254100740 2024-03-19T01:11:59.697 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T01:11:59.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.698+0000 7fe259aea640 1 -- 172.21.15.116:0/2959196560 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe23c0a2800 msgr2=0x7fe23c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.698+0000 7fe259aea640 1 --2- 172.21.15.116:0/2959196560 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe23c0a2800 0x7fe23c0a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe2541990e0 tx=0x7fe244005f90 comp rx=0 tx=0).stop 2024-03-19T01:11:59.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.698+0000 7fe259aea640 1 -- 172.21.15.116:0/2959196560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe254100740 msgr2=0x7fe254197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.698+0000 7fe259aea640 1 --2- 172.21.15.116:0/2959196560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe254100740 0x7fe254197bd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe248000c00 tx=0x7fe24802c640 comp rx=0 tx=0).stop 2024-03-19T01:11:59.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.698+0000 7fe259aea640 1 -- 172.21.15.116:0/2959196560 shutdown_connections 2024-03-19T01:11:59.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.698+0000 7fe259aea640 1 --2- 172.21.15.116:0/2959196560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2541022f0 0x7fe25419f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.698+0000 7fe259aea640 1 --2- 172.21.15.116:0/2959196560 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe23c0a2800 0x7fe23c0a4cc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.698+0000 7fe259aea640 1 --2- 172.21.15.116:0/2959196560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe254101930 0x7fe254198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.698+0000 7fe259aea640 1 --2- 172.21.15.116:0/2959196560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe254100740 0x7fe254197bd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.698+0000 7fe259aea640 1 -- 172.21.15.116:0/2959196560 >> 172.21.15.116:0/2959196560 conn(0x7fe2540fbcd0 msgr2=0x7fe2540fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:59.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.698+0000 7fe259aea640 1 -- 172.21.15.116:0/2959196560 shutdown_connections 2024-03-19T01:11:59.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.698+0000 7fe259aea640 1 -- 172.21.15.116:0/2959196560 wait complete. 2024-03-19T01:11:59.709 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:11:59.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.817+0000 7f6b03f5b640 1 -- 172.21.15.116:0/2360365429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6afc100a50 msgr2=0x7f6afc100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.817+0000 7f6b03f5b640 1 --2- 172.21.15.116:0/2360365429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6afc100a50 0x7f6afc100eb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6af0009900 tx=0x7f6af002bc30 comp rx=0 tx=0).stop 2024-03-19T01:11:59.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.817+0000 7f6b03f5b640 1 -- 172.21.15.116:0/2360365429 shutdown_connections 2024-03-19T01:11:59.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.817+0000 7f6b03f5b640 1 --2- 172.21.15.116:0/2360365429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6afc1013f0 0x7f6afc101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.817+0000 7f6b03f5b640 1 --2- 172.21.15.116:0/2360365429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6afc100a50 0x7f6afc100eb0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.817+0000 7f6b03f5b640 1 --2- 172.21.15.116:0/2360365429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6afc1000f0 0x7f6afc100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.817+0000 7f6b03f5b640 1 -- 172.21.15.116:0/2360365429 >> 172.21.15.116:0/2360365429 conn(0x7f6afc0fbcb0 msgr2=0x7f6afc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b03f5b640 1 -- 172.21.15.116:0/2360365429 shutdown_connections 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b03f5b640 1 -- 172.21.15.116:0/2360365429 wait complete. 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b03f5b640 1 Processor -- start 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b03f5b640 1 -- start start 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b03f5b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6afc1000f0 0x7f6afc199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b03f5b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6afc100a50 0x7f6afc19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b03f5b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6afc1013f0 0x7f6afc1a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b03f5b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6afc10f660 con 0x7f6afc1013f0 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b03f5b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6afc10f4e0 con 0x7f6afc1000f0 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b03f5b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6afc10f7e0 con 0x7f6afc100a50 2024-03-19T01:11:59.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b024f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6afc1013f0 0x7f6afc1a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:59.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b01cf7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6afc1000f0 0x7f6afc199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:59.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b01cf7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6afc1000f0 0x7f6afc199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:54462/0 (socket says 172.21.15.116:54462) 2024-03-19T01:11:59.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b024f8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6afc1013f0 0x7f6afc1a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59716/0 (socket says 172.21.15.116:59716) 2024-03-19T01:11:59.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b01cf7640 1 -- 172.21.15.116:0/4174559816 learned_addr learned my addr 172.21.15.116:0/4174559816 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:11:59.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.818+0000 7f6b014f6640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6afc100a50 0x7f6afc19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:59.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.819+0000 7f6b01cf7640 1 -- 172.21.15.116:0/4174559816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6afc100a50 msgr2=0x7f6afc19a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.819+0000 7f6b01cf7640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6afc100a50 0x7f6afc19a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.819+0000 7f6b01cf7640 1 -- 172.21.15.116:0/4174559816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6afc1013f0 msgr2=0x7f6afc1a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:11:59.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.819+0000 7f6b01cf7640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6afc1013f0 0x7f6afc1a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:11:59.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.819+0000 7f6b01cf7640 1 -- 172.21.15.116:0/4174559816 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6afc1a1980 con 0x7f6afc1000f0 2024-03-19T01:11:59.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.819+0000 7f6b024f8640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6afc1013f0 0x7f6afc1a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:11:59.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.819+0000 7f6b01cf7640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6afc1000f0 0x7f6afc199d60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00e9f0 tx=0x7f6aec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:59.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.819+0000 7f6afaffd640 1 -- 172.21.15.116:0/4174559816 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec00cd80 con 0x7f6afc1000f0 2024-03-19T01:11:59.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.819+0000 7f6b03f5b640 1 -- 172.21.15.116:0/4174559816 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6afc1a1c70 con 0x7f6afc1000f0 2024-03-19T01:11:59.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.819+0000 7f6afaffd640 1 -- 172.21.15.116:0/4174559816 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6aec010910 con 0x7f6afc1000f0 2024-03-19T01:11:59.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.819+0000 7f6afaffd640 1 -- 172.21.15.116:0/4174559816 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec0188f0 con 0x7f6afc1000f0 2024-03-19T01:11:59.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.819+0000 7f6b03f5b640 1 -- 172.21.15.116:0/4174559816 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6afc1a2190 con 0x7f6afc1000f0 2024-03-19T01:11:59.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.820+0000 7f6b03f5b640 1 -- 172.21.15.116:0/4174559816 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6afc10b080 con 0x7f6afc1000f0 2024-03-19T01:11:59.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.821+0000 7f6afaffd640 1 -- 172.21.15.116:0/4174559816 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6aec010ac0 con 0x7f6afc1000f0 2024-03-19T01:11:59.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.822+0000 7f6afaffd640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6ae40a2800 0x7f6ae40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:11:59.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.822+0000 7f6b014f6640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6ae40a2800 0x7f6ae40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:11:59.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.822+0000 7f6afaffd640 1 -- 172.21.15.116:0/4174559816 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6aec0cee90 con 0x7f6afc1000f0 2024-03-19T01:11:59.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.822+0000 7f6b014f6640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6ae40a2800 0x7f6ae40a4cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6af002c7b0 tx=0x7f6af0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:11:59.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:11:59.831+0000 7f6afaffd640 1 -- 172.21.15.116:0/4174559816 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6aec09b860 con 0x7f6afc1000f0 2024-03-19T01:12:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.011+0000 7f6b03f5b640 1 -- 172.21.15.116:0/4174559816 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6afc062420 con 0x7f6afc1000f0 2024-03-19T01:12:00.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.011+0000 7f6afaffd640 1 -- 172.21.15.116:0/4174559816 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f6aec09f900 con 0x7f6afc1000f0 2024-03-19T01:12:00.012 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T01:12:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.013+0000 7f6b03f5b640 1 -- 172.21.15.116:0/4174559816 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6ae40a2800 msgr2=0x7f6ae40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.013+0000 7f6b03f5b640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6ae40a2800 0x7f6ae40a4cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6af002c7b0 tx=0x7f6af0005fd0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.013+0000 7f6b03f5b640 1 -- 172.21.15.116:0/4174559816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6afc1000f0 msgr2=0x7f6afc199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.013+0000 7f6b03f5b640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6afc1000f0 0x7f6afc199d60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6aec00e9f0 tx=0x7f6aec00eeb0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.013+0000 7f6b03f5b640 1 -- 172.21.15.116:0/4174559816 shutdown_connections 2024-03-19T01:12:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.013+0000 7f6b03f5b640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6afc1013f0 0x7f6afc1a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.013+0000 7f6b03f5b640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6afc100a50 0x7f6afc19a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.013+0000 7f6b03f5b640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6ae40a2800 0x7f6ae40a4cc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.013+0000 7f6b03f5b640 1 --2- 172.21.15.116:0/4174559816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6afc1000f0 0x7f6afc199d60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.013+0000 7f6b03f5b640 1 -- 172.21.15.116:0/4174559816 >> 172.21.15.116:0/4174559816 conn(0x7f6afc0fbcb0 msgr2=0x7f6afc0fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.013+0000 7f6b03f5b640 1 -- 172.21.15.116:0/4174559816 shutdown_connections 2024-03-19T01:12:00.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.013+0000 7f6b03f5b640 1 -- 172.21.15.116:0/4174559816 wait complete. 2024-03-19T01:12:00.025 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:12:00.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.127+0000 7f4723f90640 1 -- 172.21.15.116:0/3839230054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f471c101930 msgr2=0x7f471c101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.127+0000 7f4723f90640 1 --2- 172.21.15.116:0/3839230054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f471c101930 0x7f471c101db0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f470c009900 tx=0x7f470c02bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:00.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.127+0000 7f4723f90640 1 -- 172.21.15.116:0/3839230054 shutdown_connections 2024-03-19T01:12:00.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.127+0000 7f4723f90640 1 --2- 172.21.15.116:0/3839230054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f471c1022f0 0x7f471c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.127+0000 7f4723f90640 1 --2- 172.21.15.116:0/3839230054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f471c101930 0x7f471c101db0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.127+0000 7f4723f90640 1 --2- 172.21.15.116:0/3839230054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f471c100740 0x7f471c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.127+0000 7f4723f90640 1 -- 172.21.15.116:0/3839230054 >> 172.21.15.116:0/3839230054 conn(0x7f471c0fbcb0 msgr2=0x7f471c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:00.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.127+0000 7f4723f90640 1 -- 172.21.15.116:0/3839230054 shutdown_connections 2024-03-19T01:12:00.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.127+0000 7f4723f90640 1 -- 172.21.15.116:0/3839230054 wait complete. 2024-03-19T01:12:00.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f4723f90640 1 Processor -- start 2024-03-19T01:12:00.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f4723f90640 1 -- start start 2024-03-19T01:12:00.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f4723f90640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f471c100740 0x7f471c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:00.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f4723f90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f471c101930 0x7f471c198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:00.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f4723f90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f471c1022f0 0x7f471c19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:00.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f4723f90640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f471c10d4e0 con 0x7f471c1022f0 2024-03-19T01:12:00.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f4723f90640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f471c10d360 con 0x7f471c100740 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f4723f90640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f471c10d660 con 0x7f471c101930 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f472252d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f471c1022f0 0x7f471c19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f472252d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f471c1022f0 0x7f471c19f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59732/0 (socket says 172.21.15.116:59732) 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f472252d640 1 -- 172.21.15.116:0/4144606068 learned_addr learned my addr 172.21.15.116:0/4144606068 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f4721d2c640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f471c100740 0x7f471c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f472152b640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f471c101930 0x7f471c198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f472252d640 1 -- 172.21.15.116:0/4144606068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f471c101930 msgr2=0x7f471c198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f472252d640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f471c101930 0x7f471c198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f472252d640 1 -- 172.21.15.116:0/4144606068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f471c100740 msgr2=0x7f471c197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f472252d640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f471c100740 0x7f471c197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.128+0000 7f472252d640 1 -- 172.21.15.116:0/4144606068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f471c19f730 con 0x7f471c1022f0 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.129+0000 7f4721d2c640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f471c100740 0x7f471c197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.129+0000 7f472252d640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f471c1022f0 0x7f471c19f0c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f471400eb10 tx=0x7f471400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.129+0000 7f4712ffd640 1 -- 172.21.15.116:0/4144606068 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4714004b20 con 0x7f471c1022f0 2024-03-19T01:12:00.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.129+0000 7f4712ffd640 1 -- 172.21.15.116:0/4144606068 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4714004cc0 con 0x7f471c1022f0 2024-03-19T01:12:00.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.129+0000 7f4723f90640 1 -- 172.21.15.116:0/4144606068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f471c19fa20 con 0x7f471c1022f0 2024-03-19T01:12:00.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.129+0000 7f4712ffd640 1 -- 172.21.15.116:0/4144606068 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4714010760 con 0x7f471c1022f0 2024-03-19T01:12:00.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.129+0000 7f4723f90640 1 -- 172.21.15.116:0/4144606068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f471c19fef0 con 0x7f471c1022f0 2024-03-19T01:12:00.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.130+0000 7f4723f90640 1 -- 172.21.15.116:0/4144606068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f471c107a60 con 0x7f471c1022f0 2024-03-19T01:12:00.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.131+0000 7f4712ffd640 1 -- 172.21.15.116:0/4144606068 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f4714005020 con 0x7f471c1022f0 2024-03-19T01:12:00.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.131+0000 7f4712ffd640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47080a2800 0x7f47080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:00.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.131+0000 7f4721d2c640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47080a2800 0x7f47080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:00.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.132+0000 7f4712ffd640 1 -- 172.21.15.116:0/4144606068 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f47140ceff0 con 0x7f471c1022f0 2024-03-19T01:12:00.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.132+0000 7f4721d2c640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47080a2800 0x7f47080a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f471c101790 tx=0x7f4718005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:00.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.136+0000 7f4712ffd640 1 -- 172.21.15.116:0/4144606068 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f471409c070 con 0x7f471c1022f0 2024-03-19T01:12:00.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.357+0000 7f4723f90640 1 -- 172.21.15.116:0/4144606068 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f471c062420 con 0x7f471c1022f0 2024-03-19T01:12:00.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.358+0000 7f4712ffd640 1 -- 172.21.15.116:0/4144606068 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11938 (secure 0 0 0) 0x7f471409b970 con 0x7f471c1022f0 2024-03-19T01:12:00.359 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:00.359 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T01:10:32.487607+0000","modified":"2024-03-19T01:10:38.982004+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24488,"mds_2":24473,"mds_3":24460,"mds_4":24445},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1242639081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1242639081},{"type":"v1","addr":"172.21.15.116:6839","nonce":1242639081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24394":{"gid":24394,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":3,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.162:6839/441486392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":441486392},{"type":"v1","addr":"172.21.15.162:6839","nonce":441486392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"a","rank":1,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6841/3343188504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3343188504},{"type":"v1","addr":"172.21.15.116:6841","nonce":3343188504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1601537427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1601537427},{"type":"v1","addr":"172.21.15.156:6839","nonce":1601537427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24473,24445,24500,24460,24488]},"id":1}]} 2024-03-19T01:12:00.360 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2024-03-19T01:12:00.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.360+0000 7f4723f90640 1 -- 172.21.15.116:0/4144606068 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47080a2800 msgr2=0x7f47080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.360+0000 7f4723f90640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47080a2800 0x7f47080a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f471c101790 tx=0x7f4718005f90 comp rx=0 tx=0).stop 2024-03-19T01:12:00.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.360+0000 7f4723f90640 1 -- 172.21.15.116:0/4144606068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f471c1022f0 msgr2=0x7f471c19f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.360+0000 7f4723f90640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f471c1022f0 0x7f471c19f0c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f471400eb10 tx=0x7f471400c2d0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.360+0000 7f4723f90640 1 -- 172.21.15.116:0/4144606068 shutdown_connections 2024-03-19T01:12:00.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.360+0000 7f4723f90640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f471c1022f0 0x7f471c19f0c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.360+0000 7f4723f90640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f471c101930 0x7f471c198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.360+0000 7f4723f90640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47080a2800 0x7f47080a4cc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.360+0000 7f4723f90640 1 --2- 172.21.15.116:0/4144606068 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f471c100740 0x7f471c197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.360+0000 7f4723f90640 1 -- 172.21.15.116:0/4144606068 >> 172.21.15.116:0/4144606068 conn(0x7f471c0fbcb0 msgr2=0x7f471c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:00.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.360+0000 7f4723f90640 1 -- 172.21.15.116:0/4144606068 shutdown_connections 2024-03-19T01:12:00.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.360+0000 7f4723f90640 1 -- 172.21.15.116:0/4144606068 wait complete. 2024-03-19T01:12:00.371 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:00 smithi116 ceph-mon[62701]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 12 op/s 2024-03-19T01:12:00.371 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:00 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2855338483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:12:00.371 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:00 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2959196560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:12:00.371 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:00 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4174559816' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:12:00.372 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:12:00.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:00 smithi156 ceph-mon[66799]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 12 op/s 2024-03-19T01:12:00.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:00 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2855338483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:12:00.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:00 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2959196560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:12:00.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:00 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/4174559816' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:12:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.473+0000 7f2b27e7e640 1 -- 172.21.15.116:0/3161120267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b20103410 msgr2=0x7f2b20109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.473+0000 7f2b27e7e640 1 --2- 172.21.15.116:0/3161120267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b20103410 0x7f2b20109ca0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2b18009ca0 tx=0x7f2b1802c0f0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.473+0000 7f2b27e7e640 1 -- 172.21.15.116:0/3161120267 shutdown_connections 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 --2- 172.21.15.116:0/3161120267 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b20103410 0x7f2b20109ca0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 --2- 172.21.15.116:0/3161120267 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b20102a50 0x7f2b20102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 --2- 172.21.15.116:0/3161120267 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b20101860 0x7f2b20101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 -- 172.21.15.116:0/3161120267 >> 172.21.15.116:0/3161120267 conn(0x7f2b200fcdd0 msgr2=0x7f2b200ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 -- 172.21.15.116:0/3161120267 shutdown_connections 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 -- 172.21.15.116:0/3161120267 wait complete. 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 Processor -- start 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 -- start start 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b20101860 0x7f2b20197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b20102a50 0x7f2b20198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b20103410 0x7f2b2019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2010d4e0 con 0x7f2b20102a50 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2010d360 con 0x7f2b20103410 2024-03-19T01:12:00.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b27e7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2010d660 con 0x7f2b20101860 2024-03-19T01:12:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b25419640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b20102a50 0x7f2b20198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b25419640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b20102a50 0x7f2b20198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59740/0 (socket says 172.21.15.116:59740) 2024-03-19T01:12:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b25c1a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b20101860 0x7f2b20197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b25419640 1 -- 172.21.15.116:0/2972121464 learned_addr learned my addr 172.21.15.116:0/2972121464 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.474+0000 7f2b2641b640 1 --2- 172.21.15.116:0/2972121464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b20103410 0x7f2b2019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.475+0000 7f2b25419640 1 -- 172.21.15.116:0/2972121464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b20101860 msgr2=0x7f2b20197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.475+0000 7f2b25419640 1 --2- 172.21.15.116:0/2972121464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b20101860 0x7f2b20197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.475+0000 7f2b25419640 1 -- 172.21.15.116:0/2972121464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b20103410 msgr2=0x7f2b2019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.475+0000 7f2b25419640 1 --2- 172.21.15.116:0/2972121464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b20103410 0x7f2b2019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.475+0000 7f2b25419640 1 -- 172.21.15.116:0/2972121464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b2019f710 con 0x7f2b20102a50 2024-03-19T01:12:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.475+0000 7f2b25419640 1 --2- 172.21.15.116:0/2972121464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b20102a50 0x7f2b20198030 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000ea60 tx=0x7f2b1000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.475+0000 7f2b16ffd640 1 -- 172.21.15.116:0/2972121464 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1000cde0 con 0x7f2b20102a50 2024-03-19T01:12:00.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.475+0000 7f2b16ffd640 1 -- 172.21.15.116:0/2972121464 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b100035f0 con 0x7f2b20102a50 2024-03-19T01:12:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.475+0000 7f2b16ffd640 1 -- 172.21.15.116:0/2972121464 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b10010670 con 0x7f2b20102a50 2024-03-19T01:12:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.475+0000 7f2b27e7e640 1 -- 172.21.15.116:0/2972121464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b2019fa00 con 0x7f2b20102a50 2024-03-19T01:12:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.475+0000 7f2b27e7e640 1 -- 172.21.15.116:0/2972121464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b2019ff50 con 0x7f2b20102a50 2024-03-19T01:12:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.476+0000 7f2b27e7e640 1 -- 172.21.15.116:0/2972121464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b20108b80 con 0x7f2b20102a50 2024-03-19T01:12:00.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.478+0000 7f2b16ffd640 1 -- 172.21.15.116:0/2972121464 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f2b100037a0 con 0x7f2b20102a50 2024-03-19T01:12:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.478+0000 7f2b16ffd640 1 --2- 172.21.15.116:0/2972121464 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2b0c0a2800 0x7f2b0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.478+0000 7f2b16ffd640 1 -- 172.21.15.116:0/2972121464 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2b100cee10 con 0x7f2b20102a50 2024-03-19T01:12:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.478+0000 7f2b25c1a640 1 --2- 172.21.15.116:0/2972121464 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2b0c0a2800 0x7f2b0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:00.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.479+0000 7f2b25c1a640 1 --2- 172.21.15.116:0/2972121464 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2b0c0a2800 0x7f2b0c0a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2b201028b0 tx=0x7f2b1c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:00.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.481+0000 7f2b16ffd640 1 -- 172.21.15.116:0/2972121464 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2b1009bf20 con 0x7f2b20102a50 2024-03-19T01:12:00.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:00 smithi162 ceph-mon[67178]: pgmap v333: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 12 op/s 2024-03-19T01:12:00.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:00 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2855338483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:12:00.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:00 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2959196560' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:12:00.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:00 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/4174559816' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:12:00.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.729+0000 7f2b27e7e640 1 -- 172.21.15.116:0/2972121464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2b2002d3c0 con 0x7f2b20102a50 2024-03-19T01:12:00.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.730+0000 7f2b16ffd640 1 -- 172.21.15.116:0/2972121464 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22804 (secure 0 0 0) 0x7f2b1009b820 con 0x7f2b20102a50 2024-03-19T01:12:00.731 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:00.732 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:11:28.144841+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:12:00.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.732+0000 7f2b27e7e640 1 -- 172.21.15.116:0/2972121464 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2b0c0a2800 msgr2=0x7f2b0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.732+0000 7f2b27e7e640 1 --2- 172.21.15.116:0/2972121464 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2b0c0a2800 0x7f2b0c0a4cc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2b201028b0 tx=0x7f2b1c005f90 comp rx=0 tx=0).stop 2024-03-19T01:12:00.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.732+0000 7f2b27e7e640 1 -- 172.21.15.116:0/2972121464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b20102a50 msgr2=0x7f2b20198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.732+0000 7f2b27e7e640 1 --2- 172.21.15.116:0/2972121464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b20102a50 0x7f2b20198030 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000ea60 tx=0x7f2b1000ef20 comp rx=0 tx=0).stop 2024-03-19T01:12:00.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.732+0000 7f2b27e7e640 1 -- 172.21.15.116:0/2972121464 shutdown_connections 2024-03-19T01:12:00.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.732+0000 7f2b27e7e640 1 --2- 172.21.15.116:0/2972121464 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2b0c0a2800 0x7f2b0c0a4cc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.732+0000 7f2b27e7e640 1 --2- 172.21.15.116:0/2972121464 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b20103410 0x7f2b2019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.732+0000 7f2b27e7e640 1 --2- 172.21.15.116:0/2972121464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b20102a50 0x7f2b20198030 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.732+0000 7f2b27e7e640 1 --2- 172.21.15.116:0/2972121464 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2b20101860 0x7f2b20197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.732+0000 7f2b27e7e640 1 -- 172.21.15.116:0/2972121464 >> 172.21.15.116:0/2972121464 conn(0x7f2b200fcdd0 msgr2=0x7f2b20105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:00.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.732+0000 7f2b27e7e640 1 -- 172.21.15.116:0/2972121464 shutdown_connections 2024-03-19T01:12:00.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.733+0000 7f2b27e7e640 1 -- 172.21.15.116:0/2972121464 wait complete. 2024-03-19T01:12:00.747 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:12:00.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.848+0000 7ff9b9977640 1 -- 172.21.15.116:0/1466456025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9b4101930 msgr2=0x7ff9b4101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.848+0000 7ff9b9977640 1 --2- 172.21.15.116:0/1466456025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9b4101930 0x7ff9b4101db0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009900 tx=0x7ff9a802bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.848+0000 7ff9b9977640 1 -- 172.21.15.116:0/1466456025 shutdown_connections 2024-03-19T01:12:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.848+0000 7ff9b9977640 1 --2- 172.21.15.116:0/1466456025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9b41022f0 0x7ff9b4108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.848+0000 7ff9b9977640 1 --2- 172.21.15.116:0/1466456025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9b4101930 0x7ff9b4101db0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.848+0000 7ff9b9977640 1 --2- 172.21.15.116:0/1466456025 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9b4100740 0x7ff9b4100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.848+0000 7ff9b9977640 1 -- 172.21.15.116:0/1466456025 >> 172.21.15.116:0/1466456025 conn(0x7ff9b40fbcb0 msgr2=0x7ff9b40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.848+0000 7ff9b9977640 1 -- 172.21.15.116:0/1466456025 shutdown_connections 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.848+0000 7ff9b9977640 1 -- 172.21.15.116:0/1466456025 wait complete. 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.849+0000 7ff9b9977640 1 Processor -- start 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.849+0000 7ff9b9977640 1 -- start start 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.849+0000 7ff9b9977640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9b4100740 0x7ff9b4197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.849+0000 7ff9b9977640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9b4101930 0x7ff9b4198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.849+0000 7ff9b9977640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9b41022f0 0x7ff9b419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.849+0000 7ff9b9977640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b410c3a0 con 0x7ff9b41022f0 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.849+0000 7ff9b9977640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b410c220 con 0x7ff9b4100740 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.849+0000 7ff9b9977640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b410c520 con 0x7ff9b4101930 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.849+0000 7ff9b37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9b41022f0 0x7ff9b419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.849+0000 7ff9b37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9b41022f0 0x7ff9b419f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59760/0 (socket says 172.21.15.116:59760) 2024-03-19T01:12:00.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.849+0000 7ff9b37fe640 1 -- 172.21.15.116:0/1500918162 learned_addr learned my addr 172.21.15.116:0/1500918162 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.849+0000 7ff9b2ffd640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9b4100740 0x7ff9b4197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.849+0000 7ff9b27fc640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9b4101930 0x7ff9b4198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.850+0000 7ff9b37fe640 1 -- 172.21.15.116:0/1500918162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9b4101930 msgr2=0x7ff9b4198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.850+0000 7ff9b37fe640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9b4101930 0x7ff9b4198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.850+0000 7ff9b37fe640 1 -- 172.21.15.116:0/1500918162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9b4100740 msgr2=0x7ff9b4197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.850+0000 7ff9b37fe640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9b4100740 0x7ff9b4197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.850+0000 7ff9b37fe640 1 -- 172.21.15.116:0/1500918162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b419f760 con 0x7ff9b41022f0 2024-03-19T01:12:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.850+0000 7ff9b37fe640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9b41022f0 0x7ff9b419f0f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00eb10 tx=0x7ff9ac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.850+0000 7ff9b8975640 1 -- 172.21.15.116:0/1500918162 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac004b20 con 0x7ff9b41022f0 2024-03-19T01:12:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.850+0000 7ff9b9977640 1 -- 172.21.15.116:0/1500918162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9b419fa50 con 0x7ff9b41022f0 2024-03-19T01:12:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.850+0000 7ff9b8975640 1 -- 172.21.15.116:0/1500918162 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9ac004cc0 con 0x7ff9b41022f0 2024-03-19T01:12:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.850+0000 7ff9b8975640 1 -- 172.21.15.116:0/1500918162 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac010760 con 0x7ff9b41022f0 2024-03-19T01:12:00.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.850+0000 7ff9b9977640 1 -- 172.21.15.116:0/1500918162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b419ff20 con 0x7ff9b41022f0 2024-03-19T01:12:00.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.851+0000 7ff9b9977640 1 -- 172.21.15.116:0/1500918162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9b41a0150 con 0x7ff9b41022f0 2024-03-19T01:12:00.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.853+0000 7ff9b8975640 1 -- 172.21.15.116:0/1500918162 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7ff9ac005020 con 0x7ff9b41022f0 2024-03-19T01:12:00.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.853+0000 7ff9b8975640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff99c0a2800 0x7ff99c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:00.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.853+0000 7ff9b2ffd640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff99c0a2800 0x7ff99c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:00.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.853+0000 7ff9b8975640 1 -- 172.21.15.116:0/1500918162 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7ff9ac0ceff0 con 0x7ff9b41022f0 2024-03-19T01:12:00.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.854+0000 7ff9b2ffd640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff99c0a2800 0x7ff99c0a4cc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40095f0 tx=0x7ff9a4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:00.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:00.858+0000 7ff9b8975640 1 -- 172.21.15.116:0/1500918162 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff9ac09c070 con 0x7ff9b41022f0 2024-03-19T01:12:01.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.075+0000 7ff9b9977640 1 -- 172.21.15.116:0/1500918162 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9b41a08b0 con 0x7ff9b41022f0 2024-03-19T01:12:01.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.076+0000 7ff9b8975640 1 -- 172.21.15.116:0/1500918162 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11938 (secure 0 0 0) 0x7ff9ac09b970 con 0x7ff9b41022f0 2024-03-19T01:12:01.077 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:01.077 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T01:10:32.487607+0000","modified":"2024-03-19T01:10:38.982004+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24488,"mds_2":24473,"mds_3":24460,"mds_4":24445},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1242639081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1242639081},{"type":"v1","addr":"172.21.15.116:6839","nonce":1242639081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24394":{"gid":24394,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":3,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.162:6839/441486392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":441486392},{"type":"v1","addr":"172.21.15.162:6839","nonce":441486392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"a","rank":1,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6841/3343188504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3343188504},{"type":"v1","addr":"172.21.15.116:6841","nonce":3343188504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1601537427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1601537427},{"type":"v1","addr":"172.21.15.156:6839","nonce":1601537427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24473,24445,24500,24460,24488]},"id":1}]} 2024-03-19T01:12:01.077 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2024-03-19T01:12:01.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.078+0000 7ff9b9977640 1 -- 172.21.15.116:0/1500918162 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff99c0a2800 msgr2=0x7ff99c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.078+0000 7ff9b9977640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff99c0a2800 0x7ff99c0a4cc0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff9a40095f0 tx=0x7ff9a4005f90 comp rx=0 tx=0).stop 2024-03-19T01:12:01.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.078+0000 7ff9b9977640 1 -- 172.21.15.116:0/1500918162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9b41022f0 msgr2=0x7ff9b419f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.078+0000 7ff9b9977640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9b41022f0 0x7ff9b419f0f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00eb10 tx=0x7ff9ac00c2d0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.078+0000 7ff9b9977640 1 -- 172.21.15.116:0/1500918162 shutdown_connections 2024-03-19T01:12:01.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.078+0000 7ff9b9977640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9b41022f0 0x7ff9b419f0f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.078+0000 7ff9b9977640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff9b4101930 0x7ff9b4198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.078+0000 7ff9b9977640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff99c0a2800 0x7ff99c0a4cc0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.078+0000 7ff9b9977640 1 --2- 172.21.15.116:0/1500918162 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff9b4100740 0x7ff9b4197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.078+0000 7ff9b9977640 1 -- 172.21.15.116:0/1500918162 >> 172.21.15.116:0/1500918162 conn(0x7ff9b40fbcb0 msgr2=0x7ff9b40fd790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.078+0000 7ff9b9977640 1 -- 172.21.15.116:0/1500918162 shutdown_connections 2024-03-19T01:12:01.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.078+0000 7ff9b9977640 1 -- 172.21.15.116:0/1500918162 wait complete. 2024-03-19T01:12:01.090 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-19T01:10:32.487607+0000', 'modified': '2024-03-19T01:10:38.982004+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24500, 'mds_1': 24488, 'mds_2': 24473, 'mds_3': 24460, 'mds_4': 24445}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14613': {'gid': 14613, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6839/1242639081', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 1242639081}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 1242639081}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24394': {'gid': 24394, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6833/3113382416', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6832', 'nonce': 3113382416}, {'type': 'v1', 'addr': '172.21.15.162:6833', 'nonce': 3113382416}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24410': {'gid': 24410, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6833/1647822466', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6832', 'nonce': 1647822466}, {'type': 'v1', 'addr': '172.21.15.156:6833', 'nonce': 1647822466}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24427': {'gid': 24427, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6835/1592939032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6834', 'nonce': 1592939032}, {'type': 'v1', 'addr': '172.21.15.162:6835', 'nonce': 1592939032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24434': {'gid': 24434, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6835/1738017780', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6834', 'nonce': 1738017780}, {'type': 'v1', 'addr': '172.21.15.156:6835', 'nonce': 1738017780}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'c', 'rank': 4, 'incarnation': 22, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.162:6837/585664575', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6836', 'nonce': 585664575}, {'type': 'v1', 'addr': '172.21.15.162:6837', 'nonce': 585664575}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 3, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.162:6839/441486392', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6838', 'nonce': 441486392}, {'type': 'v1', 'addr': '172.21.15.162:6839', 'nonce': 441486392}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.156:6837/4147060396', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6836', 'nonce': 4147060396}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 4147060396}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'a', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.116:6841/3343188504', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6840', 'nonce': 3343188504}, {'type': 'v1', 'addr': '172.21.15.116:6841', 'nonce': 3343188504}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.156:6839/1601537427', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 1601537427}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 1601537427}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24500, 'qdb_cluster': [24473, 24445, 24500, 24460, 24488]} 2024-03-19T01:12:01.090 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-19T01:12:01.090 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-19T01:12:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.202+0000 7f417f1b3640 1 -- 172.21.15.116:0/3438951076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178101350 msgr2=0x7f4178108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.202+0000 7f417f1b3640 1 --2- 172.21.15.116:0/3438951076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178101350 0x7f4178108bf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4170009ca0 tx=0x7f417002c140 comp rx=0 tx=0).stop 2024-03-19T01:12:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.202+0000 7f417f1b3640 1 -- 172.21.15.116:0/3438951076 shutdown_connections 2024-03-19T01:12:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.202+0000 7f417f1b3640 1 --2- 172.21.15.116:0/3438951076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178101350 0x7f4178108bf0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.202+0000 7f417f1b3640 1 --2- 172.21.15.116:0/3438951076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4178100990 0x7f4178100e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.202+0000 7f417f1b3640 1 --2- 172.21.15.116:0/3438951076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4178101dd0 0x7f41781021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.202+0000 7f417f1b3640 1 -- 172.21.15.116:0/3438951076 >> 172.21.15.116:0/3438951076 conn(0x7f41780fb420 msgr2=0x7f41780fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.202+0000 7f417f1b3640 1 -- 172.21.15.116:0/3438951076 shutdown_connections 2024-03-19T01:12:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.202+0000 7f417f1b3640 1 -- 172.21.15.116:0/3438951076 wait complete. 2024-03-19T01:12:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417f1b3640 1 Processor -- start 2024-03-19T01:12:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417f1b3640 1 -- start start 2024-03-19T01:12:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417f1b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178100990 0x7f41781003f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417f1b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4178101350 0x7f417819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417f1b3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4178101dd0 0x7f41781a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417f1b3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f417810c3e0 con 0x7f4178100990 2024-03-19T01:12:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417f1b3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f417810c260 con 0x7f4178101dd0 2024-03-19T01:12:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417f1b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f417810c560 con 0x7f4178101350 2024-03-19T01:12:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417cf4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178100990 0x7f41781003f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417cf4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178100990 0x7f41781003f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59778/0 (socket says 172.21.15.116:59778) 2024-03-19T01:12:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417cf4f640 1 -- 172.21.15.116:0/2349323281 learned_addr learned my addr 172.21.15.116:0/2349323281 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f4177fff640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4178101350 0x7f417819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417d750640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4178101dd0 0x7f41781a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417cf4f640 1 -- 172.21.15.116:0/2349323281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4178101350 msgr2=0x7f417819ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417cf4f640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4178101350 0x7f417819ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417cf4f640 1 -- 172.21.15.116:0/2349323281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4178101dd0 msgr2=0x7f41781a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417cf4f640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4178101dd0 0x7f41781a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417cf4f640 1 -- 172.21.15.116:0/2349323281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41781a1950 con 0x7f4178100990 2024-03-19T01:12:01.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417cf4f640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178100990 0x7f41781003f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f416c00e9f0 tx=0x7f416c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f4175ffb640 1 -- 172.21.15.116:0/2349323281 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c00cd80 con 0x7f4178100990 2024-03-19T01:12:01.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f4175ffb640 1 -- 172.21.15.116:0/2349323281 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f416c005500 con 0x7f4178100990 2024-03-19T01:12:01.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f4175ffb640 1 -- 172.21.15.116:0/2349323281 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c010690 con 0x7f4178100990 2024-03-19T01:12:01.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.203+0000 7f417f1b3640 1 -- 172.21.15.116:0/2349323281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41781a1be0 con 0x7f4178100990 2024-03-19T01:12:01.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.204+0000 7f417f1b3640 1 -- 172.21.15.116:0/2349323281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41781a2130 con 0x7f4178100990 2024-03-19T01:12:01.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.206+0000 7f4175ffb640 1 -- 172.21.15.116:0/2349323281 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f416c0046c0 con 0x7f4178100990 2024-03-19T01:12:01.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.206+0000 7f4181d99740 1 Processor -- start 2024-03-19T01:12:01.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.206+0000 7f4181d99740 1 -- start start 2024-03-19T01:12:01.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.206+0000 7f4181d99740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a4751410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.206+0000 7f4181d99740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a47519e0 0x5559a47564e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.206+0000 7f4181d99740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a4756b50 0x5559a4758fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.206+0000 7f4181d99740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5559a46b5de0 con 0x5559a4756b50 2024-03-19T01:12:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.206+0000 7f4181d99740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5559a465ec80 con 0x5559a47519e0 2024-03-19T01:12:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.206+0000 7f4181d99740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5559a46b1e80 con 0x5559a4750ff0 2024-03-19T01:12:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.206+0000 7f4175ffb640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f41680a28b0 0x7f41680a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.206+0000 7f4177fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a4756b50 0x5559a4758fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.206+0000 7f417d750640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a4751410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f4177fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a4756b50 0x5559a4758fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59782/0 (socket says 172.21.15.116:59782) 2024-03-19T01:12:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f417cf4f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a47519e0 0x5559a47564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f4177fff640 1 -- 172.21.15.116:0/2978882779 learned_addr learned my addr 172.21.15.116:0/2978882779 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f417d750640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a4751410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59600/0 (socket says 172.21.15.116:59600) 2024-03-19T01:12:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f417d750640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f41680a28b0 0x7f41680a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f4175ffb640 1 -- 172.21.15.116:0/2349323281 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f416c0cee00 con 0x7f4178100990 2024-03-19T01:12:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f4177fff640 1 -- 172.21.15.116:0/2978882779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 msgr2=0x5559a4751410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f4177fff640 1 --2- 172.21.15.116:0/2978882779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a4751410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f4177fff640 1 -- 172.21.15.116:0/2978882779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a47519e0 msgr2=0x5559a47564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f4177fff640 1 --2- 172.21.15.116:0/2978882779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a47519e0 0x5559a47564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f4177fff640 1 -- 172.21.15.116:0/2978882779 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559a4670fb0 con 0x5559a4756b50 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f4177fff640 1 --2- 172.21.15.116:0/2978882779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a4756b50 0x5559a4758fe0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4164009c50 tx=0x7f4164011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42cee05eac9e81f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f417d750640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f41680a28b0 0x7f41680a4d70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4170005cc0 tx=0x7f4170005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f4162ffd640 1 -- 172.21.15.116:0/2978882779 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164004030 con 0x5559a4756b50 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f4162ffd640 1 -- 172.21.15.116:0/2978882779 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41640041d0 con 0x5559a4756b50 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.207+0000 7f4162ffd640 1 -- 172.21.15.116:0/2978882779 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164019720 con 0x5559a4756b50 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.208+0000 7f4181d99740 1 -- 172.21.15.116:0/2978882779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a4756b50 msgr2=0x5559a4758fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.208+0000 7f4181d99740 1 --2- 172.21.15.116:0/2978882779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a4756b50 0x5559a4758fe0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4164009c50 tx=0x7f4164011810 comp rx=0 tx=0).stop 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.208+0000 7f4181d99740 1 -- 172.21.15.116:0/2978882779 shutdown_connections 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.208+0000 7f4181d99740 1 --2- 172.21.15.116:0/2978882779 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a4756b50 0x5559a4758fe0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.208+0000 7f4181d99740 1 --2- 172.21.15.116:0/2978882779 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a47519e0 0x5559a47564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.208+0000 7f4181d99740 1 --2- 172.21.15.116:0/2978882779 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a4751410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.208+0000 7f4181d99740 1 -- 172.21.15.116:0/2978882779 >> 172.21.15.116:0/2978882779 conn(0x5559a474c560 msgr2=0x5559a474c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.208+0000 7f4181d99740 1 -- 172.21.15.116:0/2978882779 shutdown_connections 2024-03-19T01:12:01.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.208+0000 7f4181d99740 1 -- 172.21.15.116:0/2978882779 wait complete. 2024-03-19T01:12:01.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.208+0000 7f4181d99740 1 -- start start 2024-03-19T01:12:01.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.208+0000 7f4181d99740 1 Processor -- start 2024-03-19T01:12:01.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f4181d99740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a475b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f4181d99740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a47519e0 0x5559a475b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f4181d99740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a4756b50 0x5559a47629c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f4181d99740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5559a4759b00 con 0x5559a4756b50 2024-03-19T01:12:01.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f4181d99740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5559a4671fc0 con 0x5559a47519e0 2024-03-19T01:12:01.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f4181d99740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5559a4759c80 con 0x5559a4750ff0 2024-03-19T01:12:01.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f4181d99740 10 client.0 _ll_register_callbacks cb 0x5559a46ada20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:01.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f417d750640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a4756b50 0x5559a47629c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f417cf4f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a475b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f4177fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a47519e0 0x5559a475b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f417cf4f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a475b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59610/0 (socket says 172.21.15.116:59610) 2024-03-19T01:12:01.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f417cf4f640 1 -- 172.21.15.116:0/1494374648 learned_addr learned my addr 172.21.15.116:0/1494374648 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f4177fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a47519e0 0x5559a475b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:54554/0 (socket says 172.21.15.116:54554) 2024-03-19T01:12:01.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f417cf4f640 1 -- 172.21.15.116:0/1494374648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a47519e0 msgr2=0x5559a475b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f417cf4f640 1 --2- 172.21.15.116:0/1494374648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a47519e0 0x5559a475b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f417cf4f640 1 -- 172.21.15.116:0/1494374648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a4756b50 msgr2=0x5559a47629c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f417cf4f640 1 --2- 172.21.15.116:0/1494374648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a4756b50 0x5559a47629c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f417cf4f640 1 -- 172.21.15.116:0/1494374648 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559a43ca2d0 con 0x5559a4750ff0 2024-03-19T01:12:01.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f417cf4f640 1 --2- 172.21.15.116:0/1494374648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a475b440 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f416c098150 tx=0x7f416c00aea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c12ee6e857bacdb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f4157fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:12:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f4157fff640 1 -- 172.21.15.116:0/1494374648 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c0b3020 con 0x5559a4750ff0 2024-03-19T01:12:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f4157fff640 1 -- 172.21.15.116:0/1494374648 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f416c0a3a90 con 0x5559a4750ff0 2024-03-19T01:12:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.209+0000 7f4181d99740 1 -- 172.21.15.116:0/1494374648 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41640041d0 con 0x5559a4750ff0 2024-03-19T01:12:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.210+0000 7f4157fff640 1 -- 172.21.15.116:0/1494374648 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f416c09be80 con 0x5559a4750ff0 2024-03-19T01:12:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.210+0000 7f4181d99740 10 client.24871 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.210+0000 7f4181d99740 1 -- 172.21.15.116:0/1494374648 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5559a44946b0 con 0x5559a4750ff0 2024-03-19T01:12:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.210+0000 7f4157fff640 1 -- 172.21.15.116:0/1494374648 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41640041d0 con 0x5559a4750ff0 2024-03-19T01:12:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.210+0000 7f4157fff640 1 -- 172.21.15.116:0/1494374648 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f416c097040 con 0x5559a4750ff0 2024-03-19T01:12:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.210+0000 7f4181d99740 10 client.24871 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.210+0000 7f4181d99740 10 client.24871 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-19T01:12:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.210+0000 7f4181d99740 20 client.24871 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.210+0000 7f4181d99740 1 --2- 172.21.15.116:0/1494374648 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x5559a4768350 0x5559a47697c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.210+0000 7f4181d99740 4 client.24871 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:12:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.210+0000 7f4181d99740 1 -- 172.21.15.116:0/1494374648 --> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559a46c3320 con 0x5559a4768350 2024-03-19T01:12:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.210+0000 7f4177fff640 1 --2- 172.21.15.116:0/1494374648 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x5559a4768350 0x5559a47697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4177fff640 1 --2- 172.21.15.116:0/1494374648 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x5559a4768350 0x5559a47697c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3b954d5a9fbeadb9 server_cookie=13bef0fcea83760a in_seq=0 out_seq=0 2024-03-19T01:12:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4157fff640 10 client.24871 ms_handle_connect on v2:172.21.15.162:6836/585664575 2024-03-19T01:12:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4157fff640 1 -- 172.21.15.116:0/1494374648 <== mds.4 v2:172.21.15.162:6836/585664575 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5559a46c3320 con 0x5559a4768350 2024-03-19T01:12:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4157fff640 10 client.24871 handle_command_reply: tid=0 2024-03-19T01:12:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4181d99740 1 client.24871 shutdown 2024-03-19T01:12:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4181d99740 1 -- 172.21.15.116:0/1494374648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 msgr2=0x5559a475b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4181d99740 1 --2- 172.21.15.116:0/1494374648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a475b440 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f416c098150 tx=0x7f416c00aea0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4181d99740 1 -- 172.21.15.116:0/1494374648 shutdown_connections 2024-03-19T01:12:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4181d99740 1 --2- 172.21.15.116:0/1494374648 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x5559a4768350 0x5559a47697c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4181d99740 1 --2- 172.21.15.116:0/1494374648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a4756b50 0x5559a47629c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4157fff640 0 client.24871 ms_handle_reset on v2:172.21.15.162:6836/585664575 2024-03-19T01:12:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4181d99740 1 --2- 172.21.15.116:0/1494374648 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a47519e0 0x5559a475b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4181d99740 1 --2- 172.21.15.116:0/1494374648 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a475b440 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.211+0000 7f4181d99740 1 -- 172.21.15.116:0/1494374648 >> 172.21.15.116:0/1494374648 conn(0x5559a474c560 msgr2=0x5559a474b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.212+0000 7f4181d99740 1 -- 172.21.15.116:0/1494374648 shutdown_connections 2024-03-19T01:12:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.212+0000 7f4181d99740 1 -- 172.21.15.116:0/1494374648 wait complete. 2024-03-19T01:12:01.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.212+0000 7f4181d99740 20 client.24871 trim_cache size 0 max 16384 2024-03-19T01:12:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.229+0000 7f4181d99740 1 Processor -- start 2024-03-19T01:12:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.229+0000 7f4181d99740 1 -- start start 2024-03-19T01:12:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4181d99740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a4750ff0 0x5559a4758f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4181d99740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 0x5559a4762950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4181d99740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4756b50 0x5559a4752240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4181d99740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5559a4767b00 con 0x5559a47519e0 2024-03-19T01:12:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4181d99740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5559a47681d0 con 0x5559a4750ff0 2024-03-19T01:12:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4181d99740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5559a47688a0 con 0x5559a4756b50 2024-03-19T01:12:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4177fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 0x5559a4762950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4177fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 0x5559a4762950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59796/0 (socket says 172.21.15.116:59796) 2024-03-19T01:12:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4177fff640 1 -- 172.21.15.116:0/2250669241 learned_addr learned my addr 172.21.15.116:0/2250669241 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f417cf4f640 1 --2- 172.21.15.116:0/2250669241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a4750ff0 0x5559a4758f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f417d750640 1 --2- 172.21.15.116:0/2250669241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4756b50 0x5559a4752240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4177fff640 1 -- 172.21.15.116:0/2250669241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4756b50 msgr2=0x5559a4752240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4177fff640 1 --2- 172.21.15.116:0/2250669241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4756b50 0x5559a4752240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4177fff640 1 -- 172.21.15.116:0/2250669241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a4750ff0 msgr2=0x5559a4758f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4177fff640 1 --2- 172.21.15.116:0/2250669241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a4750ff0 0x5559a4758f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4177fff640 1 -- 172.21.15.116:0/2250669241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559a4747670 con 0x5559a47519e0 2024-03-19T01:12:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f4177fff640 1 --2- 172.21.15.116:0/2250669241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 0x5559a4762950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x5559a474d4d0 tx=0x7f41640099a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db8a19e121540a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f41627fc640 1 -- 172.21.15.116:0/2250669241 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416401f040 con 0x5559a47519e0 2024-03-19T01:12:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f41627fc640 1 -- 172.21.15.116:0/2250669241 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4164003ad0 con 0x5559a47519e0 2024-03-19T01:12:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.230+0000 7f41627fc640 1 -- 172.21.15.116:0/2250669241 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416402e440 con 0x5559a47519e0 2024-03-19T01:12:01.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.231+0000 7f4181d99740 1 -- 172.21.15.116:0/2250669241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 msgr2=0x5559a4762950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.231+0000 7f4181d99740 1 --2- 172.21.15.116:0/2250669241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 0x5559a4762950 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x5559a474d4d0 tx=0x7f41640099a0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.231+0000 7f4181d99740 1 -- 172.21.15.116:0/2250669241 shutdown_connections 2024-03-19T01:12:01.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.231+0000 7f4181d99740 1 --2- 172.21.15.116:0/2250669241 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4756b50 0x5559a4752240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.231+0000 7f4181d99740 1 --2- 172.21.15.116:0/2250669241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 0x5559a4762950 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.231+0000 7f4181d99740 1 --2- 172.21.15.116:0/2250669241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a4750ff0 0x5559a4758f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.231+0000 7f4181d99740 1 -- 172.21.15.116:0/2250669241 >> 172.21.15.116:0/2250669241 conn(0x5559a474c560 msgr2=0x5559a474c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.231+0000 7f4181d99740 1 -- 172.21.15.116:0/2250669241 shutdown_connections 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.231+0000 7f4181d99740 1 -- 172.21.15.116:0/2250669241 wait complete. 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.231+0000 7f4181d99740 1 -- start start 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.231+0000 7f4181d99740 1 Processor -- start 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4181d99740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a475e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4181d99740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 0x5559a475e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4181d99740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a4756b50 0x5559a4761a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4181d99740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5559a4769640 con 0x5559a47519e0 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4181d99740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5559a4768f70 con 0x5559a4756b50 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4181d99740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5559a4769d10 con 0x5559a4750ff0 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4181d99740 10 client.0 _ll_register_callbacks cb 0x7f416c0056b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4177fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 0x5559a475e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4177fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 0x5559a475e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59806/0 (socket says 172.21.15.116:59806) 2024-03-19T01:12:01.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4177fff640 1 -- 172.21.15.116:0/1526900899 learned_addr learned my addr 172.21.15.116:0/1526900899 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f417cf4f640 1 --2- 172.21.15.116:0/1526900899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a475e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f417d750640 1 --2- 172.21.15.116:0/1526900899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a4756b50 0x5559a4761a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4177fff640 1 -- 172.21.15.116:0/1526900899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 msgr2=0x5559a475e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4177fff640 1 --2- 172.21.15.116:0/1526900899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a475e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4177fff640 1 -- 172.21.15.116:0/1526900899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a4756b50 msgr2=0x5559a4761a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4177fff640 1 --2- 172.21.15.116:0/1526900899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a4756b50 0x5559a4761a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4177fff640 1 -- 172.21.15.116:0/1526900899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559a4761ff0 con 0x5559a47519e0 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4177fff640 1 --2- 172.21.15.116:0/1526900899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 0x5559a475e9e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4164004120 tx=0x7f4164003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96371023992f3a63 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4157fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4157fff640 1 -- 172.21.15.116:0/1526900899 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164009310 con 0x5559a47519e0 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4181d99740 1 -- 172.21.15.116:0/1526900899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5559a46bd670 con 0x5559a47519e0 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4157fff640 1 -- 172.21.15.116:0/1526900899 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4164005040 con 0x5559a47519e0 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4157fff640 1 -- 172.21.15.116:0/1526900899 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f41640054d0 con 0x5559a47519e0 2024-03-19T01:12:01.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4181d99740 10 client.15057 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.232+0000 7f4181d99740 1 -- 172.21.15.116:0/1526900899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5559a47ce520 con 0x5559a47519e0 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.233+0000 7f4157fff640 1 -- 172.21.15.116:0/1526900899 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5559a46bd670 con 0x5559a47519e0 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.233+0000 7f4157fff640 1 -- 172.21.15.116:0/1526900899 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f416402c020 con 0x5559a47519e0 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.233+0000 7f4181d99740 10 client.15057 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.233+0000 7f4181d99740 10 client.15057 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.233+0000 7f4181d99740 20 client.15057 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.233+0000 7f4181d99740 1 --2- 172.21.15.116:0/1526900899 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x5559a47d2b30 0x5559a47d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.233+0000 7f4181d99740 4 client.15057 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.233+0000 7f4181d99740 1 -- 172.21.15.116:0/1526900899 --> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] -- command(tid 0: {"prefix": "status"}) -- 0x5559a463ba00 con 0x5559a47d2b30 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.233+0000 7f417cf4f640 1 --2- 172.21.15.116:0/1526900899 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x5559a47d2b30 0x5559a47d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.233+0000 7f4157fff640 10 client.15057 ms_handle_connect on v2:172.21.15.162:6836/585664575 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.233+0000 7f417cf4f640 1 --2- 172.21.15.116:0/1526900899 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x5559a47d2b30 0x5559a47d1840 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3fdd7c8c4cc2aaec server_cookie=eae6a33eec38304c in_seq=0 out_seq=0 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4157fff640 1 -- 172.21.15.116:0/1526900899 <== mds.4 v2:172.21.15.162:6836/585664575 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5559a463ba00 con 0x5559a47d2b30 2024-03-19T01:12:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4157fff640 10 client.15057 handle_command_reply: tid=0 2024-03-19T01:12:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4181d99740 1 client.15057 shutdown 2024-03-19T01:12:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4181d99740 1 -- 172.21.15.116:0/1526900899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 msgr2=0x5559a475e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4181d99740 1 --2- 172.21.15.116:0/1526900899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 0x5559a475e9e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f4164004120 tx=0x7f4164003d00 comp rx=0 tx=0).stop 2024-03-19T01:12:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4181d99740 1 -- 172.21.15.116:0/1526900899 shutdown_connections 2024-03-19T01:12:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4181d99740 1 --2- 172.21.15.116:0/1526900899 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x5559a47d2b30 0x5559a47d1840 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4181d99740 1 --2- 172.21.15.116:0/1526900899 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5559a4756b50 0x5559a4761a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4181d99740 1 --2- 172.21.15.116:0/1526900899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5559a47519e0 0x5559a475e9e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4157fff640 0 client.15057 ms_handle_reset on v2:172.21.15.162:6836/585664575 2024-03-19T01:12:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4181d99740 1 --2- 172.21.15.116:0/1526900899 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5559a4750ff0 0x5559a475e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4181d99740 1 -- 172.21.15.116:0/1526900899 >> 172.21.15.116:0/1526900899 conn(0x5559a474c560 msgr2=0x5559a47520d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4181d99740 1 -- 172.21.15.116:0/1526900899 shutdown_connections 2024-03-19T01:12:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4181d99740 1 -- 172.21.15.116:0/1526900899 wait complete. 2024-03-19T01:12:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.234+0000 7f4181d99740 20 client.15057 trim_cache size 0 max 16384 2024-03-19T01:12:01.243 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:12:01.243 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "35912c14-e58c-11ee-95c9-87774f69a715", 2024-03-19T01:12:01.243 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 4, 2024-03-19T01:12:01.243 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24445, 2024-03-19T01:12:01.243 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T01:12:01.243 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T01:12:01.244 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T01:12:01.244 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 83.255512737000004, 2024-03-19T01:12:01.244 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 24, 2024-03-19T01:12:01.244 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 87, 2024-03-19T01:12:01.244 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 85, 2024-03-19T01:12:01.244 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 108.27545804 2024-03-19T01:12:01.244 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:12:01.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.235+0000 7f417f1b3640 1 -- 172.21.15.116:0/2349323281 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f41680a28b0 msgr2=0x7f41680a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.235+0000 7f417f1b3640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f41680a28b0 0x7f41680a4d70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4170005cc0 tx=0x7f4170005c50 comp rx=0 tx=0).stop 2024-03-19T01:12:01.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.235+0000 7f417f1b3640 1 -- 172.21.15.116:0/2349323281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178100990 msgr2=0x7f41781003f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.235+0000 7f417f1b3640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178100990 0x7f41781003f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f416c00e9f0 tx=0x7f416c00edf0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.235+0000 7f417f1b3640 1 -- 172.21.15.116:0/2349323281 shutdown_connections 2024-03-19T01:12:01.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.235+0000 7f417f1b3640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f41680a28b0 0x7f41680a4d70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.235+0000 7f417f1b3640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4178101dd0 0x7f41781a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.235+0000 7f417f1b3640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4178101350 0x7f417819ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.235+0000 7f417f1b3640 1 --2- 172.21.15.116:0/2349323281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4178100990 0x7f41781003f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.235+0000 7f417f1b3640 1 -- 172.21.15.116:0/2349323281 >> 172.21.15.116:0/2349323281 conn(0x7f41780fb420 msgr2=0x7f4178104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.235+0000 7f417f1b3640 1 -- 172.21.15.116:0/2349323281 shutdown_connections 2024-03-19T01:12:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.235+0000 7f417f1b3640 1 -- 172.21.15.116:0/2349323281 wait complete. 2024-03-19T01:12:01.247 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-19T01:12:01.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.352+0000 7f1635e44640 1 -- 172.21.15.116:0/4047639246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1630102a40 msgr2=0x7f1630102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.352+0000 7f1635e44640 1 --2- 172.21.15.116:0/4047639246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1630102a40 0x7f1630102ec0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1624009900 tx=0x7f162402bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:01.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.352+0000 7f1635e44640 1 -- 172.21.15.116:0/4047639246 shutdown_connections 2024-03-19T01:12:01.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.352+0000 7f1635e44640 1 --2- 172.21.15.116:0/4047639246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1630103400 0x7f1630109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.352+0000 7f1635e44640 1 --2- 172.21.15.116:0/4047639246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1630102a40 0x7f1630102ec0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.352+0000 7f1635e44640 1 --2- 172.21.15.116:0/4047639246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630101850 0x7f1630101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.352+0000 7f1635e44640 1 -- 172.21.15.116:0/4047639246 >> 172.21.15.116:0/4047639246 conn(0x7f16300fcdc0 msgr2=0x7f16300ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.352+0000 7f1635e44640 1 -- 172.21.15.116:0/4047639246 shutdown_connections 2024-03-19T01:12:01.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.352+0000 7f1635e44640 1 -- 172.21.15.116:0/4047639246 wait complete. 2024-03-19T01:12:01.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f1635e44640 1 Processor -- start 2024-03-19T01:12:01.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f1635e44640 1 -- start start 2024-03-19T01:12:01.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f1635e44640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1630101850 0x7f1630199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f1635e44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630102a40 0x7f163019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f1635e44640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1630103400 0x7f16301a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f1635e44640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f163010d520 con 0x7f1630102a40 2024-03-19T01:12:01.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f1635e44640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f163010d3a0 con 0x7f1630101850 2024-03-19T01:12:01.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f1635e44640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f163010d6a0 con 0x7f1630103400 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f162effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630102a40 0x7f163019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f162effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630102a40 0x7f163019a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59820/0 (socket says 172.21.15.116:59820) 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f162effd640 1 -- 172.21.15.116:0/722145903 learned_addr learned my addr 172.21.15.116:0/722145903 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f162f7fe640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1630101850 0x7f1630199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f162ffff640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1630103400 0x7f16301a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f162effd640 1 -- 172.21.15.116:0/722145903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1630103400 msgr2=0x7f16301a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f162effd640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1630103400 0x7f16301a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f162effd640 1 -- 172.21.15.116:0/722145903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1630101850 msgr2=0x7f1630199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f162effd640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1630101850 0x7f1630199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.353+0000 7f162effd640 1 -- 172.21.15.116:0/722145903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16301a1990 con 0x7f1630102a40 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.354+0000 7f162effd640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630102a40 0x7f163019a2b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1624000c00 tx=0x7f162402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.354+0000 7f162cff9640 1 -- 172.21.15.116:0/722145903 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1624043020 con 0x7f1630102a40 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.354+0000 7f1635e44640 1 -- 172.21.15.116:0/722145903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16301a1c20 con 0x7f1630102a40 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.354+0000 7f162cff9640 1 -- 172.21.15.116:0/722145903 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16240054e0 con 0x7f1630102a40 2024-03-19T01:12:01.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.354+0000 7f162cff9640 1 -- 172.21.15.116:0/722145903 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1624003c40 con 0x7f1630102a40 2024-03-19T01:12:01.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.354+0000 7f1635e44640 1 -- 172.21.15.116:0/722145903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16301a21c0 con 0x7f1630102a40 2024-03-19T01:12:01.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.356+0000 7f162cff9640 1 -- 172.21.15.116:0/722145903 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f1624046020 con 0x7f1630102a40 2024-03-19T01:12:01.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.356+0000 7f1638a2b740 1 Processor -- start 2024-03-19T01:12:01.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.356+0000 7f1638a2b740 1 -- start start 2024-03-19T01:12:01.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.356+0000 7f162cff9640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f16180a2800 0x7f16180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.356+0000 7f162ffff640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f16180a2800 0x7f16180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.356+0000 7f1638a2b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee591cff0 0x563ee591d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f1638a2b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee59224e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f162cff9640 1 -- 172.21.15.116:0/722145903 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f16240f0af0 con 0x7f1630102a40 2024-03-19T01:12:01.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f1638a2b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee5922b50 0x563ee5924fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f1638a2b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563ee5881de0 con 0x563ee591d9e0 2024-03-19T01:12:01.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f1638a2b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563ee582ac80 con 0x563ee5922b50 2024-03-19T01:12:01.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f1638a2b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563ee587de80 con 0x563ee591cff0 2024-03-19T01:12:01.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f162effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee59224e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f162effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee59224e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59832/0 (socket says 172.21.15.116:59832) 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f162effd640 1 -- 172.21.15.116:0/1050970841 learned_addr learned my addr 172.21.15.116:0/1050970841 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f162f7fe640 1 --2- 172.21.15.116:0/1050970841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee591cff0 0x563ee591d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f162ffff640 1 --2- 172.21.15.116:0/1050970841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee5922b50 0x563ee5924fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f162effd640 1 -- 172.21.15.116:0/1050970841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee591cff0 msgr2=0x563ee591d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f162effd640 1 --2- 172.21.15.116:0/1050970841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee591cff0 0x563ee591d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f162effd640 1 -- 172.21.15.116:0/1050970841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee5922b50 msgr2=0x563ee5924fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f162effd640 1 --2- 172.21.15.116:0/1050970841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee5922b50 0x563ee5924fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f162effd640 1 -- 172.21.15.116:0/1050970841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ee583cfb0 con 0x563ee591d9e0 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f162ffff640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f16180a2800 0x7f16180a4cc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1628005fd0 tx=0x7f1628011bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f162effd640 1 --2- 172.21.15.116:0/1050970841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee59224e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f16240bd170 tx=0x7f1624005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=255917eee300a00c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f1615ffb640 1 -- 172.21.15.116:0/1050970841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1624005690 con 0x563ee591d9e0 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f1615ffb640 1 -- 172.21.15.116:0/1050970841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1624004c60 con 0x563ee591d9e0 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.357+0000 7f1615ffb640 1 -- 172.21.15.116:0/1050970841 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16240bb510 con 0x563ee591d9e0 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.358+0000 7f1638a2b740 1 -- 172.21.15.116:0/1050970841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 msgr2=0x563ee59224e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.358+0000 7f1638a2b740 1 --2- 172.21.15.116:0/1050970841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee59224e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f16240bd170 tx=0x7f1624005e50 comp rx=0 tx=0).stop 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.358+0000 7f1638a2b740 1 -- 172.21.15.116:0/1050970841 shutdown_connections 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.358+0000 7f1638a2b740 1 --2- 172.21.15.116:0/1050970841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee5922b50 0x563ee5924fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.358+0000 7f1638a2b740 1 --2- 172.21.15.116:0/1050970841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee59224e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.358+0000 7f1638a2b740 1 --2- 172.21.15.116:0/1050970841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee591cff0 0x563ee591d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.358+0000 7f1638a2b740 1 -- 172.21.15.116:0/1050970841 >> 172.21.15.116:0/1050970841 conn(0x563ee5918560 msgr2=0x563ee5918960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.358+0000 7f1638a2b740 1 -- 172.21.15.116:0/1050970841 shutdown_connections 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.358+0000 7f1638a2b740 1 -- 172.21.15.116:0/1050970841 wait complete. 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.358+0000 7f1638a2b740 1 -- start start 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f1638a2b740 1 Processor -- start 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f1638a2b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee59271d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f1638a2b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee5927710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f1638a2b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee5922b50 0x563ee592e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f1638a2b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563ee59258c0 con 0x563ee591d9e0 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f1638a2b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563ee583dfc0 con 0x563ee591cff0 2024-03-19T01:12:01.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f1638a2b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563ee5925a40 con 0x563ee5922b50 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f1638a2b740 10 client.0 _ll_register_callbacks cb 0x563ee5879a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee5927710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee59271d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee59271d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:54606/0 (socket says 172.21.15.116:54606) 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee5927710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59842/0 (socket says 172.21.15.116:59842) 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee5922b50 0x563ee592e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162f7fe640 1 -- 172.21.15.116:0/377494810 learned_addr learned my addr 172.21.15.116:0/377494810 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162effd640 1 -- 172.21.15.116:0/377494810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee5922b50 msgr2=0x563ee592e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162effd640 1 --2- 172.21.15.116:0/377494810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee5922b50 0x563ee592e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162effd640 1 -- 172.21.15.116:0/377494810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 msgr2=0x563ee59271d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162effd640 1 --2- 172.21.15.116:0/377494810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee59271d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162effd640 1 -- 172.21.15.116:0/377494810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ee55962d0 con 0x563ee591d9e0 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162f7fe640 1 --2- 172.21.15.116:0/377494810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee59271d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:01.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162ffff640 1 --2- 172.21.15.116:0/377494810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee5922b50 0x563ee592e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f162effd640 1 --2- 172.21.15.116:0/377494810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee5927710 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x563ee591dde0 tx=0x7f16240b9810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c72354b2a82158 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f160effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f160effd640 1 -- 172.21.15.116:0/377494810 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1624005690 con 0x563ee591d9e0 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f160effd640 1 -- 172.21.15.116:0/377494810 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f16240bb510 con 0x563ee591d9e0 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.359+0000 7f1638a2b740 1 -- 172.21.15.116:0/377494810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1624004c60 con 0x563ee591d9e0 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.360+0000 7f160effd640 1 -- 172.21.15.116:0/377494810 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f16240bd470 con 0x563ee591d9e0 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.360+0000 7f1638a2b740 10 client.15069 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.360+0000 7f1638a2b740 1 -- 172.21.15.116:0/377494810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563ee56606b0 con 0x563ee591d9e0 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.360+0000 7f160effd640 1 -- 172.21.15.116:0/377494810 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1624004c60 con 0x563ee591d9e0 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.360+0000 7f160effd640 1 -- 172.21.15.116:0/377494810 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f16240c1120 con 0x563ee591d9e0 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.360+0000 7f1638a2b740 10 client.15069 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.360+0000 7f1638a2b740 10 client.15069 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.360+0000 7f1638a2b740 20 client.15069 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:01.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.360+0000 7f1638a2b740 1 --2- 172.21.15.116:0/377494810 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x563ee5933d80 0x563ee59351f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.360+0000 7f1638a2b740 4 client.15069 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:12:01.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.360+0000 7f1638a2b740 1 -- 172.21.15.116:0/377494810 --> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ee588f320 con 0x563ee5933d80 2024-03-19T01:12:01.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.360+0000 7f162f7fe640 1 --2- 172.21.15.116:0/377494810 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x563ee5933d80 0x563ee59351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.361+0000 7f160effd640 10 client.15069 ms_handle_connect on v2:172.21.15.162:6838/441486392 2024-03-19T01:12:01.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.361+0000 7f162f7fe640 1 --2- 172.21.15.116:0/377494810 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x563ee5933d80 0x563ee59351f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c99710d6134bbf21 server_cookie=1786fc746034f0f6 in_seq=0 out_seq=0 2024-03-19T01:12:01.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.361+0000 7f160effd640 1 -- 172.21.15.116:0/377494810 <== mds.3 v2:172.21.15.162:6838/441486392 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ee588f320 con 0x563ee5933d80 2024-03-19T01:12:01.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.361+0000 7f160effd640 10 client.15069 handle_command_reply: tid=0 2024-03-19T01:12:01.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.361+0000 7f1638a2b740 1 client.15069 shutdown 2024-03-19T01:12:01.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.361+0000 7f1638a2b740 1 -- 172.21.15.116:0/377494810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 msgr2=0x563ee5927710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.361+0000 7f1638a2b740 1 --2- 172.21.15.116:0/377494810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee5927710 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x563ee591dde0 tx=0x7f16240b9810 comp rx=0 tx=0).stop 2024-03-19T01:12:01.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.361+0000 7f1638a2b740 1 -- 172.21.15.116:0/377494810 shutdown_connections 2024-03-19T01:12:01.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.361+0000 7f1638a2b740 1 --2- 172.21.15.116:0/377494810 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x563ee5933d80 0x563ee59351f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.361+0000 7f1638a2b740 1 --2- 172.21.15.116:0/377494810 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee5922b50 0x563ee592e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.361+0000 7f1638a2b740 1 --2- 172.21.15.116:0/377494810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee5927710 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.361+0000 7f160effd640 0 client.15069 ms_handle_reset on v2:172.21.15.162:6838/441486392 2024-03-19T01:12:01.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.361+0000 7f1638a2b740 1 --2- 172.21.15.116:0/377494810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee59271d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.362+0000 7f1638a2b740 1 -- 172.21.15.116:0/377494810 >> 172.21.15.116:0/377494810 conn(0x563ee5918560 msgr2=0x563ee59177e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.362+0000 7f1638a2b740 1 -- 172.21.15.116:0/377494810 shutdown_connections 2024-03-19T01:12:01.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.362+0000 7f1638a2b740 1 -- 172.21.15.116:0/377494810 wait complete. 2024-03-19T01:12:01.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.362+0000 7f1638a2b740 20 client.15069 trim_cache size 0 max 16384 2024-03-19T01:12:01.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.379+0000 7f1638a2b740 1 Processor -- start 2024-03-19T01:12:01.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.379+0000 7f1638a2b740 1 -- start start 2024-03-19T01:12:01.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.379+0000 7f1638a2b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee5924f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.379+0000 7f1638a2b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee591d9e0 0x563ee592e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.379+0000 7f1638a2b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee5922b50 0x563ee591e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.379+0000 7f1638a2b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563ee5933530 con 0x563ee5922b50 2024-03-19T01:12:01.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.379+0000 7f1638a2b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563ee5933c00 con 0x563ee591cff0 2024-03-19T01:12:01.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.379+0000 7f1638a2b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563ee59342d0 con 0x563ee591d9e0 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f162ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee5922b50 0x563ee591e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f162f7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee5924f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f162ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee5922b50 0x563ee591e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59844/0 (socket says 172.21.15.116:59844) 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f162ffff640 1 -- 172.21.15.116:0/3722575595 learned_addr learned my addr 172.21.15.116:0/3722575595 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f162effd640 1 --2- 172.21.15.116:0/3722575595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee591d9e0 0x563ee592e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f162ffff640 1 -- 172.21.15.116:0/3722575595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee591d9e0 msgr2=0x563ee592e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f162ffff640 1 --2- 172.21.15.116:0/3722575595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee591d9e0 0x563ee592e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f162ffff640 1 -- 172.21.15.116:0/3722575595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 msgr2=0x563ee5924f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f162ffff640 1 --2- 172.21.15.116:0/3722575595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee5924f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f162ffff640 1 -- 172.21.15.116:0/3722575595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ee59194a0 con 0x563ee5922b50 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f162ffff640 1 --2- 172.21.15.116:0/3722575595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee5922b50 0x563ee591e0d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1628009d40 tx=0x7f162801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4082a192fe723643 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f16157fa640 1 -- 172.21.15.116:0/3722575595 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f162802d070 con 0x563ee5922b50 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f16157fa640 1 -- 172.21.15.116:0/3722575595 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1628028ad0 con 0x563ee5922b50 2024-03-19T01:12:01.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.380+0000 7f16157fa640 1 -- 172.21.15.116:0/3722575595 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f162801a040 con 0x563ee5922b50 2024-03-19T01:12:01.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.381+0000 7f1638a2b740 1 -- 172.21.15.116:0/3722575595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee5922b50 msgr2=0x563ee591e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.381+0000 7f1638a2b740 1 --2- 172.21.15.116:0/3722575595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee5922b50 0x563ee591e0d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1628009d40 tx=0x7f162801f000 comp rx=0 tx=0).stop 2024-03-19T01:12:01.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.381+0000 7f1638a2b740 1 -- 172.21.15.116:0/3722575595 shutdown_connections 2024-03-19T01:12:01.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.381+0000 7f1638a2b740 1 --2- 172.21.15.116:0/3722575595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee5922b50 0x563ee591e0d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.381+0000 7f1638a2b740 1 --2- 172.21.15.116:0/3722575595 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee591d9e0 0x563ee592e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.381+0000 7f1638a2b740 1 --2- 172.21.15.116:0/3722575595 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee5924f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.381+0000 7f1638a2b740 1 -- 172.21.15.116:0/3722575595 >> 172.21.15.116:0/3722575595 conn(0x563ee5918560 msgr2=0x563ee5925d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.381+0000 7f1638a2b740 1 -- 172.21.15.116:0/3722575595 shutdown_connections 2024-03-19T01:12:01.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.381+0000 7f1638a2b740 1 -- 172.21.15.116:0/3722575595 wait complete. 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.381+0000 7f1638a2b740 1 -- start start 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f1638a2b740 1 Processor -- start 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f1638a2b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee592c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f1638a2b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee5995ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f1638a2b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee5922b50 0x563ee5998020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f1638a2b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x563ee5935070 con 0x563ee591d9e0 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f1638a2b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563ee59349a0 con 0x563ee591cff0 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f1638a2b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x563ee5935740 con 0x563ee5922b50 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f1638a2b740 10 client.0 _ll_register_callbacks cb 0x7f1628011e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f162effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee5995ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f162effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee5995ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59852/0 (socket says 172.21.15.116:59852) 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f162effd640 1 -- 172.21.15.116:0/3266886762 learned_addr learned my addr 172.21.15.116:0/3266886762 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f162f7fe640 1 --2- 172.21.15.116:0/3266886762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee592c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f162ffff640 1 --2- 172.21.15.116:0/3266886762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee5922b50 0x563ee5998020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f162effd640 1 -- 172.21.15.116:0/3266886762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee5922b50 msgr2=0x563ee5998020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f162effd640 1 --2- 172.21.15.116:0/3266886762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee5922b50 0x563ee5998020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f162effd640 1 -- 172.21.15.116:0/3266886762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 msgr2=0x563ee592c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f162effd640 1 --2- 172.21.15.116:0/3266886762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee592c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f162effd640 1 -- 172.21.15.116:0/3266886762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ee5998560 con 0x563ee591d9e0 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f162effd640 1 --2- 172.21.15.116:0/3266886762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee5995ad0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f16240c5ac0 tx=0x7f1624009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e41b94ae0d1515e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f160effd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f160effd640 1 -- 172.21.15.116:0/3266886762 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16240bd240 con 0x563ee591d9e0 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f160effd640 1 -- 172.21.15.116:0/3266886762 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f16240c5780 con 0x563ee591d9e0 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.382+0000 7f1638a2b740 1 -- 172.21.15.116:0/3266886762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1628028ad0 con 0x563ee591d9e0 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f160effd640 1 -- 172.21.15.116:0/3266886762 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f16240ca050 con 0x563ee591d9e0 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f1638a2b740 10 client.15075 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f1638a2b740 1 -- 172.21.15.116:0/3266886762 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563ee5998560 con 0x563ee591d9e0 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f160effd640 1 -- 172.21.15.116:0/3266886762 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1628028ad0 con 0x563ee591d9e0 2024-03-19T01:12:01.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f160effd640 1 -- 172.21.15.116:0/3266886762 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f16240c1120 con 0x563ee591d9e0 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f1638a2b740 10 client.15075 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f1638a2b740 10 client.15075 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f1638a2b740 20 client.15075 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f1638a2b740 1 --2- 172.21.15.116:0/3266886762 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x563ee599e180 0x563ee599ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f1638a2b740 4 client.15075 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f1638a2b740 1 -- 172.21.15.116:0/3266886762 --> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] -- command(tid 0: {"prefix": "status"}) -- 0x563ee5889670 con 0x563ee599e180 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f162f7fe640 1 --2- 172.21.15.116:0/3266886762 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x563ee599e180 0x563ee599ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f160effd640 10 client.15075 ms_handle_connect on v2:172.21.15.162:6838/441486392 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.383+0000 7f162f7fe640 1 --2- 172.21.15.116:0/3266886762 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x563ee599e180 0x563ee599ce90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=19731cdd430c0c49 server_cookie=4c749b44c52a4452 in_seq=0 out_seq=0 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f160effd640 1 -- 172.21.15.116:0/3266886762 <== mds.3 v2:172.21.15.162:6838/441486392 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x563ee5889670 con 0x563ee599e180 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f160effd640 10 client.15075 handle_command_reply: tid=0 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1638a2b740 1 client.15075 shutdown 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1638a2b740 1 -- 172.21.15.116:0/3266886762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 msgr2=0x563ee5995ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1638a2b740 1 --2- 172.21.15.116:0/3266886762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee5995ad0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f16240c5ac0 tx=0x7f1624009880 comp rx=0 tx=0).stop 2024-03-19T01:12:01.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1638a2b740 1 -- 172.21.15.116:0/3266886762 shutdown_connections 2024-03-19T01:12:01.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1638a2b740 1 --2- 172.21.15.116:0/3266886762 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x563ee599e180 0x563ee599ce90 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1638a2b740 1 --2- 172.21.15.116:0/3266886762 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x563ee5922b50 0x563ee5998020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1638a2b740 1 --2- 172.21.15.116:0/3266886762 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x563ee591d9e0 0x563ee5995ad0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f160effd640 0 client.15075 ms_handle_reset on v2:172.21.15.162:6838/441486392 2024-03-19T01:12:01.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1638a2b740 1 --2- 172.21.15.116:0/3266886762 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563ee591cff0 0x563ee592c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1638a2b740 1 -- 172.21.15.116:0/3266886762 >> 172.21.15.116:0/3266886762 conn(0x563ee5918560 msgr2=0x563ee591e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1638a2b740 1 -- 172.21.15.116:0/3266886762 shutdown_connections 2024-03-19T01:12:01.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1638a2b740 1 -- 172.21.15.116:0/3266886762 wait complete. 2024-03-19T01:12:01.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1638a2b740 20 client.15075 trim_cache size 0 max 16384 2024-03-19T01:12:01.395 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:12:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "35912c14-e58c-11ee-95c9-87774f69a715", 2024-03-19T01:12:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 3, 2024-03-19T01:12:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24460, 2024-03-19T01:12:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T01:12:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T01:12:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T01:12:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 84.413573389999996, 2024-03-19T01:12:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 24, 2024-03-19T01:12:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 87, 2024-03-19T01:12:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 85, 2024-03-19T01:12:01.395 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 102.67870592600001 2024-03-19T01:12:01.395 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:12:01.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1635e44640 1 -- 172.21.15.116:0/722145903 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f16180a2800 msgr2=0x7f16180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1635e44640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f16180a2800 0x7f16180a4cc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1628005fd0 tx=0x7f1628011bc0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.384+0000 7f1635e44640 1 -- 172.21.15.116:0/722145903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630102a40 msgr2=0x7f163019a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.385+0000 7f1635e44640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630102a40 0x7f163019a2b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1624000c00 tx=0x7f162402c650 comp rx=0 tx=0).stop 2024-03-19T01:12:01.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.385+0000 7f1635e44640 1 -- 172.21.15.116:0/722145903 shutdown_connections 2024-03-19T01:12:01.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.385+0000 7f1635e44640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1630103400 0x7f16301a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.385+0000 7f1635e44640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1630102a40 0x7f163019a2b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.385+0000 7f1635e44640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f16180a2800 0x7f16180a4cc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.385+0000 7f1635e44640 1 --2- 172.21.15.116:0/722145903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1630101850 0x7f1630199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.385+0000 7f1635e44640 1 -- 172.21.15.116:0/722145903 >> 172.21.15.116:0/722145903 conn(0x7f16300fcdc0 msgr2=0x7f1630073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.385+0000 7f1635e44640 1 -- 172.21.15.116:0/722145903 shutdown_connections 2024-03-19T01:12:01.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.385+0000 7f1635e44640 1 -- 172.21.15.116:0/722145903 wait complete. 2024-03-19T01:12:01.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:01 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4144606068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:12:01.397 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:01 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2972121464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:12:01.398 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:01 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1500918162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:12:01.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-19T01:12:01.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:01 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/4144606068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:12:01.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:01 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2972121464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:12:01.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:01 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1500918162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:12:01.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.500+0000 7f06d308a640 1 -- 172.21.15.116:0/2804333036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06cc104c30 msgr2=0x7f06cc107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.500+0000 7f06d308a640 1 --2- 172.21.15.116:0/2804333036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06cc104c30 0x7f06cc107020 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f06b8009900 tx=0x7f06b802bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:01.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.500+0000 7f06d308a640 1 -- 172.21.15.116:0/2804333036 shutdown_connections 2024-03-19T01:12:01.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.500+0000 7f06d308a640 1 --2- 172.21.15.116:0/2804333036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06cc107560 0x7f06cc109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.500+0000 7f06d308a640 1 --2- 172.21.15.116:0/2804333036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06cc104c30 0x7f06cc107020 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.500+0000 7f06d308a640 1 --2- 172.21.15.116:0/2804333036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06cc102300 0x7f06cc1046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.500+0000 7f06d308a640 1 -- 172.21.15.116:0/2804333036 >> 172.21.15.116:0/2804333036 conn(0x7f06cc0fbcb0 msgr2=0x7f06cc0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.500+0000 7f06d308a640 1 -- 172.21.15.116:0/2804333036 shutdown_connections 2024-03-19T01:12:01.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.500+0000 7f06d308a640 1 -- 172.21.15.116:0/2804333036 wait complete. 2024-03-19T01:12:01.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d308a640 1 Processor -- start 2024-03-19T01:12:01.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d308a640 1 -- start start 2024-03-19T01:12:01.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d308a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06cc102300 0x7f06cc197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d308a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06cc104c30 0x7f06cc198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d308a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06cc107560 0x7f06cc19f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d308a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc10d470 con 0x7f06cc107560 2024-03-19T01:12:01.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d308a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc10d2f0 con 0x7f06cc102300 2024-03-19T01:12:01.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d308a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f06cc10d5f0 con 0x7f06cc104c30 2024-03-19T01:12:01.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d1627640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06cc107560 0x7f06cc19f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d0e26640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06cc102300 0x7f06cc197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d1627640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06cc107560 0x7f06cc19f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59866/0 (socket says 172.21.15.116:59866) 2024-03-19T01:12:01.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d1627640 1 -- 172.21.15.116:0/1878781650 learned_addr learned my addr 172.21.15.116:0/1878781650 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06cbfff640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06cc104c30 0x7f06cc198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d1627640 1 -- 172.21.15.116:0/1878781650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06cc104c30 msgr2=0x7f06cc198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d1627640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06cc104c30 0x7f06cc198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d1627640 1 -- 172.21.15.116:0/1878781650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06cc102300 msgr2=0x7f06cc197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d1627640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06cc102300 0x7f06cc197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d1627640 1 -- 172.21.15.116:0/1878781650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06cc19f7e0 con 0x7f06cc107560 2024-03-19T01:12:01.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d0e26640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06cc102300 0x7f06cc197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:01.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.501+0000 7f06d1627640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06cc107560 0x7f06cc19f170 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f06c400eb10 tx=0x7f06c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.502+0000 7f06c9ffb640 1 -- 172.21.15.116:0/1878781650 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4004b20 con 0x7f06cc107560 2024-03-19T01:12:01.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.502+0000 7f06c9ffb640 1 -- 172.21.15.116:0/1878781650 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06c4004cc0 con 0x7f06cc107560 2024-03-19T01:12:01.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.502+0000 7f06d308a640 1 -- 172.21.15.116:0/1878781650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06cc19fad0 con 0x7f06cc107560 2024-03-19T01:12:01.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.502+0000 7f06c9ffb640 1 -- 172.21.15.116:0/1878781650 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c4010760 con 0x7f06cc107560 2024-03-19T01:12:01.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.502+0000 7f06d308a640 1 -- 172.21.15.116:0/1878781650 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06cc19ffa0 con 0x7f06cc107560 2024-03-19T01:12:01.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.504+0000 7f06c9ffb640 1 -- 172.21.15.116:0/1878781650 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f06c4005020 con 0x7f06cc107560 2024-03-19T01:12:01.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.504+0000 7f06d5c70740 1 Processor -- start 2024-03-19T01:12:01.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.504+0000 7f06d5c70740 1 -- start start 2024-03-19T01:12:01.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.504+0000 7f06d5c70740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d0ff0 0x5649e65d1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.504+0000 7f06d5c70740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d19e0 0x5649e65d64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.504+0000 7f06d5c70740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d6b50 0x5649e65d8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.504+0000 7f06d5c70740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649e6535de0 con 0x5649e65d6b50 2024-03-19T01:12:01.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.504+0000 7f06d5c70740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5649e64dec80 con 0x5649e65d0ff0 2024-03-19T01:12:01.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.504+0000 7f06d5c70740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5649e6531e80 con 0x5649e65d19e0 2024-03-19T01:12:01.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.504+0000 7f06c9ffb640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f06bc0a2800 0x7f06bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d6b50 0x5649e65d8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06cbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d0ff0 0x5649e65d1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06cbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d0ff0 0x5649e65d1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:54656/0 (socket says 172.21.15.116:54656) 2024-03-19T01:12:01.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06cbfff640 1 -- 172.21.15.116:0/3358176592 learned_addr learned my addr 172.21.15.116:0/3358176592 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06d0e26640 1 --2- 172.21.15.116:0/3358176592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d19e0 0x5649e65d64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06d1627640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f06bc0a2800 0x7f06bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06cbfff640 1 -- 172.21.15.116:0/3358176592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d19e0 msgr2=0x5649e65d64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06cbfff640 1 --2- 172.21.15.116:0/3358176592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d19e0 0x5649e65d64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06c9ffb640 1 -- 172.21.15.116:0/1878781650 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f06c40ceff0 con 0x7f06cc107560 2024-03-19T01:12:01.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06cbfff640 1 -- 172.21.15.116:0/3358176592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d6b50 msgr2=0x5649e65d8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06cbfff640 1 --2- 172.21.15.116:0/3358176592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d6b50 0x5649e65d8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06cbfff640 1 -- 172.21.15.116:0/3358176592 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649e64f0fb0 con 0x5649e65d0ff0 2024-03-19T01:12:01.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06cbfff640 1 --2- 172.21.15.116:0/3358176592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d0ff0 0x5649e65d1410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f06b802cfd0 tx=0x7f06b8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=613a1b0ad6a3dc82 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06d1627640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f06bc0a2800 0x7f06bc0a4cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f06c40a3be0 tx=0x7f06c40c63c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06b6ffd640 1 -- 172.21.15.116:0/3358176592 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b8048070 con 0x5649e65d0ff0 2024-03-19T01:12:01.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06b6ffd640 1 -- 172.21.15.116:0/3358176592 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06b80035f0 con 0x5649e65d0ff0 2024-03-19T01:12:01.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.505+0000 7f06b6ffd640 1 -- 172.21.15.116:0/3358176592 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b803f680 con 0x5649e65d0ff0 2024-03-19T01:12:01.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.506+0000 7f06d5c70740 1 -- 172.21.15.116:0/3358176592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d0ff0 msgr2=0x5649e65d1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.506+0000 7f06d5c70740 1 --2- 172.21.15.116:0/3358176592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d0ff0 0x5649e65d1410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f06b802cfd0 tx=0x7f06b8005fd0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.506+0000 7f06d5c70740 1 -- 172.21.15.116:0/3358176592 shutdown_connections 2024-03-19T01:12:01.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.506+0000 7f06d5c70740 1 --2- 172.21.15.116:0/3358176592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d6b50 0x5649e65d8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.506+0000 7f06d5c70740 1 --2- 172.21.15.116:0/3358176592 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d19e0 0x5649e65d64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.506+0000 7f06d5c70740 1 --2- 172.21.15.116:0/3358176592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d0ff0 0x5649e65d1410 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.506+0000 7f06d5c70740 1 -- 172.21.15.116:0/3358176592 >> 172.21.15.116:0/3358176592 conn(0x5649e65cc560 msgr2=0x5649e65cc960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.506+0000 7f06d5c70740 1 -- 172.21.15.116:0/3358176592 shutdown_connections 2024-03-19T01:12:01.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.506+0000 7f06d5c70740 1 -- 172.21.15.116:0/3358176592 wait complete. 2024-03-19T01:12:01.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.506+0000 7f06d5c70740 1 -- start start 2024-03-19T01:12:01.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.506+0000 7f06d5c70740 1 Processor -- start 2024-03-19T01:12:01.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06d5c70740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d0ff0 0x5649e65db500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06d5c70740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d19e0 0x5649e65dba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06d5c70740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d6b50 0x5649e65e2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06d5c70740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649e65d9aa0 con 0x5649e65d0ff0 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06d5c70740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5649e64f1fc0 con 0x5649e65d19e0 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06d5c70740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5649e65d9c20 con 0x5649e65d6b50 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06d5c70740 10 client.0 _ll_register_callbacks cb 0x5649e652da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d0ff0 0x5649e65db500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d0ff0 0x5649e65db500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59888/0 (socket says 172.21.15.116:59888) 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06cbfff640 1 -- 172.21.15.116:0/3587467580 learned_addr learned my addr 172.21.15.116:0/3587467580 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06d0e26640 1 --2- 172.21.15.116:0/3587467580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d19e0 0x5649e65dba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06cbfff640 1 --2- 172.21.15.116:0/3587467580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d6b50 0x5649e65e2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06cbfff640 1 -- 172.21.15.116:0/3587467580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d6b50 msgr2=0x5649e65e2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06cbfff640 1 --2- 172.21.15.116:0/3587467580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d6b50 0x5649e65e2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06cbfff640 1 -- 172.21.15.116:0/3587467580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d19e0 msgr2=0x5649e65dba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06cbfff640 1 --2- 172.21.15.116:0/3587467580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d19e0 0x5649e65dba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06cbfff640 1 -- 172.21.15.116:0/3587467580 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649e624a2d0 con 0x5649e65d0ff0 2024-03-19T01:12:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06cbfff640 1 --2- 172.21.15.116:0/3587467580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d0ff0 0x5649e65db500 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f06b802cac0 tx=0x7f06b802ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953c4f9a002d3523 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06abfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06abfff640 1 -- 172.21.15.116:0/3587467580 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b8053060 con 0x5649e65d0ff0 2024-03-19T01:12:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06abfff640 1 -- 172.21.15.116:0/3587467580 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f06b804d470 con 0x5649e65d0ff0 2024-03-19T01:12:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.507+0000 7f06d5c70740 1 -- 172.21.15.116:0/3587467580 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06b80035f0 con 0x5649e65d0ff0 2024-03-19T01:12:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06abfff640 1 -- 172.21.15.116:0/3587467580 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f06b8048040 con 0x5649e65d0ff0 2024-03-19T01:12:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06d5c70740 10 client.15087 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06d5c70740 1 -- 172.21.15.116:0/3587467580 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5649e63146b0 con 0x5649e65d0ff0 2024-03-19T01:12:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06abfff640 1 -- 172.21.15.116:0/3587467580 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06b80035f0 con 0x5649e65d0ff0 2024-03-19T01:12:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06abfff640 1 -- 172.21.15.116:0/3587467580 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f06b803f5d0 con 0x5649e65d0ff0 2024-03-19T01:12:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06d5c70740 10 client.15087 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06d5c70740 10 client.15087 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T01:12:01.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06d5c70740 20 client.15087 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06d5c70740 1 --2- 172.21.15.116:0/3587467580 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x5649e65e7e10 0x5649e65e9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06d5c70740 4 client.15087 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:12:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06d5c70740 1 -- 172.21.15.116:0/3587467580 --> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649e6543320 con 0x5649e65e7e10 2024-03-19T01:12:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06d0e26640 1 --2- 172.21.15.116:0/3587467580 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x5649e65e7e10 0x5649e65e9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06abfff640 10 client.15087 ms_handle_connect on v2:172.21.15.156:6836/4147060396 2024-03-19T01:12:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.508+0000 7f06d0e26640 1 --2- 172.21.15.116:0/3587467580 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x5649e65e7e10 0x5649e65e9280 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=398c5aee751b12c5 server_cookie=a681bc8784fc8da7 in_seq=0 out_seq=0 2024-03-19T01:12:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.509+0000 7f06abfff640 1 -- 172.21.15.116:0/3587467580 <== mds.2 v2:172.21.15.156:6836/4147060396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5649e6543320 con 0x5649e65e7e10 2024-03-19T01:12:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.509+0000 7f06abfff640 10 client.15087 handle_command_reply: tid=0 2024-03-19T01:12:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.509+0000 7f06d5c70740 1 client.15087 shutdown 2024-03-19T01:12:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.509+0000 7f06d5c70740 1 -- 172.21.15.116:0/3587467580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d0ff0 msgr2=0x5649e65db500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.509+0000 7f06d5c70740 1 --2- 172.21.15.116:0/3587467580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d0ff0 0x5649e65db500 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f06b802cac0 tx=0x7f06b802ca30 comp rx=0 tx=0).stop 2024-03-19T01:12:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.509+0000 7f06d5c70740 1 -- 172.21.15.116:0/3587467580 shutdown_connections 2024-03-19T01:12:01.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.509+0000 7f06d5c70740 1 --2- 172.21.15.116:0/3587467580 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x5649e65e7e10 0x5649e65e9280 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.509+0000 7f06d5c70740 1 --2- 172.21.15.116:0/3587467580 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d6b50 0x5649e65e2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.509+0000 7f06d5c70740 1 --2- 172.21.15.116:0/3587467580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d19e0 0x5649e65dba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.509+0000 7f06abfff640 0 client.15087 ms_handle_reset on v2:172.21.15.156:6836/4147060396 2024-03-19T01:12:01.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.509+0000 7f06d5c70740 1 --2- 172.21.15.116:0/3587467580 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d0ff0 0x5649e65db500 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.509+0000 7f06d5c70740 1 -- 172.21.15.116:0/3587467580 >> 172.21.15.116:0/3587467580 conn(0x5649e65cc560 msgr2=0x5649e65cb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.510+0000 7f06d5c70740 1 -- 172.21.15.116:0/3587467580 shutdown_connections 2024-03-19T01:12:01.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.510+0000 7f06d5c70740 1 -- 172.21.15.116:0/3587467580 wait complete. 2024-03-19T01:12:01.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.510+0000 7f06d5c70740 20 client.15087 trim_cache size 0 max 16384 2024-03-19T01:12:01.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.533+0000 7f06d5c70740 1 Processor -- start 2024-03-19T01:12:01.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.533+0000 7f06d5c70740 1 -- start start 2024-03-19T01:12:01.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.533+0000 7f06d5c70740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d0ff0 0x5649e65d2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.533+0000 7f06d5c70740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d19e0 0x5649e65e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.533+0000 7f06d5c70740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d6b50 0x5649e65d3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.533+0000 7f06d5c70740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649e65e75c0 con 0x5649e65d19e0 2024-03-19T01:12:01.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06d5c70740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5649e65e7c90 con 0x5649e65d6b50 2024-03-19T01:12:01.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06d5c70740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5649e65e8360 con 0x5649e65d0ff0 2024-03-19T01:12:01.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06d0e26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d19e0 0x5649e65e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06d0e26640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d19e0 0x5649e65e2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59890/0 (socket says 172.21.15.116:59890) 2024-03-19T01:12:01.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06d0e26640 1 -- 172.21.15.116:0/2982687436 learned_addr learned my addr 172.21.15.116:0/2982687436 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06cbfff640 1 --2- 172.21.15.116:0/2982687436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d0ff0 0x5649e65d2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06cbfff640 1 --2- 172.21.15.116:0/2982687436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d6b50 0x5649e65d3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06d0e26640 1 -- 172.21.15.116:0/2982687436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d0ff0 msgr2=0x5649e65d2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06d0e26640 1 --2- 172.21.15.116:0/2982687436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d0ff0 0x5649e65d2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06d0e26640 1 -- 172.21.15.116:0/2982687436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d6b50 msgr2=0x5649e65d3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06d0e26640 1 --2- 172.21.15.116:0/2982687436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d6b50 0x5649e65d3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06d0e26640 1 -- 172.21.15.116:0/2982687436 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649e65ccfb0 con 0x5649e65d19e0 2024-03-19T01:12:01.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06d0e26640 1 --2- 172.21.15.116:0/2982687436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d19e0 0x5649e65e2a40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5649e65d1de0 tx=0x7f06c0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc36bcf10aa20b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06b67fc640 1 -- 172.21.15.116:0/2982687436 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c00108e0 con 0x5649e65d19e0 2024-03-19T01:12:01.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06b67fc640 1 -- 172.21.15.116:0/2982687436 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06c0025020 con 0x5649e65d19e0 2024-03-19T01:12:01.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.534+0000 7f06b67fc640 1 -- 172.21.15.116:0/2982687436 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c001e650 con 0x5649e65d19e0 2024-03-19T01:12:01.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 -- 172.21.15.116:0/2982687436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d19e0 msgr2=0x5649e65e2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 --2- 172.21.15.116:0/2982687436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d19e0 0x5649e65e2a40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x5649e65d1de0 tx=0x7f06c0005f90 comp rx=0 tx=0).stop 2024-03-19T01:12:01.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 -- 172.21.15.116:0/2982687436 shutdown_connections 2024-03-19T01:12:01.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 --2- 172.21.15.116:0/2982687436 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d6b50 0x5649e65d3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 --2- 172.21.15.116:0/2982687436 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d19e0 0x5649e65e2a40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 --2- 172.21.15.116:0/2982687436 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d0ff0 0x5649e65d2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 -- 172.21.15.116:0/2982687436 >> 172.21.15.116:0/2982687436 conn(0x5649e65cc560 msgr2=0x5649e65da7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 -- 172.21.15.116:0/2982687436 shutdown_connections 2024-03-19T01:12:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 -- 172.21.15.116:0/2982687436 wait complete. 2024-03-19T01:12:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 -- start start 2024-03-19T01:12:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 Processor -- start 2024-03-19T01:12:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d0ff0 0x5649e65de5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d19e0 0x5649e65deb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d6b50 0x5649e65e1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5649e65e9100 con 0x5649e65d6b50 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5649e65e8a30 con 0x5649e65d0ff0 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5649e65e97d0 con 0x5649e65d19e0 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.535+0000 7f06d5c70740 10 client.0 _ll_register_callbacks cb 0x7f06b8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d6b50 0x5649e65e1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d6b50 0x5649e65e1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59906/0 (socket says 172.21.15.116:59906) 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06cbfff640 1 -- 172.21.15.116:0/1234044832 learned_addr learned my addr 172.21.15.116:0/1234044832 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06cbfff640 1 --2- 172.21.15.116:0/1234044832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d0ff0 0x5649e65de5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06d0e26640 1 --2- 172.21.15.116:0/1234044832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d19e0 0x5649e65deb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06cbfff640 1 -- 172.21.15.116:0/1234044832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d19e0 msgr2=0x5649e65deb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06cbfff640 1 --2- 172.21.15.116:0/1234044832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d19e0 0x5649e65deb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06cbfff640 1 -- 172.21.15.116:0/1234044832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d0ff0 msgr2=0x5649e65de5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06cbfff640 1 --2- 172.21.15.116:0/1234044832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d0ff0 0x5649e65de5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06cbfff640 1 -- 172.21.15.116:0/1234044832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649e65e2050 con 0x5649e65d6b50 2024-03-19T01:12:01.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06cbfff640 1 --2- 172.21.15.116:0/1234044832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d6b50 0x5649e65e1a60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f06b802c170 tx=0x7f06b802ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53dcdf7ac435e72d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06abfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06abfff640 1 -- 172.21.15.116:0/1234044832 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b805e040 con 0x5649e65d6b50 2024-03-19T01:12:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06abfff640 1 -- 172.21.15.116:0/1234044832 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f06b8034490 con 0x5649e65d6b50 2024-03-19T01:12:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06d5c70740 1 -- 172.21.15.116:0/1234044832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06c0025020 con 0x5649e65d6b50 2024-03-19T01:12:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06abfff640 1 -- 172.21.15.116:0/1234044832 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f06b8053030 con 0x5649e65d6b50 2024-03-19T01:12:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06d5c70740 10 client.15093 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06d5c70740 1 -- 172.21.15.116:0/1234044832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5649e664df50 con 0x5649e65d6b50 2024-03-19T01:12:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.536+0000 7f06abfff640 1 -- 172.21.15.116:0/1234044832 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06c0025020 con 0x5649e65d6b50 2024-03-19T01:12:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06abfff640 1 -- 172.21.15.116:0/1234044832 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f06b80414c0 con 0x5649e65d6b50 2024-03-19T01:12:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 10 client.15093 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 10 client.15093 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T01:12:01.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 20 client.15093 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 1 --2- 172.21.15.116:0/1234044832 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x5649e66529a0 0x5649e66516b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 4 client.15093 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T01:12:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 1 -- 172.21.15.116:0/1234044832 --> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] -- command(tid 0: {"prefix": "status"}) -- 0x5649e653d670 con 0x5649e66529a0 2024-03-19T01:12:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d0e26640 1 --2- 172.21.15.116:0/1234044832 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x5649e66529a0 0x5649e66516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d0e26640 1 --2- 172.21.15.116:0/1234044832 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x5649e66529a0 0x5649e66516b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=399c42f99d7208a9 server_cookie=642f47fa27816b46 in_seq=0 out_seq=0 2024-03-19T01:12:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06abfff640 10 client.15093 ms_handle_connect on v2:172.21.15.156:6836/4147060396 2024-03-19T01:12:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06abfff640 1 -- 172.21.15.116:0/1234044832 <== mds.2 v2:172.21.15.156:6836/4147060396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5649e653d670 con 0x5649e66529a0 2024-03-19T01:12:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06abfff640 10 client.15093 handle_command_reply: tid=0 2024-03-19T01:12:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 1 client.15093 shutdown 2024-03-19T01:12:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 1 -- 172.21.15.116:0/1234044832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d6b50 msgr2=0x5649e65e1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 1 --2- 172.21.15.116:0/1234044832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d6b50 0x5649e65e1a60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f06b802c170 tx=0x7f06b802ca30 comp rx=0 tx=0).stop 2024-03-19T01:12:01.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 1 -- 172.21.15.116:0/1234044832 shutdown_connections 2024-03-19T01:12:01.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 1 --2- 172.21.15.116:0/1234044832 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x5649e66529a0 0x5649e66516b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 1 --2- 172.21.15.116:0/1234044832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5649e65d6b50 0x5649e65e1a60 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06abfff640 0 client.15093 ms_handle_reset on v2:172.21.15.156:6836/4147060396 2024-03-19T01:12:01.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 1 --2- 172.21.15.116:0/1234044832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5649e65d19e0 0x5649e65deb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.537+0000 7f06d5c70740 1 --2- 172.21.15.116:0/1234044832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5649e65d0ff0 0x5649e65de5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d5c70740 1 -- 172.21.15.116:0/1234044832 >> 172.21.15.116:0/1234044832 conn(0x5649e65cc560 msgr2=0x5649e65d2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d5c70740 1 -- 172.21.15.116:0/1234044832 shutdown_connections 2024-03-19T01:12:01.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d5c70740 1 -- 172.21.15.116:0/1234044832 wait complete. 2024-03-19T01:12:01.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d5c70740 20 client.15093 trim_cache size 0 max 16384 2024-03-19T01:12:01.548 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:12:01.548 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "35912c14-e58c-11ee-95c9-87774f69a715", 2024-03-19T01:12:01.548 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 2, 2024-03-19T01:12:01.548 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24473, 2024-03-19T01:12:01.548 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T01:12:01.548 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T01:12:01.548 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T01:12:01.548 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 85.574872685000003, 2024-03-19T01:12:01.548 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 24, 2024-03-19T01:12:01.548 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 87, 2024-03-19T01:12:01.548 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 85, 2024-03-19T01:12:01.548 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 105.675667901 2024-03-19T01:12:01.549 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:12:01.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d308a640 1 -- 172.21.15.116:0/1878781650 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f06bc0a2800 msgr2=0x7f06bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d308a640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f06bc0a2800 0x7f06bc0a4cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f06c40a3be0 tx=0x7f06c40c63c0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d308a640 1 -- 172.21.15.116:0/1878781650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06cc107560 msgr2=0x7f06cc19f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d308a640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06cc107560 0x7f06cc19f170 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f06c400eb10 tx=0x7f06c400c2d0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d308a640 1 -- 172.21.15.116:0/1878781650 shutdown_connections 2024-03-19T01:12:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d308a640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f06cc107560 0x7f06cc19f170 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d308a640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f06cc104c30 0x7f06cc198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d308a640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f06bc0a2800 0x7f06bc0a4cc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d308a640 1 --2- 172.21.15.116:0/1878781650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06cc102300 0x7f06cc197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d308a640 1 -- 172.21.15.116:0/1878781650 >> 172.21.15.116:0/1878781650 conn(0x7f06cc0fbcb0 msgr2=0x7f06cc102de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d308a640 1 -- 172.21.15.116:0/1878781650 shutdown_connections 2024-03-19T01:12:01.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.538+0000 7f06d308a640 1 -- 172.21.15.116:0/1878781650 wait complete. 2024-03-19T01:12:01.551 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-19T01:12:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:01 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/4144606068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:12:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:01 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2972121464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:12:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:01 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1500918162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:12:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.657+0000 7f53c2922640 1 -- 172.21.15.116:0/2338509598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53bc108f90 msgr2=0x7f53bc109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.657+0000 7f53c2922640 1 --2- 172.21.15.116:0/2338509598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53bc108f90 0x7f53bc109410 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009900 tx=0x7f53ac02bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.657+0000 7f53c2922640 1 -- 172.21.15.116:0/2338509598 shutdown_connections 2024-03-19T01:12:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.657+0000 7f53c2922640 1 --2- 172.21.15.116:0/2338509598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53bc109950 0x7f53bc110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.657+0000 7f53c2922640 1 --2- 172.21.15.116:0/2338509598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53bc108f90 0x7f53bc109410 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.657+0000 7f53c2922640 1 --2- 172.21.15.116:0/2338509598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53bc107e40 0x7f53bc108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.657+0000 7f53c2922640 1 -- 172.21.15.116:0/2338509598 >> 172.21.15.116:0/2338509598 conn(0x7f53bc076de0 msgr2=0x7f53bc079220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.657+0000 7f53c2922640 1 -- 172.21.15.116:0/2338509598 shutdown_connections 2024-03-19T01:12:01.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.657+0000 7f53c2922640 1 -- 172.21.15.116:0/2338509598 wait complete. 2024-03-19T01:12:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c2922640 1 Processor -- start 2024-03-19T01:12:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c2922640 1 -- start start 2024-03-19T01:12:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c2922640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53bc107e40 0x7f53bc19e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c2922640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53bc108f90 0x7f53bc19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c2922640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53bc109950 0x7f53bc1a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c2922640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc113af0 con 0x7f53bc109950 2024-03-19T01:12:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c2922640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc113970 con 0x7f53bc108f90 2024-03-19T01:12:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c2922640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc113c70 con 0x7f53bc107e40 2024-03-19T01:12:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c0ebf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53bc109950 0x7f53bc1a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c0ebf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53bc109950 0x7f53bc1a5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59922/0 (socket says 172.21.15.116:59922) 2024-03-19T01:12:01.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c0ebf640 1 -- 172.21.15.116:0/3029387296 learned_addr learned my addr 172.21.15.116:0/3029387296 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53bbfff640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53bc107e40 0x7f53bc19e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53bc108f90 0x7f53bc19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c0ebf640 1 -- 172.21.15.116:0/3029387296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53bc107e40 msgr2=0x7f53bc19e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c0ebf640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53bc107e40 0x7f53bc19e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c0ebf640 1 -- 172.21.15.116:0/3029387296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53bc108f90 msgr2=0x7f53bc19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c0ebf640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53bc108f90 0x7f53bc19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53c0ebf640 1 -- 172.21.15.116:0/3029387296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53bc1a5d80 con 0x7f53bc109950 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.658+0000 7f53bbfff640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53bc107e40 0x7f53bc19e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.659+0000 7f53c0ebf640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53bc109950 0x7f53bc1a5710 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f53b400eb10 tx=0x7f53b400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.659+0000 7f53b97fa640 1 -- 172.21.15.116:0/3029387296 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b4004b20 con 0x7f53bc109950 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.659+0000 7f53c2922640 1 -- 172.21.15.116:0/3029387296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53bc1a6070 con 0x7f53bc109950 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.659+0000 7f53b97fa640 1 -- 172.21.15.116:0/3029387296 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b4004cc0 con 0x7f53bc109950 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.659+0000 7f53b97fa640 1 -- 172.21.15.116:0/3029387296 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b4010760 con 0x7f53bc109950 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.659+0000 7f53c2922640 1 -- 172.21.15.116:0/3029387296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53bc1a65e0 con 0x7f53bc109950 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.661+0000 7f53b97fa640 1 -- 172.21.15.116:0/3029387296 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f53b4005020 con 0x7f53bc109950 2024-03-19T01:12:01.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.661+0000 7f53c55ef740 1 Processor -- start 2024-03-19T01:12:01.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.661+0000 7f53c55ef740 1 -- start start 2024-03-19T01:12:01.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.661+0000 7f53c55ef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066120ff0 0x556066121410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.661+0000 7f53c55ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x5560661264e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.661+0000 7f53c55ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066126b50 0x556066128fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53c55ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556066085de0 con 0x5560661219e0 2024-03-19T01:12:01.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53c55ef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55606602ec80 con 0x556066120ff0 2024-03-19T01:12:01.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53c55ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556066081e80 con 0x556066126b50 2024-03-19T01:12:01.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53b97fa640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53a40a2800 0x7f53a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x5560661264e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x5560661264e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59926/0 (socket says 172.21.15.116:59926) 2024-03-19T01:12:01.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53bbfff640 1 -- 172.21.15.116:0/461634904 learned_addr learned my addr 172.21.15.116:0/461634904 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/461634904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066120ff0 0x556066121410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/461634904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066126b50 0x556066128fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53c0ebf640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53a40a2800 0x7f53a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53b97fa640 1 -- 172.21.15.116:0/3029387296 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f53b40cf160 con 0x7f53bc109950 2024-03-19T01:12:01.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53bbfff640 1 -- 172.21.15.116:0/461634904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066126b50 msgr2=0x556066128fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53bbfff640 1 --2- 172.21.15.116:0/461634904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066126b50 0x556066128fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53bbfff640 1 -- 172.21.15.116:0/461634904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066120ff0 msgr2=0x556066121410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53bbfff640 1 --2- 172.21.15.116:0/461634904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066120ff0 0x556066121410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53bbfff640 1 -- 172.21.15.116:0/461634904 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556066040fb0 con 0x5560661219e0 2024-03-19T01:12:01.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/461634904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066126b50 0x556066128fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:12:01.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53bbfff640 1 --2- 172.21.15.116:0/461634904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x5560661264e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f53b00115f0 tx=0x7f53b0011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ebcfb65f74734d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53c0ebf640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53a40a2800 0x7f53a40a4cc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f53b40984b0 tx=0x7f53b4090fc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53a27fc640 1 -- 172.21.15.116:0/461634904 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b001e070 con 0x5560661219e0 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53a27fc640 1 -- 172.21.15.116:0/461634904 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b000fb20 con 0x5560661219e0 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.662+0000 7f53a27fc640 1 -- 172.21.15.116:0/461634904 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b0019bf0 con 0x5560661219e0 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.663+0000 7f53c55ef740 1 -- 172.21.15.116:0/461634904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 msgr2=0x5560661264e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.663+0000 7f53c55ef740 1 --2- 172.21.15.116:0/461634904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x5560661264e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f53b00115f0 tx=0x7f53b0011ab0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.663+0000 7f53c55ef740 1 -- 172.21.15.116:0/461634904 shutdown_connections 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.663+0000 7f53c55ef740 1 --2- 172.21.15.116:0/461634904 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066126b50 0x556066128fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.663+0000 7f53c55ef740 1 --2- 172.21.15.116:0/461634904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x5560661264e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.663+0000 7f53c55ef740 1 --2- 172.21.15.116:0/461634904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066120ff0 0x556066121410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.663+0000 7f53c55ef740 1 -- 172.21.15.116:0/461634904 >> 172.21.15.116:0/461634904 conn(0x55606611c560 msgr2=0x55606611c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.663+0000 7f53c55ef740 1 -- 172.21.15.116:0/461634904 shutdown_connections 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.663+0000 7f53c55ef740 1 -- 172.21.15.116:0/461634904 wait complete. 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.663+0000 7f53c55ef740 1 -- start start 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.663+0000 7f53c55ef740 1 Processor -- start 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53c55ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556066120ff0 0x55606612b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53c55ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560661219e0 0x55606612ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53c55ef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066126b50 0x556066132a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53c55ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556066129b00 con 0x556066120ff0 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53c55ef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556066041fc0 con 0x556066126b50 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53c55ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556066129c80 con 0x5560661219e0 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53c55ef740 10 client.0 _ll_register_callbacks cb 0x55606607da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556066120ff0 0x55606612b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53bb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556066120ff0 0x55606612b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59930/0 (socket says 172.21.15.116:59930) 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53bb7fe640 1 -- 172.21.15.116:0/3474173086 learned_addr learned my addr 172.21.15.116:0/3474173086 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53bbfff640 1 --2- 172.21.15.116:0/3474173086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560661219e0 0x55606612ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/3474173086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066126b50 0x556066132a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53bb7fe640 1 -- 172.21.15.116:0/3474173086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560661219e0 msgr2=0x55606612ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/3474173086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560661219e0 0x55606612ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53bb7fe640 1 -- 172.21.15.116:0/3474173086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066126b50 msgr2=0x556066132a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/3474173086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066126b50 0x556066132a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53bb7fe640 1 -- 172.21.15.116:0/3474173086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556065d9a2d0 con 0x556066120ff0 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/3474173086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556066120ff0 0x55606612b510 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009900 tx=0x7f53ac02c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98177bb8fb1bd607 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f539b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f539b7fe640 1 -- 172.21.15.116:0/3474173086 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac047030 con 0x556066120ff0 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f539b7fe640 1 -- 172.21.15.116:0/3474173086 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f53ac003800 con 0x556066120ff0 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53c55ef740 1 -- 172.21.15.116:0/3474173086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53b000fb20 con 0x556066120ff0 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f539b7fe640 1 -- 172.21.15.116:0/3474173086 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f53ac052070 con 0x556066120ff0 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.664+0000 7f53c55ef740 10 client.15105 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.665+0000 7f53c55ef740 1 -- 172.21.15.116:0/3474173086 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556065e646b0 con 0x556066120ff0 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.665+0000 7f539b7fe640 1 -- 172.21.15.116:0/3474173086 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b000fb20 con 0x556066120ff0 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.665+0000 7f539b7fe640 1 -- 172.21.15.116:0/3474173086 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f53ac0349d0 con 0x556066120ff0 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.665+0000 7f53c55ef740 10 client.15105 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.665+0000 7f53c55ef740 10 client.15105 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.665+0000 7f53c55ef740 20 client.15105 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:01.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.665+0000 7f53c55ef740 1 --2- 172.21.15.116:0/3474173086 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x556066138370 0x5560661397e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.665+0000 7f53c55ef740 4 client.15105 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.665+0000 7f53c55ef740 1 -- 172.21.15.116:0/3474173086 --> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556066093320 con 0x556066138370 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.665+0000 7f53bbfff640 1 --2- 172.21.15.116:0/3474173086 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x556066138370 0x5560661397e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.665+0000 7f539b7fe640 10 client.15105 ms_handle_connect on v2:172.21.15.116:6840/3343188504 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.665+0000 7f53bbfff640 1 --2- 172.21.15.116:0/3474173086 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x556066138370 0x5560661397e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8dd3f654f6551098 server_cookie=4a5288bab7d4483e in_seq=0 out_seq=0 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f539b7fe640 1 -- 172.21.15.116:0/3474173086 <== mds.1 v2:172.21.15.116:6840/3343188504 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556066093320 con 0x556066138370 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f539b7fe640 10 client.15105 handle_command_reply: tid=0 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f53c55ef740 1 client.15105 shutdown 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f53c55ef740 1 -- 172.21.15.116:0/3474173086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556066120ff0 msgr2=0x55606612b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f53c55ef740 1 --2- 172.21.15.116:0/3474173086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556066120ff0 0x55606612b510 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009900 tx=0x7f53ac02c690 comp rx=0 tx=0).stop 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f53c55ef740 1 -- 172.21.15.116:0/3474173086 shutdown_connections 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f53c55ef740 1 --2- 172.21.15.116:0/3474173086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066126b50 0x556066132a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f53c55ef740 1 --2- 172.21.15.116:0/3474173086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5560661219e0 0x55606612ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f53c55ef740 1 --2- 172.21.15.116:0/3474173086 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x556066138370 0x5560661397e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f53c55ef740 1 --2- 172.21.15.116:0/3474173086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556066120ff0 0x55606612b510 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f539b7fe640 0 client.15105 ms_handle_reset on v2:172.21.15.116:6840/3343188504 2024-03-19T01:12:01.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f53c55ef740 1 -- 172.21.15.116:0/3474173086 >> 172.21.15.116:0/3474173086 conn(0x55606611c560 msgr2=0x55606611b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f53c55ef740 1 -- 172.21.15.116:0/3474173086 shutdown_connections 2024-03-19T01:12:01.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f53c55ef740 1 -- 172.21.15.116:0/3474173086 wait complete. 2024-03-19T01:12:01.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.666+0000 7f53c55ef740 20 client.15105 trim_cache size 0 max 16384 2024-03-19T01:12:01.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.684+0000 7f53c55ef740 1 Processor -- start 2024-03-19T01:12:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.684+0000 7f53c55ef740 1 -- start start 2024-03-19T01:12:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.684+0000 7f53c55ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066120ff0 0x55606612b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.684+0000 7f53c55ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x556066132a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.684+0000 7f53c55ef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066126b50 0x55606612b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.684+0000 7f53c55ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556066137b20 con 0x5560661219e0 2024-03-19T01:12:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.684+0000 7f53c55ef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5560661381f0 con 0x556066126b50 2024-03-19T01:12:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.684+0000 7f53c55ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5560661388c0 con 0x556066120ff0 2024-03-19T01:12:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.684+0000 7f53bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x556066132a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.684+0000 7f53bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x556066132a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59942/0 (socket says 172.21.15.116:59942) 2024-03-19T01:12:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.684+0000 7f53bbfff640 1 -- 172.21.15.116:0/1592416924 learned_addr learned my addr 172.21.15.116:0/1592416924 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.684+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/1592416924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066120ff0 0x55606612b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.685+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/1592416924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066126b50 0x55606612b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.685+0000 7f53bbfff640 1 -- 172.21.15.116:0/1592416924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066120ff0 msgr2=0x55606612b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.685+0000 7f53bbfff640 1 --2- 172.21.15.116:0/1592416924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066120ff0 0x55606612b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.685+0000 7f53bbfff640 1 -- 172.21.15.116:0/1592416924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066126b50 msgr2=0x55606612b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.685+0000 7f53bbfff640 1 --2- 172.21.15.116:0/1592416924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066126b50 0x55606612b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.685+0000 7f53bbfff640 1 -- 172.21.15.116:0/1592416924 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55606612c500 con 0x5560661219e0 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.685+0000 7f53bbfff640 1 --2- 172.21.15.116:0/1592416924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x556066132a50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f53b0009230 tx=0x7f53b0013fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46a3a1562714580 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.685+0000 7f53a1ffb640 1 -- 172.21.15.116:0/1592416924 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b001e040 con 0x5560661219e0 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.685+0000 7f53a1ffb640 1 -- 172.21.15.116:0/1592416924 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b002a770 con 0x5560661219e0 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.685+0000 7f53a1ffb640 1 -- 172.21.15.116:0/1592416924 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b0019910 con 0x5560661219e0 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 -- 172.21.15.116:0/1592416924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 msgr2=0x556066132a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 --2- 172.21.15.116:0/1592416924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x556066132a50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f53b0009230 tx=0x7f53b0013fd0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 -- 172.21.15.116:0/1592416924 shutdown_connections 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 --2- 172.21.15.116:0/1592416924 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066126b50 0x55606612b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 --2- 172.21.15.116:0/1592416924 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x556066132a50 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 --2- 172.21.15.116:0/1592416924 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066120ff0 0x55606612b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 -- 172.21.15.116:0/1592416924 >> 172.21.15.116:0/1592416924 conn(0x55606611c560 msgr2=0x556066122ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 -- 172.21.15.116:0/1592416924 shutdown_connections 2024-03-19T01:12:01.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 -- 172.21.15.116:0/1592416924 wait complete. 2024-03-19T01:12:01.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 -- start start 2024-03-19T01:12:01.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 Processor -- start 2024-03-19T01:12:01.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066120ff0 0x55606612ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x55606612f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066126b50 0x556066131d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556066139660 con 0x5560661219e0 2024-03-19T01:12:01.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556066138f90 con 0x556066120ff0 2024-03-19T01:12:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556066139d30 con 0x556066126b50 2024-03-19T01:12:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53c55ef740 10 client.0 _ll_register_callbacks cb 0x7f53ac009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x55606612f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x55606612f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59944/0 (socket says 172.21.15.116:59944) 2024-03-19T01:12:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.686+0000 7f53bbfff640 1 -- 172.21.15.116:0/894712772 learned_addr learned my addr 172.21.15.116:0/894712772 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066120ff0 0x55606612ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066126b50 0x556066131d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f53bbfff640 1 -- 172.21.15.116:0/894712772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066126b50 msgr2=0x556066131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f53bbfff640 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066126b50 0x556066131d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f53bbfff640 1 -- 172.21.15.116:0/894712772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066120ff0 msgr2=0x55606612ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f53bbfff640 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066120ff0 0x55606612ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f53bbfff640 1 -- 172.21.15.116:0/894712772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556066132300 con 0x5560661219e0 2024-03-19T01:12:01.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066120ff0 0x55606612ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f53bbfff640 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x55606612f1c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f53b001c040 tx=0x7f53b0009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1a6dcbefd48711 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f539b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f539b7fe640 1 -- 172.21.15.116:0/894712772 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b0037020 con 0x5560661219e0 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f539b7fe640 1 -- 172.21.15.116:0/894712772 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f53b002a290 con 0x5560661219e0 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f53c55ef740 1 -- 172.21.15.116:0/894712772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53b002a770 con 0x5560661219e0 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f539b7fe640 1 -- 172.21.15.116:0/894712772 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f53b0042050 con 0x5560661219e0 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f53c55ef740 10 client.15111 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f53c55ef740 1 -- 172.21.15.116:0/894712772 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55606619e6b0 con 0x5560661219e0 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.687+0000 7f539b7fe640 1 -- 172.21.15.116:0/894712772 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53b002a770 con 0x5560661219e0 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f539b7fe640 1 -- 172.21.15.116:0/894712772 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f53b0019cb0 con 0x5560661219e0 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f53c55ef740 10 client.15111 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f53c55ef740 10 client.15111 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f53c55ef740 20 client.15111 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f53c55ef740 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x5560661a2c50 0x5560661a1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f53c55ef740 4 client.15111 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f53c55ef740 1 -- 172.21.15.116:0/894712772 --> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] -- command(tid 0: {"prefix": "status"}) -- 0x55606608d670 con 0x5560661a2c50 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x5560661a2c50 0x5560661a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f539b7fe640 10 client.15111 ms_handle_connect on v2:172.21.15.116:6840/3343188504 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f53bb7fe640 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x5560661a2c50 0x5560661a1960 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2954acc6c6e43292 server_cookie=1abc9ab1bdae2a88 in_seq=0 out_seq=0 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f539b7fe640 1 -- 172.21.15.116:0/894712772 <== mds.1 v2:172.21.15.116:6840/3343188504 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55606608d670 con 0x5560661a2c50 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f539b7fe640 10 client.15111 handle_command_reply: tid=0 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f53c55ef740 1 client.15111 shutdown 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f53c55ef740 1 -- 172.21.15.116:0/894712772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 msgr2=0x55606612f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f53c55ef740 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x55606612f1c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f53b001c040 tx=0x7f53b0009f60 comp rx=0 tx=0).stop 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f53c55ef740 1 -- 172.21.15.116:0/894712772 shutdown_connections 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.688+0000 7f53c55ef740 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556066126b50 0x556066131d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c55ef740 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x5560661a2c50 0x5560661a1960 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c55ef740 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5560661219e0 0x55606612f1c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f539b7fe640 0 client.15111 ms_handle_reset on v2:172.21.15.116:6840/3343188504 2024-03-19T01:12:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c55ef740 1 --2- 172.21.15.116:0/894712772 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556066120ff0 0x55606612ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c55ef740 1 -- 172.21.15.116:0/894712772 >> 172.21.15.116:0/894712772 conn(0x55606611c560 msgr2=0x55606612b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c55ef740 1 -- 172.21.15.116:0/894712772 shutdown_connections 2024-03-19T01:12:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c55ef740 1 -- 172.21.15.116:0/894712772 wait complete. 2024-03-19T01:12:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c55ef740 20 client.15111 trim_cache size 0 max 16384 2024-03-19T01:12:01.697 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:12:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "35912c14-e58c-11ee-95c9-87774f69a715", 2024-03-19T01:12:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 1, 2024-03-19T01:12:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24488, 2024-03-19T01:12:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T01:12:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T01:12:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T01:12:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 86.748309304000003, 2024-03-19T01:12:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 24, 2024-03-19T01:12:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 87, 2024-03-19T01:12:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 85, 2024-03-19T01:12:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 97.262952329000001 2024-03-19T01:12:01.698 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:12:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c2922640 1 -- 172.21.15.116:0/3029387296 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53a40a2800 msgr2=0x7f53a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c2922640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53a40a2800 0x7f53a40a4cc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f53b40984b0 tx=0x7f53b4090fc0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c2922640 1 -- 172.21.15.116:0/3029387296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53bc109950 msgr2=0x7f53bc1a5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c2922640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53bc109950 0x7f53bc1a5710 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f53b400eb10 tx=0x7f53b400c2d0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c2922640 1 -- 172.21.15.116:0/3029387296 shutdown_connections 2024-03-19T01:12:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c2922640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53bc109950 0x7f53bc1a5710 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c2922640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53a40a2800 0x7f53a40a4cc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c2922640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53bc108f90 0x7f53bc19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c2922640 1 --2- 172.21.15.116:0/3029387296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53bc107e40 0x7f53bc19e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c2922640 1 -- 172.21.15.116:0/3029387296 >> 172.21.15.116:0/3029387296 conn(0x7f53bc076de0 msgr2=0x7f53bc10c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c2922640 1 -- 172.21.15.116:0/3029387296 shutdown_connections 2024-03-19T01:12:01.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.689+0000 7f53c2922640 1 -- 172.21.15.116:0/3029387296 wait complete. 2024-03-19T01:12:01.701 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-19T01:12:01.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.806+0000 7fa63e910640 1 -- 172.21.15.116:0/506152882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa638101910 msgr2=0x7fa638101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.806+0000 7fa63e910640 1 --2- 172.21.15.116:0/506152882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa638101910 0x7fa638101d90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa628009860 tx=0x7fa62802bb70 comp rx=0 tx=0).stop 2024-03-19T01:12:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 -- 172.21.15.116:0/506152882 shutdown_connections 2024-03-19T01:12:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 --2- 172.21.15.116:0/506152882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6381022d0 0x7fa638108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 --2- 172.21.15.116:0/506152882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa638101910 0x7fa638101d90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 --2- 172.21.15.116:0/506152882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa638100720 0x7fa638100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 -- 172.21.15.116:0/506152882 >> 172.21.15.116:0/506152882 conn(0x7fa6380fbcb0 msgr2=0x7fa6380fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 -- 172.21.15.116:0/506152882 shutdown_connections 2024-03-19T01:12:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 -- 172.21.15.116:0/506152882 wait complete. 2024-03-19T01:12:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 Processor -- start 2024-03-19T01:12:01.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 -- start start 2024-03-19T01:12:01.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa638100720 0x7fa6380694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa638101910 0x7fa63819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6381022d0 0x7fa63819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa63810c3a0 con 0x7fa638100720 2024-03-19T01:12:01.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa63810c220 con 0x7fa638101910 2024-03-19T01:12:01.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.807+0000 7fa63e910640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa63810c520 con 0x7fa6381022d0 2024-03-19T01:12:01.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa637fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa638100720 0x7fa6380694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa637fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa638100720 0x7fa6380694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59976/0 (socket says 172.21.15.116:59976) 2024-03-19T01:12:01.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa637fff640 1 -- 172.21.15.116:0/4091128414 learned_addr learned my addr 172.21.15.116:0/4091128414 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa6377fe640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa638101910 0x7fa63819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa63cead640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6381022d0 0x7fa63819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa637fff640 1 -- 172.21.15.116:0/4091128414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6381022d0 msgr2=0x7fa63819ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa637fff640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6381022d0 0x7fa63819ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa637fff640 1 -- 172.21.15.116:0/4091128414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa638101910 msgr2=0x7fa63819c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa637fff640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa638101910 0x7fa63819c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa637fff640 1 -- 172.21.15.116:0/4091128414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa63819f850 con 0x7fa638100720 2024-03-19T01:12:01.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa6377fe640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa638101910 0x7fa63819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:01.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa637fff640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa638100720 0x7fa6380694f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00e9c0 tx=0x7fa62c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa6357fa640 1 -- 172.21.15.116:0/4091128414 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c00cd80 con 0x7fa638100720 2024-03-19T01:12:01.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa63e910640 1 -- 172.21.15.116:0/4091128414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa63819fae0 con 0x7fa638100720 2024-03-19T01:12:01.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa6357fa640 1 -- 172.21.15.116:0/4091128414 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa62c0035f0 con 0x7fa638100720 2024-03-19T01:12:01.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa6357fa640 1 -- 172.21.15.116:0/4091128414 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62c010690 con 0x7fa638100720 2024-03-19T01:12:01.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.808+0000 7fa63e910640 1 -- 172.21.15.116:0/4091128414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6381a0000 con 0x7fa638100720 2024-03-19T01:12:01.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa6415db740 1 Processor -- start 2024-03-19T01:12:01.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa6415db740 1 -- start start 2024-03-19T01:12:01.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa6357fa640 1 -- 172.21.15.116:0/4091128414 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fa62c0037a0 con 0x7fa638100720 2024-03-19T01:12:01.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa6415db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 0x562ac7cee410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa6415db740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cee9e0 0x562ac7cf34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa6415db740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cf3b50 0x562ac7cf5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa6415db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562ac7c52de0 con 0x562ac7cedff0 2024-03-19T01:12:01.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa6415db740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562ac7bfbc80 con 0x562ac7cf3b50 2024-03-19T01:12:01.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa6415db740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562ac7c4ee80 con 0x562ac7cee9e0 2024-03-19T01:12:01.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa63cead640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 0x562ac7cee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa63cead640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 0x562ac7cee410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59980/0 (socket says 172.21.15.116:59980) 2024-03-19T01:12:01.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa63cead640 1 -- 172.21.15.116:0/2429943940 learned_addr learned my addr 172.21.15.116:0/2429943940 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa637fff640 1 --2- 172.21.15.116:0/2429943940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cee9e0 0x562ac7cf34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.811+0000 7fa6377fe640 1 --2- 172.21.15.116:0/2429943940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cf3b50 0x562ac7cf5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.812+0000 7fa63cead640 1 -- 172.21.15.116:0/2429943940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cee9e0 msgr2=0x562ac7cf34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.812+0000 7fa63cead640 1 --2- 172.21.15.116:0/2429943940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cee9e0 0x562ac7cf34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.812+0000 7fa63cead640 1 -- 172.21.15.116:0/2429943940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cf3b50 msgr2=0x562ac7cf5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.812+0000 7fa63cead640 1 --2- 172.21.15.116:0/2429943940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cf3b50 0x562ac7cf5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.812+0000 7fa63cead640 1 -- 172.21.15.116:0/2429943940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ac7c0dfb0 con 0x562ac7cedff0 2024-03-19T01:12:01.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.812+0000 7fa6357fa640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa6200a2800 0x7fa6200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.812+0000 7fa63cead640 1 --2- 172.21.15.116:0/2429943940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 0x562ac7cee410 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa63000e450 tx=0x7fa6300095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4960baed359ea15f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.812+0000 7fa637fff640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa6200a2800 0x7fa6200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.812+0000 7fa61e7fc640 1 -- 172.21.15.116:0/2429943940 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63000d910 con 0x562ac7cedff0 2024-03-19T01:12:01.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.812+0000 7fa61e7fc640 1 -- 172.21.15.116:0/2429943940 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa63001c070 con 0x562ac7cedff0 2024-03-19T01:12:01.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.812+0000 7fa61e7fc640 1 -- 172.21.15.116:0/2429943940 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa630016380 con 0x562ac7cedff0 2024-03-19T01:12:01.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.812+0000 7fa6357fa640 1 -- 172.21.15.116:0/4091128414 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa62c0cede0 con 0x7fa638100720 2024-03-19T01:12:01.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.813+0000 7fa6415db740 1 -- 172.21.15.116:0/2429943940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 msgr2=0x562ac7cee410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.813+0000 7fa637fff640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa6200a2800 0x7fa6200a4cc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0ccc30 tx=0x7fa62c0d7040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.813+0000 7fa6415db740 1 --2- 172.21.15.116:0/2429943940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 0x562ac7cee410 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa63000e450 tx=0x7fa6300095c0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.813+0000 7fa6415db740 1 -- 172.21.15.116:0/2429943940 shutdown_connections 2024-03-19T01:12:01.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.813+0000 7fa6415db740 1 --2- 172.21.15.116:0/2429943940 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cf3b50 0x562ac7cf5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.813+0000 7fa6415db740 1 --2- 172.21.15.116:0/2429943940 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cee9e0 0x562ac7cf34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.813+0000 7fa6415db740 1 --2- 172.21.15.116:0/2429943940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 0x562ac7cee410 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.813+0000 7fa6415db740 1 -- 172.21.15.116:0/2429943940 >> 172.21.15.116:0/2429943940 conn(0x562ac7ce9560 msgr2=0x562ac7ce9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.813+0000 7fa6415db740 1 -- 172.21.15.116:0/2429943940 shutdown_connections 2024-03-19T01:12:01.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.813+0000 7fa6415db740 1 -- 172.21.15.116:0/2429943940 wait complete. 2024-03-19T01:12:01.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.813+0000 7fa6415db740 1 -- start start 2024-03-19T01:12:01.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.813+0000 7fa6415db740 1 Processor -- start 2024-03-19T01:12:01.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa6415db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 0x562ac7cf84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa6415db740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cee9e0 0x562ac7cf8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa6415db740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cf3b50 0x562ac7cffa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa6415db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562ac7cf6b00 con 0x562ac7cedff0 2024-03-19T01:12:01.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa6415db740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562ac7c0efc0 con 0x562ac7cee9e0 2024-03-19T01:12:01.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa6415db740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562ac7cf6c80 con 0x562ac7cf3b50 2024-03-19T01:12:01.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa6415db740 10 client.0 _ll_register_callbacks cb 0x562ac7c4aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:01.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa63cead640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 0x562ac7cf84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa63cead640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 0x562ac7cf84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59992/0 (socket says 172.21.15.116:59992) 2024-03-19T01:12:01.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa63cead640 1 -- 172.21.15.116:0/2503829694 learned_addr learned my addr 172.21.15.116:0/2503829694 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa6377fe640 1 --2- 172.21.15.116:0/2503829694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cee9e0 0x562ac7cf8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa63cead640 1 --2- 172.21.15.116:0/2503829694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cf3b50 0x562ac7cffa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa63cead640 1 -- 172.21.15.116:0/2503829694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cf3b50 msgr2=0x562ac7cffa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa63cead640 1 --2- 172.21.15.116:0/2503829694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cf3b50 0x562ac7cffa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa63cead640 1 -- 172.21.15.116:0/2503829694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cee9e0 msgr2=0x562ac7cf8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa63cead640 1 --2- 172.21.15.116:0/2503829694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cee9e0 0x562ac7cf8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa63cead640 1 -- 172.21.15.116:0/2503829694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ac79672d0 con 0x562ac7cedff0 2024-03-19T01:12:01.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa63cead640 1 --2- 172.21.15.116:0/2503829694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 0x562ac7cf84f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa630005e30 tx=0x7fa630005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68272a80a613bdb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa6177fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:01.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa6177fe640 1 -- 172.21.15.116:0/2503829694 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63002b030 con 0x562ac7cedff0 2024-03-19T01:12:01.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa6177fe640 1 -- 172.21.15.116:0/2503829694 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa630004c50 con 0x562ac7cedff0 2024-03-19T01:12:01.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa6415db740 1 -- 172.21.15.116:0/2503829694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa63001c070 con 0x562ac7cedff0 2024-03-19T01:12:01.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.814+0000 7fa6177fe640 1 -- 172.21.15.116:0/2503829694 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa630003970 con 0x562ac7cedff0 2024-03-19T01:12:01.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.815+0000 7fa6415db740 10 client.15123 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:01.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.815+0000 7fa6415db740 1 -- 172.21.15.116:0/2503829694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562ac7a316b0 con 0x562ac7cedff0 2024-03-19T01:12:01.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.815+0000 7fa6177fe640 1 -- 172.21.15.116:0/2503829694 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa63001c070 con 0x562ac7cedff0 2024-03-19T01:12:01.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.815+0000 7fa6177fe640 1 -- 172.21.15.116:0/2503829694 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fa63000b040 con 0x562ac7cedff0 2024-03-19T01:12:01.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.815+0000 7fa6415db740 10 client.15123 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:01.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.815+0000 7fa6415db740 10 client.15123 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T01:12:01.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.815+0000 7fa6415db740 20 client.15123 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:01.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.815+0000 7fa6415db740 1 --2- 172.21.15.116:0/2503829694 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562ac7d04e80 0x562ac7d062f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.815+0000 7fa6415db740 4 client.15123 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:12:01.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.815+0000 7fa6415db740 1 -- 172.21.15.116:0/2503829694 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ac7c60320 con 0x562ac7d04e80 2024-03-19T01:12:01.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.815+0000 7fa6377fe640 1 --2- 172.21.15.116:0/2503829694 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562ac7d04e80 0x562ac7d062f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6177fe640 10 client.15123 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:01.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6377fe640 1 --2- 172.21.15.116:0/2503829694 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562ac7d04e80 0x562ac7d062f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24ab87755fe6afbb server_cookie=86a1569379f2d6c7 in_seq=0 out_seq=0 2024-03-19T01:12:01.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6177fe640 1 -- 172.21.15.116:0/2503829694 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562ac7c60320 con 0x562ac7d04e80 2024-03-19T01:12:01.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6177fe640 10 client.15123 handle_command_reply: tid=0 2024-03-19T01:12:01.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6415db740 1 client.15123 shutdown 2024-03-19T01:12:01.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6415db740 1 -- 172.21.15.116:0/2503829694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 msgr2=0x562ac7cf84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6415db740 1 --2- 172.21.15.116:0/2503829694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 0x562ac7cf84f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa630005e30 tx=0x7fa630005d80 comp rx=0 tx=0).stop 2024-03-19T01:12:01.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6415db740 1 -- 172.21.15.116:0/2503829694 shutdown_connections 2024-03-19T01:12:01.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6415db740 1 --2- 172.21.15.116:0/2503829694 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562ac7d04e80 0x562ac7d062f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6415db740 1 --2- 172.21.15.116:0/2503829694 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cf3b50 0x562ac7cffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6177fe640 0 client.15123 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:01.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6415db740 1 --2- 172.21.15.116:0/2503829694 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cee9e0 0x562ac7cf8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6415db740 1 --2- 172.21.15.116:0/2503829694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cedff0 0x562ac7cf84f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.816+0000 7fa6415db740 1 -- 172.21.15.116:0/2503829694 >> 172.21.15.116:0/2503829694 conn(0x562ac7ce9560 msgr2=0x562ac7ce87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.817+0000 7fa6415db740 1 -- 172.21.15.116:0/2503829694 shutdown_connections 2024-03-19T01:12:01.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.817+0000 7fa6415db740 1 -- 172.21.15.116:0/2503829694 wait complete. 2024-03-19T01:12:01.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.817+0000 7fa6415db740 20 client.15123 trim_cache size 0 max 16384 2024-03-19T01:12:01.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.834+0000 7fa6415db740 1 Processor -- start 2024-03-19T01:12:01.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.834+0000 7fa6415db740 1 -- start start 2024-03-19T01:12:01.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.834+0000 7fa6415db740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cedff0 0x562ac7cf7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa6415db740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cee9e0 0x562ac7ce9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa6415db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cf3b50 0x562ac7cf8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa6415db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562ac7d04630 con 0x562ac7cf3b50 2024-03-19T01:12:01.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa6415db740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562ac7d04d00 con 0x562ac7cee9e0 2024-03-19T01:12:01.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa6415db740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562ac7d053d0 con 0x562ac7cedff0 2024-03-19T01:12:01.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa63cead640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cedff0 0x562ac7cf7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa6377fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cee9e0 0x562ac7ce9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa63cead640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cedff0 0x562ac7cf7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:59826/0 (socket says 172.21.15.116:59826) 2024-03-19T01:12:01.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa63cead640 1 -- 172.21.15.116:0/3460710440 learned_addr learned my addr 172.21.15.116:0/3460710440 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa6377fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cee9e0 0x562ac7ce9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:54794/0 (socket says 172.21.15.116:54794) 2024-03-19T01:12:01.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa63cead640 1 --2- 172.21.15.116:0/3460710440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cf3b50 0x562ac7cf8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa6377fe640 1 -- 172.21.15.116:0/3460710440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cedff0 msgr2=0x562ac7cf7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa6377fe640 1 --2- 172.21.15.116:0/3460710440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cedff0 0x562ac7cf7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa6377fe640 1 -- 172.21.15.116:0/3460710440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cf3b50 msgr2=0x562ac7cf8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa6377fe640 1 --2- 172.21.15.116:0/3460710440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cf3b50 0x562ac7cf8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa6377fe640 1 -- 172.21.15.116:0/3460710440 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ac7cf31c0 con 0x562ac7cee9e0 2024-03-19T01:12:01.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa63cead640 1 --2- 172.21.15.116:0/3460710440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cedff0 0x562ac7cf7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:01.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa6377fe640 1 --2- 172.21.15.116:0/3460710440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cee9e0 0x562ac7ce9940 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa6280349e0 tx=0x7fa628005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5427cc86b12b90f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa61dffb640 1 -- 172.21.15.116:0/3460710440 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628039040 con 0x562ac7cee9e0 2024-03-19T01:12:01.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa61dffb640 1 -- 172.21.15.116:0/3460710440 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa62804f070 con 0x562ac7cee9e0 2024-03-19T01:12:01.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.835+0000 7fa61dffb640 1 -- 172.21.15.116:0/3460710440 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62804a300 con 0x562ac7cee9e0 2024-03-19T01:12:01.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.836+0000 7fa6415db740 1 -- 172.21.15.116:0/3460710440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cee9e0 msgr2=0x562ac7ce9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.836+0000 7fa6415db740 1 --2- 172.21.15.116:0/3460710440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cee9e0 0x562ac7ce9940 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa6280349e0 tx=0x7fa628005f90 comp rx=0 tx=0).stop 2024-03-19T01:12:01.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.836+0000 7fa6415db740 1 -- 172.21.15.116:0/3460710440 shutdown_connections 2024-03-19T01:12:01.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.836+0000 7fa6415db740 1 --2- 172.21.15.116:0/3460710440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cf3b50 0x562ac7cf8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.836+0000 7fa6415db740 1 --2- 172.21.15.116:0/3460710440 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cee9e0 0x562ac7ce9940 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.836+0000 7fa6415db740 1 --2- 172.21.15.116:0/3460710440 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cedff0 0x562ac7cf7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.836+0000 7fa6415db740 1 -- 172.21.15.116:0/3460710440 >> 172.21.15.116:0/3460710440 conn(0x562ac7ce9560 msgr2=0x562ac7cef480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.836+0000 7fa6415db740 1 -- 172.21.15.116:0/3460710440 shutdown_connections 2024-03-19T01:12:01.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.836+0000 7fa6415db740 1 -- 172.21.15.116:0/3460710440 wait complete. 2024-03-19T01:12:01.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.836+0000 7fa6415db740 1 -- start start 2024-03-19T01:12:01.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.836+0000 7fa6415db740 1 Processor -- start 2024-03-19T01:12:01.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6415db740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cedff0 0x562ac7cfd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6415db740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cee9e0 0x562ac7cfdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6415db740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cf3b50 0x562ac7cfe4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6415db740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562ac7d06170 con 0x562ac7cee9e0 2024-03-19T01:12:01.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6415db740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562ac7d05aa0 con 0x562ac7cedff0 2024-03-19T01:12:01.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6415db740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562ac7d06840 con 0x562ac7cf3b50 2024-03-19T01:12:01.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6415db740 10 client.0 _ll_register_callbacks cb 0x7fa630009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:01.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6377fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cee9e0 0x562ac7cfdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa63cead640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cedff0 0x562ac7cfd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6377fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cee9e0 0x562ac7cfdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60004/0 (socket says 172.21.15.116:60004) 2024-03-19T01:12:01.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6377fe640 1 -- 172.21.15.116:0/2469505492 learned_addr learned my addr 172.21.15.116:0/2469505492 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:01.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa63cead640 1 --2- 172.21.15.116:0/2469505492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cf3b50 0x562ac7cfe4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6377fe640 1 -- 172.21.15.116:0/2469505492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cf3b50 msgr2=0x562ac7cfe4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6377fe640 1 --2- 172.21.15.116:0/2469505492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cf3b50 0x562ac7cfe4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6377fe640 1 -- 172.21.15.116:0/2469505492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cedff0 msgr2=0x562ac7cfd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:01.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6377fe640 1 --2- 172.21.15.116:0/2469505492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cedff0 0x562ac7cfd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6377fe640 1 -- 172.21.15.116:0/2469505492 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ac7cfed00 con 0x562ac7cee9e0 2024-03-19T01:12:01.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa63cead640 1 --2- 172.21.15.116:0/2469505492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cedff0 0x562ac7cfd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:01.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6377fe640 1 --2- 172.21.15.116:0/2469505492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cee9e0 0x562ac7cfdec0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa628009860 tx=0x7fa62802bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce7ab7327b77f71 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:01.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6177fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:01.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6177fe640 1 -- 172.21.15.116:0/2469505492 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628057020 con 0x562ac7cee9e0 2024-03-19T01:12:01.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6177fe640 1 -- 172.21.15.116:0/2469505492 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa628034620 con 0x562ac7cee9e0 2024-03-19T01:12:01.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6415db740 1 -- 172.21.15.116:0/2469505492 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa62804f070 con 0x562ac7cee9e0 2024-03-19T01:12:01.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.837+0000 7fa6177fe640 1 -- 172.21.15.116:0/2469505492 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa628062050 con 0x562ac7cee9e0 2024-03-19T01:12:01.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.838+0000 7fa6415db740 10 client.15129 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:01.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.838+0000 7fa6415db740 1 -- 172.21.15.116:0/2469505492 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562ac7d6af50 con 0x562ac7cee9e0 2024-03-19T01:12:01.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.838+0000 7fa6177fe640 1 -- 172.21.15.116:0/2469505492 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa62804f070 con 0x562ac7cee9e0 2024-03-19T01:12:01.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.838+0000 7fa6177fe640 1 -- 172.21.15.116:0/2469505492 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fa62802c510 con 0x562ac7cee9e0 2024-03-19T01:12:01.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.838+0000 7fa6415db740 10 client.15129 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:01.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.838+0000 7fa6415db740 10 client.15129 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T01:12:01.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.838+0000 7fa6415db740 20 client.15129 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:01.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.838+0000 7fa6415db740 1 --2- 172.21.15.116:0/2469505492 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562ac7d04e80 0x562ac7d6e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:01.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.838+0000 7fa6415db740 4 client.15129 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T01:12:01.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.838+0000 7fa6415db740 1 -- 172.21.15.116:0/2469505492 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "status"}) -- 0x562ac7c5a670 con 0x562ac7d04e80 2024-03-19T01:12:01.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.838+0000 7fa63cead640 1 --2- 172.21.15.116:0/2469505492 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562ac7d04e80 0x562ac7d6e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:01.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.838+0000 7fa6177fe640 10 client.15129 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:01.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.838+0000 7fa63cead640 1 --2- 172.21.15.116:0/2469505492 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562ac7d04e80 0x562ac7d6e3c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58116b47def03bbf server_cookie=f09b0b6063a77455 in_seq=0 out_seq=0 2024-03-19T01:12:01.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6177fe640 1 -- 172.21.15.116:0/2469505492 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x562ac7c5a670 con 0x562ac7d04e80 2024-03-19T01:12:01.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6177fe640 10 client.15129 handle_command_reply: tid=0 2024-03-19T01:12:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6415db740 1 client.15129 shutdown 2024-03-19T01:12:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6415db740 1 -- 172.21.15.116:0/2469505492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cee9e0 msgr2=0x562ac7cfdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6415db740 1 --2- 172.21.15.116:0/2469505492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cee9e0 0x562ac7cfdec0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa628009860 tx=0x7fa62802bb70 comp rx=0 tx=0).stop 2024-03-19T01:12:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6415db740 1 -- 172.21.15.116:0/2469505492 shutdown_connections 2024-03-19T01:12:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6415db740 1 --2- 172.21.15.116:0/2469505492 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562ac7d04e80 0x562ac7d6e3c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6415db740 1 --2- 172.21.15.116:0/2469505492 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562ac7cf3b50 0x562ac7cfe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6415db740 1 --2- 172.21.15.116:0/2469505492 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562ac7cee9e0 0x562ac7cfdec0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6177fe640 0 client.15129 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6415db740 1 --2- 172.21.15.116:0/2469505492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562ac7cedff0 0x562ac7cfd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6415db740 1 -- 172.21.15.116:0/2469505492 >> 172.21.15.116:0/2469505492 conn(0x562ac7ce9560 msgr2=0x562ac7ce9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6415db740 1 -- 172.21.15.116:0/2469505492 shutdown_connections 2024-03-19T01:12:01.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6415db740 1 -- 172.21.15.116:0/2469505492 wait complete. 2024-03-19T01:12:01.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa6415db740 20 client.15129 trim_cache size 0 max 16384 2024-03-19T01:12:01.852 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:12:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "35912c14-e58c-11ee-95c9-87774f69a715", 2024-03-19T01:12:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-03-19T01:12:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24500, 2024-03-19T01:12:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T01:12:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T01:12:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T01:12:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 88.690035359999996, 2024-03-19T01:12:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 24, 2024-03-19T01:12:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 87, 2024-03-19T01:12:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 85, 2024-03-19T01:12:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 94.462037265000006 2024-03-19T01:12:01.853 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:12:01.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa63e910640 1 -- 172.21.15.116:0/4091128414 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa6200a2800 msgr2=0x7fa6200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa63e910640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa6200a2800 0x7fa6200a4cc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa62c0ccc30 tx=0x7fa62c0d7040 comp rx=0 tx=0).stop 2024-03-19T01:12:01.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa63e910640 1 -- 172.21.15.116:0/4091128414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa638100720 msgr2=0x7fa6380694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:01.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.839+0000 7fa63e910640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa638100720 0x7fa6380694f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa62c00e9c0 tx=0x7fa62c00ee80 comp rx=0 tx=0).stop 2024-03-19T01:12:01.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.840+0000 7fa63e910640 1 -- 172.21.15.116:0/4091128414 shutdown_connections 2024-03-19T01:12:01.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.840+0000 7fa63e910640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6381022d0 0x7fa63819ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.840+0000 7fa63e910640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa6200a2800 0x7fa6200a4cc0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.840+0000 7fa63e910640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa638101910 0x7fa63819c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.840+0000 7fa63e910640 1 --2- 172.21.15.116:0/4091128414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa638100720 0x7fa6380694f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:01.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.840+0000 7fa63e910640 1 -- 172.21.15.116:0/4091128414 >> 172.21.15.116:0/4091128414 conn(0x7fa6380fbcb0 msgr2=0x7fa638104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:01.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.840+0000 7fa63e910640 1 -- 172.21.15.116:0/4091128414 shutdown_connections 2024-03-19T01:12:01.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:01.840+0000 7fa63e910640 1 -- 172.21.15.116:0/4091128414 wait complete. 2024-03-19T01:12:01.857 INFO:tasks.cephadm:Setup complete, yielding 2024-03-19T01:12:01.857 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T01:12:02.052 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T01:12:02.053 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-19T01:12:02.304 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:12:02.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:02 smithi116 ceph-mon[62701]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:12:02.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:02 smithi156 ceph-mon[66799]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:12:02.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:02 smithi162 ceph-mon[67178]: pgmap v334: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:12:03.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:03 smithi116 ceph-mon[62701]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:12:03.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:03 smithi156 ceph-mon[66799]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:12:03.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:03 smithi162 ceph-mon[67178]: pgmap v335: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:12:03.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.990+0000 7f55ae432700 1 -- 172.21.15.116:0/1814838349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a8103f80 msgr2=0x7f55a810a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.990+0000 7f55ae432700 1 --2- 172.21.15.116:0/1814838349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a8103f80 0x7f55a810a830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f55a000b0a0 tx=0x7f55a000b3b0 comp rx=0 tx=0).stop 2024-03-19T01:12:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.990+0000 7f55ae432700 1 -- 172.21.15.116:0/1814838349 shutdown_connections 2024-03-19T01:12:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.990+0000 7f55ae432700 1 --2- 172.21.15.116:0/1814838349 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a8103f80 0x7f55a810a830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.990+0000 7f55ae432700 1 --2- 172.21.15.116:0/1814838349 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55a8100600 0x7f55a8100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.990+0000 7f55ae432700 1 --2- 172.21.15.116:0/1814838349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a81035e0 0x7f55a8103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.990+0000 7f55ae432700 1 -- 172.21.15.116:0/1814838349 >> 172.21.15.116:0/1814838349 conn(0x7f55a80fbf80 msgr2=0x7f55a80fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.990+0000 7f55ae432700 1 -- 172.21.15.116:0/1814838349 shutdown_connections 2024-03-19T01:12:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.990+0000 7f55ae432700 1 -- 172.21.15.116:0/1814838349 wait complete. 2024-03-19T01:12:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55ae432700 1 Processor -- start 2024-03-19T01:12:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55ae432700 1 -- start start 2024-03-19T01:12:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55ae432700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a8100600 0x7f55a8198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:03.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55ae432700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a81035e0 0x7f55a8198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55ae432700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55a8103f80 0x7f55a819fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55ae432700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810df20 con 0x7f55a81035e0 2024-03-19T01:12:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55ae432700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810dda0 con 0x7f55a8100600 2024-03-19T01:12:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55ae432700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810e0a0 con 0x7f55a8103f80 2024-03-19T01:12:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55a77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a81035e0 0x7f55a8198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55a7fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a8100600 0x7f55a8198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55a77fe700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a81035e0 0x7f55a8198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60030/0 (socket says 172.21.15.116:60030) 2024-03-19T01:12:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55a77fe700 1 -- 172.21.15.116:0/1920742084 learned_addr learned my addr 172.21.15.116:0/1920742084 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55ac9cf700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55a8103f80 0x7f55a819fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55a77fe700 1 -- 172.21.15.116:0/1920742084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55a8103f80 msgr2=0x7f55a819fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55a77fe700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55a8103f80 0x7f55a819fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:03.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55a77fe700 1 -- 172.21.15.116:0/1920742084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a8100600 msgr2=0x7f55a8198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:03.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55a77fe700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a8100600 0x7f55a8198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:03.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55a77fe700 1 -- 172.21.15.116:0/1920742084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a81a0160 con 0x7f55a81035e0 2024-03-19T01:12:03.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.991+0000 7f55a7fff700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a8100600 0x7f55a8198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:03.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.992+0000 7f55a77fe700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a81035e0 0x7f55a8198a40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f559800e9d0 tx=0x7f559800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:03.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.992+0000 7f55a57fa700 1 -- 172.21.15.116:0/1920742084 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559800cbd0 con 0x7f55a81035e0 2024-03-19T01:12:03.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.992+0000 7f55ae432700 1 -- 172.21.15.116:0/1920742084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a81a0450 con 0x7f55a81035e0 2024-03-19T01:12:03.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.992+0000 7f55a57fa700 1 -- 172.21.15.116:0/1920742084 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f559800ee70 con 0x7f55a81035e0 2024-03-19T01:12:03.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.992+0000 7f55a57fa700 1 -- 172.21.15.116:0/1920742084 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5598018680 con 0x7f55a81035e0 2024-03-19T01:12:03.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.992+0000 7f55ae432700 1 -- 172.21.15.116:0/1920742084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a81a09e0 con 0x7f55a81035e0 2024-03-19T01:12:03.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.993+0000 7f55ae432700 1 -- 172.21.15.116:0/1920742084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a8068fd0 con 0x7f55a81035e0 2024-03-19T01:12:03.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.994+0000 7f55a57fa700 1 -- 172.21.15.116:0/1920742084 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f55980188e0 con 0x7f55a81035e0 2024-03-19T01:12:03.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.995+0000 7f55a57fa700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f55940a2910 0x7f55940a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:03.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.995+0000 7f55a7fff700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f55940a2910 0x7f55940a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:03.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.995+0000 7f55a57fa700 1 -- 172.21.15.116:0/1920742084 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f55980cee40 con 0x7f55a81035e0 2024-03-19T01:12:03.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.995+0000 7f55a7fff700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f55940a2910 0x7f55940a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f55a8069dc0 tx=0x7f5590009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:03.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:03.998+0000 7f55a57fa700 1 -- 172.21.15.116:0/1920742084 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f559809bec0 con 0x7f55a81035e0 2024-03-19T01:12:04.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.255+0000 7f55ae432700 1 -- 172.21.15.116:0/1920742084 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f55a8062750 con 0x7f55940a2910 2024-03-19T01:12:04.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.256+0000 7f55a57fa700 1 -- 172.21.15.116:0/1920742084 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f55a8062750 con 0x7f55940a2910 2024-03-19T01:12:04.257 INFO:teuthology.orchestra.run.smithi116.stdout:Backend: cephadm 2024-03-19T01:12:04.257 INFO:teuthology.orchestra.run.smithi116.stdout:Available: Yes 2024-03-19T01:12:04.257 INFO:teuthology.orchestra.run.smithi116.stdout:Paused: No 2024-03-19T01:12:04.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.259+0000 7f55ae432700 1 -- 172.21.15.116:0/1920742084 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f55940a2910 msgr2=0x7f55940a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:04.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.259+0000 7f55ae432700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f55940a2910 0x7f55940a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f55a8069dc0 tx=0x7f5590009260 comp rx=0 tx=0).stop 2024-03-19T01:12:04.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.259+0000 7f55ae432700 1 -- 172.21.15.116:0/1920742084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a81035e0 msgr2=0x7f55a8198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:04.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.259+0000 7f55ae432700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a81035e0 0x7f55a8198a40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f559800e9d0 tx=0x7f559800ece0 comp rx=0 tx=0).stop 2024-03-19T01:12:04.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.259+0000 7f55ae432700 1 -- 172.21.15.116:0/1920742084 shutdown_connections 2024-03-19T01:12:04.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.259+0000 7f55ae432700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f55940a2910 0x7f55940a4dd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:04.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.259+0000 7f55ae432700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55a8103f80 0x7f55a819fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:04.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.259+0000 7f55ae432700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55a8100600 0x7f55a8198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:04.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.259+0000 7f55ae432700 1 --2- 172.21.15.116:0/1920742084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55a81035e0 0x7f55a8198a40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:04.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.259+0000 7f55ae432700 1 -- 172.21.15.116:0/1920742084 >> 172.21.15.116:0/1920742084 conn(0x7f55a80fbf80 msgr2=0x7f55a81068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:04.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.259+0000 7f55ae432700 1 -- 172.21.15.116:0/1920742084 shutdown_connections 2024-03-19T01:12:04.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:04.259+0000 7f55ae432700 1 -- 172.21.15.116:0/1920742084 wait complete. 2024-03-19T01:12:04.928 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-19T01:12:05.175 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:12:06.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:06 smithi162 ceph-mon[67178]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:06.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:06 smithi162 ceph-mon[67178]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-19T01:12:06.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:06 smithi116 ceph-mon[62701]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:06.399 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:06 smithi116 ceph-mon[62701]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-19T01:12:06.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:06 smithi156 ceph-mon[66799]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:06.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:06 smithi156 ceph-mon[66799]: pgmap v336: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-19T01:12:07.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.057+0000 7f645aedc700 1 -- 172.21.15.116:0/1850441839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454105700 msgr2=0x7f6454107af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:07.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.057+0000 7f645aedc700 1 --2- 172.21.15.116:0/1850441839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454105700 0x7f6454107af0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f6440009990 tx=0x7f6440009ca0 comp rx=0 tx=0).stop 2024-03-19T01:12:07.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.057+0000 7f645aedc700 1 -- 172.21.15.116:0/1850441839 shutdown_connections 2024-03-19T01:12:07.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.057+0000 7f645aedc700 1 --2- 172.21.15.116:0/1850441839 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6454108030 0x7f645410a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:07.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.057+0000 7f645aedc700 1 --2- 172.21.15.116:0/1850441839 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6454102dd0 0x7f64541051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:07.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.057+0000 7f645aedc700 1 --2- 172.21.15.116:0/1850441839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454105700 0x7f6454107af0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:07.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.057+0000 7f645aedc700 1 -- 172.21.15.116:0/1850441839 >> 172.21.15.116:0/1850441839 conn(0x7f64540fc7e0 msgr2=0x7f64540fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:07.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.057+0000 7f645aedc700 1 -- 172.21.15.116:0/1850441839 shutdown_connections 2024-03-19T01:12:07.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.057+0000 7f645aedc700 1 -- 172.21.15.116:0/1850441839 wait complete. 2024-03-19T01:12:07.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f645aedc700 1 Processor -- start 2024-03-19T01:12:07.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f645aedc700 1 -- start start 2024-03-19T01:12:07.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f645aedc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6454102dd0 0x7f64541985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:07.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f645aedc700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6454105700 0x7f6454198b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:07.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f645aedc700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454108030 0x7f645419fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:07.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f645aedc700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f645410df50 con 0x7f6454108030 2024-03-19T01:12:07.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f645aedc700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f645410ddd0 con 0x7f6454102dd0 2024-03-19T01:12:07.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f645aedc700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f645410e0d0 con 0x7f6454105700 2024-03-19T01:12:07.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f6459479700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454108030 0x7f645419fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f6458c78700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6454102dd0 0x7f64541985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f6458c78700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6454102dd0 0x7f64541985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:45990/0 (socket says 172.21.15.116:45990) 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f6459479700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454108030 0x7f645419fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38594/0 (socket says 172.21.15.116:38594) 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f6458c78700 1 -- 172.21.15.116:0/3589578083 learned_addr learned my addr 172.21.15.116:0/3589578083 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.058+0000 7f6453fff700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6454105700 0x7f6454198b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.059+0000 7f6458c78700 1 -- 172.21.15.116:0/3589578083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6454105700 msgr2=0x7f6454198b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.059+0000 7f6458c78700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6454105700 0x7f6454198b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.059+0000 7f6458c78700 1 -- 172.21.15.116:0/3589578083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454108030 msgr2=0x7f645419fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.059+0000 7f6458c78700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454108030 0x7f645419fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.059+0000 7f6458c78700 1 -- 172.21.15.116:0/3589578083 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64541a0230 con 0x7f6454102dd0 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.059+0000 7f6459479700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454108030 0x7f645419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.059+0000 7f6453fff700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6454105700 0x7f6454198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.059+0000 7f6458c78700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6454102dd0 0x7f64541985d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f644800e9c0 tx=0x7f644800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.059+0000 7f6451ffb700 1 -- 172.21.15.116:0/3589578083 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644800cbd0 con 0x7f6454102dd0 2024-03-19T01:12:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.059+0000 7f645aedc700 1 -- 172.21.15.116:0/3589578083 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64541a0520 con 0x7f6454102dd0 2024-03-19T01:12:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.059+0000 7f6451ffb700 1 -- 172.21.15.116:0/3589578083 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f644800ee60 con 0x7f6454102dd0 2024-03-19T01:12:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.059+0000 7f6451ffb700 1 -- 172.21.15.116:0/3589578083 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6448005680 con 0x7f6454102dd0 2024-03-19T01:12:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.059+0000 7f645aedc700 1 -- 172.21.15.116:0/3589578083 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64541a0a60 con 0x7f6454102dd0 2024-03-19T01:12:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.060+0000 7f645aedc700 1 -- 172.21.15.116:0/3589578083 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6454068fd0 con 0x7f6454102dd0 2024-03-19T01:12:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.061+0000 7f6451ffb700 1 -- 172.21.15.116:0/3589578083 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6448004750 con 0x7f6454102dd0 2024-03-19T01:12:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.062+0000 7f6451ffb700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f64440a2890 0x7f64440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.062+0000 7f6453fff700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f64440a2890 0x7f64440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.062+0000 7f6451ffb700 1 -- 172.21.15.116:0/3589578083 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f64480ced30 con 0x7f6454102dd0 2024-03-19T01:12:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.062+0000 7f6453fff700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f64440a2890 0x7f64440a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f644000b7b0 tx=0x7f644000b880 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:07.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.071+0000 7f6451ffb700 1 -- 172.21.15.116:0/3589578083 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f644809bd50 con 0x7f6454102dd0 2024-03-19T01:12:07.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.287+0000 7f645aedc700 1 -- 172.21.15.116:0/3589578083 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f6454062750 con 0x7f64440a2890 2024-03-19T01:12:07.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.307+0000 7f6451ffb700 1 -- 172.21.15.116:0/3589578083 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f6454062750 con 0x7f64440a2890 2024-03-19T01:12:07.309 INFO:teuthology.orchestra.run.smithi116.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-19T01:12:07.309 INFO:teuthology.orchestra.run.smithi116.stdout:mds.a smithi116 running (103s) 16s ago 102s 20.9M - 19.0.0-2206-gb6ae2491 2312c8e42d00 ab58a3562cb4 2024-03-19T01:12:07.309 INFO:teuthology.orchestra.run.smithi116.stdout:mds.b smithi156 running (100s) 33s ago 99s 28.3M - 19.0.0-2206-gb6ae2491 2312c8e42d00 dfeadb187af2 2024-03-19T01:12:07.309 INFO:teuthology.orchestra.run.smithi116.stdout:mds.c smithi162 running (114s) 33s ago 114s 20.2M - 19.0.0-2206-gb6ae2491 2312c8e42d00 000c09a15f6a 2024-03-19T01:12:07.309 INFO:teuthology.orchestra.run.smithi116.stdout:mds.d smithi116 running (2m) 16s ago 2m 13.2M - 19.0.0-2206-gb6ae2491 2312c8e42d00 d66a0cad8ef2 2024-03-19T01:12:07.309 INFO:teuthology.orchestra.run.smithi116.stdout:mds.e smithi156 running (2m) 33s ago 119s 17.9M - 19.0.0-2206-gb6ae2491 2312c8e42d00 09e1cff74bc4 2024-03-19T01:12:07.309 INFO:teuthology.orchestra.run.smithi116.stdout:mds.f smithi162 running (117s) 33s ago 117s 16.1M - 19.0.0-2206-gb6ae2491 2312c8e42d00 53eec699f51f 2024-03-19T01:12:07.309 INFO:teuthology.orchestra.run.smithi116.stdout:mds.g smithi116 running (106s) 16s ago 105s 17.1M - 19.0.0-2206-gb6ae2491 2312c8e42d00 d1da46433e9c 2024-03-19T01:12:07.309 INFO:teuthology.orchestra.run.smithi116.stdout:mds.h smithi156 running (111s) 33s ago 111s 20.2M - 19.0.0-2206-gb6ae2491 2312c8e42d00 5c7c19489976 2024-03-19T01:12:07.309 INFO:teuthology.orchestra.run.smithi116.stdout:mds.i smithi162 running (108s) 33s ago 108s 19.3M - 19.0.0-2206-gb6ae2491 2312c8e42d00 d91e846e05c3 2024-03-19T01:12:07.309 INFO:teuthology.orchestra.run.smithi116.stdout:mds.j smithi116 running (2m) 16s ago 2m 14.0M - 19.0.0-2206-gb6ae2491 2312c8e42d00 cd92888c4567 2024-03-19T01:12:07.310 INFO:teuthology.orchestra.run.smithi116.stdout:mds.k smithi156 running (2m) 33s ago 2m 16.9M - 19.0.0-2206-gb6ae2491 2312c8e42d00 128f16c9a1ba 2024-03-19T01:12:07.310 INFO:teuthology.orchestra.run.smithi116.stdout:mds.l smithi162 running (2m) 33s ago 2m 18.4M - 19.0.0-2206-gb6ae2491 2312c8e42d00 e7c84b2d954c 2024-03-19T01:12:07.310 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.x smithi116 *:9283,8765 running (9m) 16s ago 9m 513M - 19.0.0-2206-gb6ae2491 2312c8e42d00 1e8e8ed90d70 2024-03-19T01:12:07.310 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.y smithi156 *:8443,8765 running (7m) 33s ago 7m 451M - 19.0.0-2206-gb6ae2491 2312c8e42d00 a73594c62eb8 2024-03-19T01:12:07.310 INFO:teuthology.orchestra.run.smithi116.stdout:mgr.z smithi162 *:8443,8765 running (7m) 33s ago 7m 453M - 19.0.0-2206-gb6ae2491 2312c8e42d00 edc477972867 2024-03-19T01:12:07.310 INFO:teuthology.orchestra.run.smithi116.stdout:mon.a smithi116 running (9m) 16s ago 9m 49.8M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 a41987fc4037 2024-03-19T01:12:07.310 INFO:teuthology.orchestra.run.smithi116.stdout:mon.b smithi156 running (7m) 33s ago 7m 39.1M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 bc093f4fc87f 2024-03-19T01:12:07.310 INFO:teuthology.orchestra.run.smithi116.stdout:mon.c smithi162 running (7m) 33s ago 7m 39.9M 2048M 19.0.0-2206-gb6ae2491 2312c8e42d00 8c9c409520b3 2024-03-19T01:12:07.310 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 smithi116 running (7m) 16s ago 7m 56.4M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 0b756cb5b23a 2024-03-19T01:12:07.310 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 smithi156 running (6m) 33s ago 6m 46.3M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 9ba4ea87209c 2024-03-19T01:12:07.310 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 smithi162 running (6m) 33s ago 6m 60.5M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 d214d261da72 2024-03-19T01:12:07.310 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 smithi116 running (5m) 16s ago 5m 53.9M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 9dd435112350 2024-03-19T01:12:07.311 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 smithi156 running (5m) 33s ago 5m 53.4M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 50fa4a6d9406 2024-03-19T01:12:07.311 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 smithi162 running (5m) 33s ago 5m 53.9M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 4da553d7f7eb 2024-03-19T01:12:07.311 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 smithi116 running (4m) 16s ago 4m 41.7M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 d04294efa9a2 2024-03-19T01:12:07.311 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 smithi156 running (4m) 33s ago 4m 53.1M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 aacc5bb63433 2024-03-19T01:12:07.311 INFO:teuthology.orchestra.run.smithi116.stdout:osd.8 smithi162 running (3m) 33s ago 3m 53.3M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 203f3c6d51be 2024-03-19T01:12:07.311 INFO:teuthology.orchestra.run.smithi116.stdout:osd.9 smithi116 running (3m) 16s ago 3m 42.0M 4272M 19.0.0-2206-gb6ae2491 2312c8e42d00 f5159dbddb5b 2024-03-19T01:12:07.311 INFO:teuthology.orchestra.run.smithi116.stdout:osd.10 smithi156 running (2m) 33s ago 2m 56.8M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 760672c2ee53 2024-03-19T01:12:07.311 INFO:teuthology.orchestra.run.smithi116.stdout:osd.11 smithi162 running (2m) 33s ago 2m 41.5M 4270M 19.0.0-2206-gb6ae2491 2312c8e42d00 80f1a29a9335 2024-03-19T01:12:07.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.309+0000 7f645aedc700 1 -- 172.21.15.116:0/3589578083 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f64440a2890 msgr2=0x7f64440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:07.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.309+0000 7f645aedc700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f64440a2890 0x7f64440a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f644000b7b0 tx=0x7f644000b880 comp rx=0 tx=0).stop 2024-03-19T01:12:07.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.309+0000 7f645aedc700 1 -- 172.21.15.116:0/3589578083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6454102dd0 msgr2=0x7f64541985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:07.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.309+0000 7f645aedc700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6454102dd0 0x7f64541985d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f644800e9c0 tx=0x7f644800ecd0 comp rx=0 tx=0).stop 2024-03-19T01:12:07.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.309+0000 7f645aedc700 1 -- 172.21.15.116:0/3589578083 shutdown_connections 2024-03-19T01:12:07.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.309+0000 7f645aedc700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454108030 0x7f645419fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:07.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.309+0000 7f645aedc700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6454102dd0 0x7f64541985d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:07.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.309+0000 7f645aedc700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f64440a2890 0x7f64440a4d50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:07.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.309+0000 7f645aedc700 1 --2- 172.21.15.116:0/3589578083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6454105700 0x7f6454198b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:07.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.309+0000 7f645aedc700 1 -- 172.21.15.116:0/3589578083 >> 172.21.15.116:0/3589578083 conn(0x7f64540fc7e0 msgr2=0x7f64541038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:07.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.309+0000 7f645aedc700 1 -- 172.21.15.116:0/3589578083 shutdown_connections 2024-03-19T01:12:07.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:07.309+0000 7f645aedc700 1 -- 172.21.15.116:0/3589578083 wait complete. 2024-03-19T01:12:08.029 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-19T01:12:08.322 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:12:08.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:08 smithi162 ceph-mon[67178]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:12:08.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:08 smithi116 ceph-mon[62701]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:12:08.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:08 smithi156 ceph-mon[66799]: pgmap v337: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-19T01:12:09.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:09 smithi162 ceph-mon[67178]: from='client.24983 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:09.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:09 smithi116 ceph-mon[62701]: from='client.24983 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:09.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:09 smithi156 ceph-mon[66799]: from='client.24983 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:10.175 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:10 smithi116 ceph-mon[62701]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-19T01:12:10.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.274+0000 7f30d7a2d700 1 -- 172.21.15.116:0/2008764460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0100c50 msgr2=0x7f30d0101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:10.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.274+0000 7f30d7a2d700 1 --2- 172.21.15.116:0/2008764460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0100c50 0x7f30d0101070 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f30cc009990 tx=0x7f30cc009ca0 comp rx=0 tx=0).stop 2024-03-19T01:12:10.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.274+0000 7f30d7a2d700 1 -- 172.21.15.116:0/2008764460 shutdown_connections 2024-03-19T01:12:10.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.274+0000 7f30d7a2d700 1 --2- 172.21.15.116:0/2008764460 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30d0101f70 0x7f30d01023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:10.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.275+0000 7f30d7a2d700 1 --2- 172.21.15.116:0/2008764460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0100c50 0x7f30d0101070 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:10.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.275+0000 7f30d7a2d700 1 --2- 172.21.15.116:0/2008764460 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30d01015b0 0x7f30d0101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:10.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.275+0000 7f30d7a2d700 1 -- 172.21.15.116:0/2008764460 >> 172.21.15.116:0/2008764460 conn(0x7f30d00fc810 msgr2=0x7f30d00fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:10.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.275+0000 7f30d7a2d700 1 -- 172.21.15.116:0/2008764460 shutdown_connections 2024-03-19T01:12:10.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.275+0000 7f30d7a2d700 1 -- 172.21.15.116:0/2008764460 wait complete. 2024-03-19T01:12:10.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.275+0000 7f30d7a2d700 1 Processor -- start 2024-03-19T01:12:10.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.275+0000 7f30d7a2d700 1 -- start start 2024-03-19T01:12:10.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d7a2d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30d0100c50 0x7f30d019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:10.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d7a2d700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30d01015b0 0x7f30d019ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:10.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d7a2d700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0101f70 0x7f30d01a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:10.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d7a2d700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f30d0110140 con 0x7f30d0101f70 2024-03-19T01:12:10.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d7a2d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f30d010ffc0 con 0x7f30d0100c50 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d7a2d700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f30d01102c0 con 0x7f30d01015b0 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d57c9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30d0100c50 0x7f30d019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d5fca700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0101f70 0x7f30d01a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d4fc8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30d01015b0 0x7f30d019ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d5fca700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0101f70 0x7f30d01a1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38608/0 (socket says 172.21.15.116:38608) 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d57c9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30d0100c50 0x7f30d019a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:46018/0 (socket says 172.21.15.116:46018) 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d5fca700 1 -- 172.21.15.116:0/3078970631 learned_addr learned my addr 172.21.15.116:0/3078970631 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d5fca700 1 -- 172.21.15.116:0/3078970631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30d01015b0 msgr2=0x7f30d019ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d5fca700 1 --2- 172.21.15.116:0/3078970631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30d01015b0 0x7f30d019ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d5fca700 1 -- 172.21.15.116:0/3078970631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30d0100c50 msgr2=0x7f30d019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d5fca700 1 --2- 172.21.15.116:0/3078970631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30d0100c50 0x7f30d019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d5fca700 1 -- 172.21.15.116:0/3078970631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d01a2380 con 0x7f30d0101f70 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d4fc8700 1 --2- 172.21.15.116:0/3078970631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30d01015b0 0x7f30d019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30d5fca700 1 --2- 172.21.15.116:0/3078970631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0101f70 0x7f30d01a1ce0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f30c800ef60 tx=0x7f30c800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30c67fc700 1 -- 172.21.15.116:0/3078970631 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c80092e0 con 0x7f30d0101f70 2024-03-19T01:12:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30c67fc700 1 -- 172.21.15.116:0/3078970631 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30c8004bb0 con 0x7f30d0101f70 2024-03-19T01:12:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.276+0000 7f30c67fc700 1 -- 172.21.15.116:0/3078970631 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8004280 con 0x7f30d0101f70 2024-03-19T01:12:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.277+0000 7f30d7a2d700 1 -- 172.21.15.116:0/3078970631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30d01a2610 con 0x7f30d0101f70 2024-03-19T01:12:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.277+0000 7f30d7a2d700 1 -- 172.21.15.116:0/3078970631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30d01a2b00 con 0x7f30d0101f70 2024-03-19T01:12:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.277+0000 7f30d7a2d700 1 -- 172.21.15.116:0/3078970631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30d0068fd0 con 0x7f30d0101f70 2024-03-19T01:12:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.278+0000 7f30c67fc700 1 -- 172.21.15.116:0/3078970631 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f30c801b030 con 0x7f30d0101f70 2024-03-19T01:12:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.279+0000 7f30c67fc700 1 --2- 172.21.15.116:0/3078970631 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f30bc0a29e0 0x7f30bc0a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.279+0000 7f30d57c9700 1 --2- 172.21.15.116:0/3078970631 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f30bc0a29e0 0x7f30bc0a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.279+0000 7f30c67fc700 1 -- 172.21.15.116:0/3078970631 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f30c80ced60 con 0x7f30d0101f70 2024-03-19T01:12:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.280+0000 7f30d57c9700 1 --2- 172.21.15.116:0/3078970631 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f30bc0a29e0 0x7f30bc0a4ea0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00b2e0 tx=0x7f30cc005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:10.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.283+0000 7f30c67fc700 1 -- 172.21.15.116:0/3078970631 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30c8022020 con 0x7f30d0101f70 2024-03-19T01:12:10.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:10 smithi162 ceph-mon[67178]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-19T01:12:10.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:10 smithi156 ceph-mon[66799]: pgmap v338: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 426 B/s wr, 12 op/s 2024-03-19T01:12:10.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.494+0000 7f30d7a2d700 1 -- 172.21.15.116:0/3078970631 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f30d0062750 con 0x7f30bc0a29e0 2024-03-19T01:12:10.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.500+0000 7f30c67fc700 1 -- 172.21.15.116:0/3078970631 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f30d0062750 con 0x7f30bc0a29e0 2024-03-19T01:12:10.502 INFO:teuthology.orchestra.run.smithi116.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-19T01:12:10.502 INFO:teuthology.orchestra.run.smithi116.stdout:mds.all 12/12 36s ago 2m smithi116=a;smithi116=d;smithi116=g;smithi116=j;smithi156=b;smithi156=e;smithi156=h;smithi156=k;smithi162=c;smithi162=f;smithi162=i;smithi162=l;count:12 2024-03-19T01:12:10.502 INFO:teuthology.orchestra.run.smithi116.stdout:mgr 3/3 36s ago 7m smithi116=x;smithi156=y;smithi162=z;count:3 2024-03-19T01:12:10.502 INFO:teuthology.orchestra.run.smithi116.stdout:mon 3/3 36s ago 8m smithi116:172.21.15.116=a;smithi156:172.21.15.156=b;smithi162:172.21.15.162=c;count:3 2024-03-19T01:12:10.503 INFO:teuthology.orchestra.run.smithi116.stdout:osd 12 36s ago - 2024-03-19T01:12:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.503+0000 7f30d7a2d700 1 -- 172.21.15.116:0/3078970631 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f30bc0a29e0 msgr2=0x7f30bc0a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.503+0000 7f30d7a2d700 1 --2- 172.21.15.116:0/3078970631 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f30bc0a29e0 0x7f30bc0a4ea0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00b2e0 tx=0x7f30cc005fd0 comp rx=0 tx=0).stop 2024-03-19T01:12:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.503+0000 7f30d7a2d700 1 -- 172.21.15.116:0/3078970631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0101f70 msgr2=0x7f30d01a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.503+0000 7f30d7a2d700 1 --2- 172.21.15.116:0/3078970631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0101f70 0x7f30d01a1ce0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f30c800ef60 tx=0x7f30c800ef90 comp rx=0 tx=0).stop 2024-03-19T01:12:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.503+0000 7f30d7a2d700 1 -- 172.21.15.116:0/3078970631 shutdown_connections 2024-03-19T01:12:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.503+0000 7f30d7a2d700 1 --2- 172.21.15.116:0/3078970631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0101f70 0x7f30d01a1ce0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:10.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.503+0000 7f30d7a2d700 1 --2- 172.21.15.116:0/3078970631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f30d0100c50 0x7f30d019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.503+0000 7f30d7a2d700 1 --2- 172.21.15.116:0/3078970631 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f30bc0a29e0 0x7f30bc0a4ea0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.503+0000 7f30d7a2d700 1 --2- 172.21.15.116:0/3078970631 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f30d01015b0 0x7f30d019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.503+0000 7f30d7a2d700 1 -- 172.21.15.116:0/3078970631 >> 172.21.15.116:0/3078970631 conn(0x7f30d00fc810 msgr2=0x7f30d00fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.503+0000 7f30d7a2d700 1 -- 172.21.15.116:0/3078970631 shutdown_connections 2024-03-19T01:12:10.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:10.503+0000 7f30d7a2d700 1 -- 172.21.15.116:0/3078970631 wait complete. 2024-03-19T01:12:11.138 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-19T01:12:11.393 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:12:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:12 smithi162 ceph-mon[67178]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:12 smithi162 ceph-mon[67178]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-19T01:12:12.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:12 smithi116 ceph-mon[62701]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:12.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:12 smithi116 ceph-mon[62701]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-19T01:12:12.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:12 smithi156 ceph-mon[66799]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:12.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:12 smithi156 ceph-mon[66799]: pgmap v339: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-19T01:12:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.163+0000 7f72a615f700 1 -- 172.21.15.116:0/1695022596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0102430 msgr2=0x7f72a0102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.163+0000 7f72a615f700 1 --2- 172.21.15.116:0/1695022596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0102430 0x7f72a0102890 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f72880099c0 tx=0x7f7288009cd0 comp rx=0 tx=0).stop 2024-03-19T01:12:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.163+0000 7f72a615f700 1 -- 172.21.15.116:0/1695022596 shutdown_connections 2024-03-19T01:12:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.163+0000 7f72a615f700 1 --2- 172.21.15.116:0/1695022596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72a0102dd0 0x7f72a0109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.163+0000 7f72a615f700 1 --2- 172.21.15.116:0/1695022596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a0101230 0x7f72a0101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.163+0000 7f72a615f700 1 --2- 172.21.15.116:0/1695022596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0102430 0x7f72a0102890 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:13.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.163+0000 7f72a615f700 1 -- 172.21.15.116:0/1695022596 >> 172.21.15.116:0/1695022596 conn(0x7f72a00fc7d0 msgr2=0x7f72a00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.163+0000 7f72a615f700 1 -- 172.21.15.116:0/1695022596 shutdown_connections 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.163+0000 7f72a615f700 1 -- 172.21.15.116:0/1695022596 wait complete. 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.164+0000 7f72a615f700 1 Processor -- start 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.164+0000 7f72a615f700 1 -- start start 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.164+0000 7f72a615f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a0101230 0x7f72a0105360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.164+0000 7f72a615f700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0102430 0x7f72a01039b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.164+0000 7f72a615f700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72a0102dd0 0x7f72a0103ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.164+0000 7f72a615f700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f72a010ce10 con 0x7f72a0102430 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.164+0000 7f72a615f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f72a010cc90 con 0x7f72a0101230 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.164+0000 7f72a615f700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f72a010cf90 con 0x7f72a0102dd0 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.164+0000 7f7297fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0102430 0x7f72a01039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.164+0000 7f729f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a0101230 0x7f72a0105360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.164+0000 7f7297fff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0102430 0x7f72a01039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38626/0 (socket says 172.21.15.116:38626) 2024-03-19T01:12:13.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.164+0000 7f7297fff700 1 -- 172.21.15.116:0/480653408 learned_addr learned my addr 172.21.15.116:0/480653408 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.164+0000 7f729ffff700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72a0102dd0 0x7f72a0103ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.165+0000 7f729f7fe700 1 -- 172.21.15.116:0/480653408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72a0102dd0 msgr2=0x7f72a0103ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.165+0000 7f729f7fe700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72a0102dd0 0x7f72a0103ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.165+0000 7f729f7fe700 1 -- 172.21.15.116:0/480653408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0102430 msgr2=0x7f72a01039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.165+0000 7f729f7fe700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0102430 0x7f72a01039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.165+0000 7f729f7fe700 1 -- 172.21.15.116:0/480653408 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a01047a0 con 0x7f72a0101230 2024-03-19T01:12:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.165+0000 7f7297fff700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0102430 0x7f72a01039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.165+0000 7f729f7fe700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a0101230 0x7f72a0105360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f729000e9c0 tx=0x7f729000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.165+0000 7f729d7fa700 1 -- 172.21.15.116:0/480653408 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290004220 con 0x7f72a0101230 2024-03-19T01:12:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.165+0000 7f72a615f700 1 -- 172.21.15.116:0/480653408 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a01a26a0 con 0x7f72a0101230 2024-03-19T01:12:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.165+0000 7f729d7fa700 1 -- 172.21.15.116:0/480653408 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72900043c0 con 0x7f72a0101230 2024-03-19T01:12:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.165+0000 7f729d7fa700 1 -- 172.21.15.116:0/480653408 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7290003cc0 con 0x7f72a0101230 2024-03-19T01:12:13.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.165+0000 7f72a615f700 1 -- 172.21.15.116:0/480653408 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a01a2bb0 con 0x7f72a0101230 2024-03-19T01:12:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.166+0000 7f72a615f700 1 -- 172.21.15.116:0/480653408 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72a0068fd0 con 0x7f72a0101230 2024-03-19T01:12:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.167+0000 7f729d7fa700 1 -- 172.21.15.116:0/480653408 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f729000c7c0 con 0x7f72a0101230 2024-03-19T01:12:13.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.167+0000 7f729d7fa700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f728c0a2890 0x7f728c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:13.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.168+0000 7f7297fff700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f728c0a2890 0x7f728c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:13.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.168+0000 7f729d7fa700 1 -- 172.21.15.116:0/480653408 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f72900cece0 con 0x7f72a0101230 2024-03-19T01:12:13.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.168+0000 7f7297fff700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f728c0a2890 0x7f728c0a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7288003790 tx=0x7f7288003860 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:13.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.177+0000 7f729d7fa700 1 -- 172.21.15.116:0/480653408 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f729009bd60 con 0x7f72a0101230 2024-03-19T01:12:13.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.388+0000 7f72a615f700 1 -- 172.21.15.116:0/480653408 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f72a006af60 con 0x7f728c0a2890 2024-03-19T01:12:13.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.389+0000 7f729d7fa700 1 -- 172.21.15.116:0/480653408 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f72a006af60 con 0x7f728c0a2890 2024-03-19T01:12:13.391 INFO:teuthology.orchestra.run.smithi116.stdout:HOST ADDR LABELS STATUS 2024-03-19T01:12:13.391 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116 172.21.15.116 2024-03-19T01:12:13.391 INFO:teuthology.orchestra.run.smithi116.stdout:smithi156 172.21.15.156 2024-03-19T01:12:13.391 INFO:teuthology.orchestra.run.smithi116.stdout:smithi162 172.21.15.162 2024-03-19T01:12:13.392 INFO:teuthology.orchestra.run.smithi116.stdout:3 hosts in cluster 2024-03-19T01:12:13.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.392+0000 7f72a615f700 1 -- 172.21.15.116:0/480653408 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f728c0a2890 msgr2=0x7f728c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:13.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.392+0000 7f72a615f700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f728c0a2890 0x7f728c0a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7288003790 tx=0x7f7288003860 comp rx=0 tx=0).stop 2024-03-19T01:12:13.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.392+0000 7f72a615f700 1 -- 172.21.15.116:0/480653408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a0101230 msgr2=0x7f72a0105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:13.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.392+0000 7f72a615f700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a0101230 0x7f72a0105360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f729000e9c0 tx=0x7f729000ecd0 comp rx=0 tx=0).stop 2024-03-19T01:12:13.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.392+0000 7f72a615f700 1 -- 172.21.15.116:0/480653408 shutdown_connections 2024-03-19T01:12:13.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.392+0000 7f72a615f700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f728c0a2890 0x7f728c0a4d50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:13.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.392+0000 7f72a615f700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f72a0102dd0 0x7f72a0103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:13.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.392+0000 7f72a615f700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f72a0101230 0x7f72a0105360 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:13.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.392+0000 7f72a615f700 1 --2- 172.21.15.116:0/480653408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72a0102430 0x7f72a01039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:13.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.392+0000 7f72a615f700 1 -- 172.21.15.116:0/480653408 >> 172.21.15.116:0/480653408 conn(0x7f72a00fc7d0 msgr2=0x7f72a006a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:13.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.392+0000 7f72a615f700 1 -- 172.21.15.116:0/480653408 shutdown_connections 2024-03-19T01:12:13.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:13.393+0000 7f72a615f700 1 -- 172.21.15.116:0/480653408 wait complete. 2024-03-19T01:12:14.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-19T01:12:14.366 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:12:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:14 smithi162 ceph-mon[67178]: pgmap v340: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:12:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:14 smithi116 ceph-mon[62701]: pgmap v340: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:12:14.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:14 smithi156 ceph-mon[66799]: pgmap v340: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:12:15.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:15 smithi162 ceph-mon[67178]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:15.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:15 smithi116 ceph-mon[62701]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:15.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:15 smithi156 ceph-mon[66799]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:15.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.984+0000 7f18e6fd4700 1 -- 172.21.15.116:0/1464082364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18e01056f0 msgr2=0x7f18e0107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:15.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.984+0000 7f18e6fd4700 1 --2- 172.21.15.116:0/1464082364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18e01056f0 0x7f18e0107ae0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009990 tx=0x7f18cc009ca0 comp rx=0 tx=0).stop 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 -- 172.21.15.116:0/1464082364 shutdown_connections 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 --2- 172.21.15.116:0/1464082364 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18e0108020 0x7f18e010a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 --2- 172.21.15.116:0/1464082364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18e0102dc0 0x7f18e01051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 --2- 172.21.15.116:0/1464082364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18e01056f0 0x7f18e0107ae0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 -- 172.21.15.116:0/1464082364 >> 172.21.15.116:0/1464082364 conn(0x7f18e00fc7d0 msgr2=0x7f18e00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 -- 172.21.15.116:0/1464082364 shutdown_connections 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 -- 172.21.15.116:0/1464082364 wait complete. 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 Processor -- start 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 -- start start 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18e0102dc0 0x7f18e0198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18e01056f0 0x7f18e0198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18e0108020 0x7f18e019fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f18e010df40 con 0x7f18e0108020 2024-03-19T01:12:15.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f18e010ddc0 con 0x7f18e0102dc0 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.985+0000 7f18e6fd4700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f18e010e0c0 con 0x7f18e01056f0 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18e5571700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18e0108020 0x7f18e019fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18e4d70700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18e0102dc0 0x7f18e0198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18dffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18e01056f0 0x7f18e0198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18e5571700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18e0108020 0x7f18e019fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38642/0 (socket says 172.21.15.116:38642) 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18dffff700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18e01056f0 0x7f18e0198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54060/0 (socket says 172.21.15.116:54060) 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18e5571700 1 -- 172.21.15.116:0/222200027 learned_addr learned my addr 172.21.15.116:0/222200027 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18e4d70700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18e0102dc0 0x7f18e0198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:46074/0 (socket says 172.21.15.116:46074) 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18dffff700 1 -- 172.21.15.116:0/222200027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18e0102dc0 msgr2=0x7f18e0198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18dffff700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18e0102dc0 0x7f18e0198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18dffff700 1 -- 172.21.15.116:0/222200027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18e0108020 msgr2=0x7f18e019fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18dffff700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18e0108020 0x7f18e019fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18dffff700 1 -- 172.21.15.116:0/222200027 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18e01a0180 con 0x7f18e01056f0 2024-03-19T01:12:15.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18e5571700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18e0108020 0x7f18e019fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:15.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18e4d70700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18e0102dc0 0x7f18e0198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:15.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18dffff700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18e01056f0 0x7f18e0198ab0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f18cc000c00 tx=0x7f18cc0057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:15.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18ddffb700 1 -- 172.21.15.116:0/222200027 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18cc02c070 con 0x7f18e01056f0 2024-03-19T01:12:15.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18e6fd4700 1 -- 172.21.15.116:0/222200027 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18e0100bb0 con 0x7f18e01056f0 2024-03-19T01:12:15.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18ddffb700 1 -- 172.21.15.116:0/222200027 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18cc00bcb0 con 0x7f18e01056f0 2024-03-19T01:12:15.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.986+0000 7f18ddffb700 1 -- 172.21.15.116:0/222200027 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18cc004230 con 0x7f18e01056f0 2024-03-19T01:12:15.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.987+0000 7f18e6fd4700 1 -- 172.21.15.116:0/222200027 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18e01010e0 con 0x7f18e01056f0 2024-03-19T01:12:15.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.987+0000 7f18e6fd4700 1 -- 172.21.15.116:0/222200027 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18e0068fd0 con 0x7f18e01056f0 2024-03-19T01:12:15.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.989+0000 7f18ddffb700 1 -- 172.21.15.116:0/222200027 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f18cc02f070 con 0x7f18e01056f0 2024-03-19T01:12:15.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.989+0000 7f18ddffb700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f18d00a2890 0x7f18d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:15.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.989+0000 7f18e4d70700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f18d00a2890 0x7f18d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:15.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.989+0000 7f18ddffb700 1 -- 172.21.15.116:0/222200027 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f18cc0d43a0 con 0x7f18e01056f0 2024-03-19T01:12:15.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.990+0000 7f18e4d70700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f18d00a2890 0x7f18d00a4d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f18d4010420 tx=0x7f18d40073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:15.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:15.993+0000 7f18ddffb700 1 -- 172.21.15.116:0/222200027 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f18cc036020 con 0x7f18e01056f0 2024-03-19T01:12:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.211+0000 7f18e6fd4700 1 -- 172.21.15.116:0/222200027 --> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f18e0062750 con 0x7f18d00a2890 2024-03-19T01:12:16.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.213+0000 7f18ddffb700 1 -- 172.21.15.116:0/222200027 <== mgr.14150 v2:172.21.15.116:6800/2389438183 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f18e0062750 con 0x7f18d00a2890 2024-03-19T01:12:16.215 INFO:teuthology.orchestra.run.smithi116.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-19T01:12:16.215 INFO:teuthology.orchestra.run.smithi116.stdout:smithi116 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VJ400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-19T01:12:16.215 INFO:teuthology.orchestra.run.smithi116.stdout:smithi156 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB74760040480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-03-19T01:12:16.215 INFO:teuthology.orchestra.run.smithi116.stdout:smithi162 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300JU400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-19T01:12:16.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.215+0000 7f18e6fd4700 1 -- 172.21.15.116:0/222200027 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f18d00a2890 msgr2=0x7f18d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:16.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.215+0000 7f18e6fd4700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f18d00a2890 0x7f18d00a4d50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f18d4010420 tx=0x7f18d40073d0 comp rx=0 tx=0).stop 2024-03-19T01:12:16.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.215+0000 7f18e6fd4700 1 -- 172.21.15.116:0/222200027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18e01056f0 msgr2=0x7f18e0198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:16.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.215+0000 7f18e6fd4700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18e01056f0 0x7f18e0198ab0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f18cc000c00 tx=0x7f18cc0057c0 comp rx=0 tx=0).stop 2024-03-19T01:12:16.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.215+0000 7f18e6fd4700 1 -- 172.21.15.116:0/222200027 shutdown_connections 2024-03-19T01:12:16.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.215+0000 7f18e6fd4700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18e0108020 0x7f18e019fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:16.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.215+0000 7f18e6fd4700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18e0102dc0 0x7f18e0198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:16.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.215+0000 7f18e6fd4700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f18d00a2890 0x7f18d00a4d50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:16.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.215+0000 7f18e6fd4700 1 --2- 172.21.15.116:0/222200027 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18e01056f0 0x7f18e0198ab0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:16.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.215+0000 7f18e6fd4700 1 -- 172.21.15.116:0/222200027 >> 172.21.15.116:0/222200027 conn(0x7f18e00fc7d0 msgr2=0x7f18e0103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:16.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.215+0000 7f18e6fd4700 1 -- 172.21.15.116:0/222200027 shutdown_connections 2024-03-19T01:12:16.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:16.215+0000 7f18e6fd4700 1 -- 172.21.15.116:0/222200027 wait complete. 2024-03-19T01:12:16.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:16 smithi116 ceph-mon[62701]: pgmap v341: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:12:16.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:16 smithi156 ceph-mon[66799]: pgmap v341: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:12:16.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:16 smithi162 ceph-mon[67178]: pgmap v341: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:12:16.932 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-19T01:12:16.942 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T01:12:16.942 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-19T01:12:17.189 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:12:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:18 smithi116 ceph-mon[62701]: from='client.24979 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:18.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:18 smithi116 ceph-mon[62701]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:12:18.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:18 smithi156 ceph-mon[66799]: from='client.24979 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:18.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:18 smithi156 ceph-mon[66799]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:12:18.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:18 smithi162 ceph-mon[67178]: from='client.24979 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-19T01:12:18.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:18 smithi162 ceph-mon[67178]: pgmap v342: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:12:18.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.985+0000 7f6b743b8700 1 -- 172.21.15.116:0/882790438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b6c103f80 msgr2=0x7f6b6c10a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:18.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.985+0000 7f6b743b8700 1 --2- 172.21.15.116:0/882790438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b6c103f80 0x7f6b6c10a830 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6b68009990 tx=0x7f6b68009ca0 comp rx=0 tx=0).stop 2024-03-19T01:12:18.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 -- 172.21.15.116:0/882790438 shutdown_connections 2024-03-19T01:12:18.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 --2- 172.21.15.116:0/882790438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b6c103f80 0x7f6b6c10a830 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:18.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 --2- 172.21.15.116:0/882790438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b6c100600 0x7f6b6c100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:18.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 --2- 172.21.15.116:0/882790438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b6c1035e0 0x7f6b6c103a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:18.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 -- 172.21.15.116:0/882790438 >> 172.21.15.116:0/882790438 conn(0x7f6b6c0fbf80 msgr2=0x7f6b6c0fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:18.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 -- 172.21.15.116:0/882790438 shutdown_connections 2024-03-19T01:12:18.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 -- 172.21.15.116:0/882790438 wait complete. 2024-03-19T01:12:18.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 Processor -- start 2024-03-19T01:12:18.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 -- start start 2024-03-19T01:12:18.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b6c100600 0x7f6b6c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:18.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b6c1035e0 0x7f6b6c19ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:18.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b6c103f80 0x7f6b6c1a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:18.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c10df00 con 0x7f6b6c100600 2024-03-19T01:12:18.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c10dd80 con 0x7f6b6c103f80 2024-03-19T01:12:18.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b743b8700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c10e080 con 0x7f6b6c1035e0 2024-03-19T01:12:18.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.986+0000 7f6b72154700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b6c100600 0x7f6b6c19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:18.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.987+0000 7f6b72154700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b6c100600 0x7f6b6c19a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35112/0 (socket says 172.21.15.116:35112) 2024-03-19T01:12:18.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.987+0000 7f6b72154700 1 -- 172.21.15.116:0/1626096084 learned_addr learned my addr 172.21.15.116:0/1626096084 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:18.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.987+0000 7f6b72955700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b6c103f80 0x7f6b6c1a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:18.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.987+0000 7f6b71953700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b6c1035e0 0x7f6b6c19ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:18.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.987+0000 7f6b72154700 1 -- 172.21.15.116:0/1626096084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b6c1035e0 msgr2=0x7f6b6c19ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:18.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.987+0000 7f6b72154700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b6c1035e0 0x7f6b6c19ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:18.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.987+0000 7f6b72154700 1 -- 172.21.15.116:0/1626096084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b6c103f80 msgr2=0x7f6b6c1a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:18.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.987+0000 7f6b72154700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b6c103f80 0x7f6b6c1a1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:18.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.987+0000 7f6b72154700 1 -- 172.21.15.116:0/1626096084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6c1a2380 con 0x7f6b6c100600 2024-03-19T01:12:18.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.987+0000 7f6b72154700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b6c100600 0x7f6b6c19a720 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6b64009f40 tx=0x7f6b6400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:18.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.987+0000 7f6b637fe700 1 -- 172.21.15.116:0/1626096084 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b640092e0 con 0x7f6b6c100600 2024-03-19T01:12:18.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.987+0000 7f6b637fe700 1 -- 172.21.15.116:0/1626096084 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b6400c9c0 con 0x7f6b6c100600 2024-03-19T01:12:18.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.987+0000 7f6b637fe700 1 -- 172.21.15.116:0/1626096084 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b64004580 con 0x7f6b6c100600 2024-03-19T01:12:18.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.988+0000 7f6b743b8700 1 -- 172.21.15.116:0/1626096084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b6c1a2610 con 0x7f6b6c100600 2024-03-19T01:12:18.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.988+0000 7f6b743b8700 1 -- 172.21.15.116:0/1626096084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b6c1a2ad0 con 0x7f6b6c100600 2024-03-19T01:12:18.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.989+0000 7f6b743b8700 1 -- 172.21.15.116:0/1626096084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b6c068fd0 con 0x7f6b6c100600 2024-03-19T01:12:18.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.990+0000 7f6b637fe700 1 -- 172.21.15.116:0/1626096084 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6b6401b070 con 0x7f6b6c100600 2024-03-19T01:12:18.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.990+0000 7f6b637fe700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b580a2910 0x7f6b580a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:18.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.990+0000 7f6b71953700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b580a2910 0x7f6b580a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:18.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.991+0000 7f6b637fe700 1 -- 172.21.15.116:0/1626096084 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6b640ce420 con 0x7f6b6c100600 2024-03-19T01:12:18.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.991+0000 7f6b71953700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b580a2910 0x7f6b580a4dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c009c60 tx=0x7f6b5c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:18.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:18.995+0000 7f6b637fe700 1 -- 172.21.15.116:0/1626096084 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6b64010040 con 0x7f6b6c100600 2024-03-19T01:12:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.244+0000 7f6b743b8700 1 -- 172.21.15.116:0/1626096084 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f6b6c1a3540 con 0x7f6b6c100600 2024-03-19T01:12:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.245+0000 7f6b637fe700 1 -- 172.21.15.116:0/1626096084 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3094 (secure 0 0 0) 0x7f6b6409b4a0 con 0x7f6b6c100600 2024-03-19T01:12:19.246 INFO:teuthology.orchestra.run.smithi116.stdout:e24 2024-03-19T01:12:19.246 INFO:teuthology.orchestra.run.smithi116.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-19T01:12:19.246 INFO:teuthology.orchestra.run.smithi116.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T01:12:19.246 INFO:teuthology.orchestra.run.smithi116.stdout:legacy client fscid: 1 2024-03-19T01:12:19.246 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:19.246 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem 'cephfs' (1) 2024-03-19T01:12:19.246 INFO:teuthology.orchestra.run.smithi116.stdout:fs_name cephfs 2024-03-19T01:12:19.247 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 24 2024-03-19T01:12:19.247 INFO:teuthology.orchestra.run.smithi116.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-19T01:12:19.247 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-19T01:10:32.487607+0000 2024-03-19T01:12:19.247 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-19T01:10:38.982004+0000 2024-03-19T01:12:19.247 INFO:teuthology.orchestra.run.smithi116.stdout:tableserver 0 2024-03-19T01:12:19.247 INFO:teuthology.orchestra.run.smithi116.stdout:root 0 2024-03-19T01:12:19.247 INFO:teuthology.orchestra.run.smithi116.stdout:session_timeout 300 2024-03-19T01:12:19.247 INFO:teuthology.orchestra.run.smithi116.stdout:session_autoclose 300 2024-03-19T01:12:19.247 INFO:teuthology.orchestra.run.smithi116.stdout:max_file_size 1099511627776 2024-03-19T01:12:19.247 INFO:teuthology.orchestra.run.smithi116.stdout:max_xattr_size 65536 2024-03-19T01:12:19.247 INFO:teuthology.orchestra.run.smithi116.stdout:required_client_features {} 2024-03-19T01:12:19.248 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure 0 2024-03-19T01:12:19.248 INFO:teuthology.orchestra.run.smithi116.stdout:last_failure_osd_epoch 0 2024-03-19T01:12:19.248 INFO:teuthology.orchestra.run.smithi116.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-19T01:12:19.248 INFO:teuthology.orchestra.run.smithi116.stdout:max_mds 5 2024-03-19T01:12:19.248 INFO:teuthology.orchestra.run.smithi116.stdout:in 0,1,2,3,4 2024-03-19T01:12:19.248 INFO:teuthology.orchestra.run.smithi116.stdout:up {0=24500,1=24488,2=24473,3=24460,4=24445} 2024-03-19T01:12:19.248 INFO:teuthology.orchestra.run.smithi116.stdout:failed 2024-03-19T01:12:19.248 INFO:teuthology.orchestra.run.smithi116.stdout:damaged 2024-03-19T01:12:19.248 INFO:teuthology.orchestra.run.smithi116.stdout:stopped 2024-03-19T01:12:19.248 INFO:teuthology.orchestra.run.smithi116.stdout:data_pools [3] 2024-03-19T01:12:19.248 INFO:teuthology.orchestra.run.smithi116.stdout:metadata_pool 2 2024-03-19T01:12:19.248 INFO:teuthology.orchestra.run.smithi116.stdout:inline_data disabled 2024-03-19T01:12:19.249 INFO:teuthology.orchestra.run.smithi116.stdout:balancer 2024-03-19T01:12:19.249 INFO:teuthology.orchestra.run.smithi116.stdout:bal_rank_mask -1 2024-03-19T01:12:19.249 INFO:teuthology.orchestra.run.smithi116.stdout:standby_count_wanted 0 2024-03-19T01:12:19.249 INFO:teuthology.orchestra.run.smithi116.stdout:qdb_cluster leader: 24500 members: 24473,24445,24500,24460,24488 2024-03-19T01:12:19.249 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b{0:24500} state up:active seq 3 addr [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T01:12:19.249 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.e{0:24434} state up:standby-replay seq 1 addr [v2:172.21.15.156:6834/1738017780,v1:172.21.15.156:6835/1738017780] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T01:12:19.249 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a{1:24488} state up:active seq 4 addr [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T01:12:19.249 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.f{1:24427} state up:standby-replay seq 1 addr [v2:172.21.15.162:6834/1592939032,v1:172.21.15.162:6835/1592939032] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T01:12:19.249 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.h{2:24473} state up:active seq 7 addr [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T01:12:19.249 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.k{2:24410} state up:standby-replay seq 1 addr [v2:172.21.15.156:6832/1647822466,v1:172.21.15.156:6833/1647822466] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T01:12:19.249 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.i{3:24460} state up:active seq 6 addr [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T01:12:19.249 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.l{3:24394} state up:standby-replay seq 1 addr [v2:172.21.15.162:6832/3113382416,v1:172.21.15.162:6833/3113382416] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T01:12:19.250 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c{4:24445} state up:active seq 8 addr [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T01:12:19.250 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.g{4:14613} state up:standby-replay seq 1 addr [v2:172.21.15.116:6838/1242639081,v1:172.21.15.116:6839/1242639081] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T01:12:19.250 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:19.250 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:19.250 INFO:teuthology.orchestra.run.smithi116.stdout:Standby daemons: 2024-03-19T01:12:19.250 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:19.250 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.j{-1:14535} state up:standby seq 1 addr [v2:172.21.15.116:6834/909235,v1:172.21.15.116:6835/909235] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T01:12:19.250 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d{-1:14547} state up:standby seq 1 addr [v2:172.21.15.116:6836/3764874477,v1:172.21.15.116:6837/3764874477] compat {c=[1],r=[1],i=[1fff]}] 2024-03-19T01:12:19.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.247+0000 7f6b743b8700 1 -- 172.21.15.116:0/1626096084 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b580a2910 msgr2=0x7f6b580a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:19.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.247+0000 7f6b743b8700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b580a2910 0x7f6b580a4dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c009c60 tx=0x7f6b5c009260 comp rx=0 tx=0).stop 2024-03-19T01:12:19.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.247+0000 7f6b743b8700 1 -- 172.21.15.116:0/1626096084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b6c100600 msgr2=0x7f6b6c19a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:19.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.247+0000 7f6b743b8700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b6c100600 0x7f6b6c19a720 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6b64009f40 tx=0x7f6b6400c3a0 comp rx=0 tx=0).stop 2024-03-19T01:12:19.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.247+0000 7f6b743b8700 1 -- 172.21.15.116:0/1626096084 shutdown_connections 2024-03-19T01:12:19.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.247+0000 7f6b743b8700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b6c103f80 0x7f6b6c1a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:19.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.247+0000 7f6b743b8700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b6c100600 0x7f6b6c19a720 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:19.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.247+0000 7f6b743b8700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b580a2910 0x7f6b580a4dd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:19.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.247+0000 7f6b743b8700 1 --2- 172.21.15.116:0/1626096084 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b6c1035e0 0x7f6b6c19ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:19.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.247+0000 7f6b743b8700 1 -- 172.21.15.116:0/1626096084 >> 172.21.15.116:0/1626096084 conn(0x7f6b6c0fbf80 msgr2=0x7f6b6c1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:19.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.247+0000 7f6b743b8700 1 -- 172.21.15.116:0/1626096084 shutdown_connections 2024-03-19T01:12:19.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:19.247+0000 7f6b743b8700 1 -- 172.21.15.116:0/1626096084 wait complete. 2024-03-19T01:12:19.252 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2024-03-19T01:12:19.967 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-19T01:12:20.220 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:12:20.345 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:20 smithi116 ceph-mon[62701]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:12:20.345 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:20 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1626096084' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T01:12:20.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:20 smithi156 ceph-mon[66799]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:12:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:20 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1626096084' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T01:12:20.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:20 smithi162 ceph-mon[67178]: pgmap v343: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:12:20.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:20 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1626096084' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-19T01:12:22.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.046+0000 7fa9838bf700 1 -- 172.21.15.116:0/2247301977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa97c101240 msgr2=0x7fa97c101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:22.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.046+0000 7fa9838bf700 1 --2- 172.21.15.116:0/2247301977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa97c101240 0x7fa97c101660 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa9780099c0 tx=0x7fa978009cd0 comp rx=0 tx=0).stop 2024-03-19T01:12:22.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.046+0000 7fa9838bf700 1 -- 172.21.15.116:0/2247301977 shutdown_connections 2024-03-19T01:12:22.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.046+0000 7fa9838bf700 1 --2- 172.21.15.116:0/2247301977 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa97c102de0 0x7fa97c109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:22.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.046+0000 7fa9838bf700 1 --2- 172.21.15.116:0/2247301977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa97c101240 0x7fa97c101660 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:22.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.046+0000 7fa9838bf700 1 --2- 172.21.15.116:0/2247301977 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa97c102440 0x7fa97c1028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:22.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.046+0000 7fa9838bf700 1 -- 172.21.15.116:0/2247301977 >> 172.21.15.116:0/2247301977 conn(0x7fa97c0fc7e0 msgr2=0x7fa97c0fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:22.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.046+0000 7fa9838bf700 1 -- 172.21.15.116:0/2247301977 shutdown_connections 2024-03-19T01:12:22.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.046+0000 7fa9838bf700 1 -- 172.21.15.116:0/2247301977 wait complete. 2024-03-19T01:12:22.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.047+0000 7fa9838bf700 1 Processor -- start 2024-03-19T01:12:22.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.047+0000 7fa9838bf700 1 -- start start 2024-03-19T01:12:22.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.047+0000 7fa9838bf700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa97c101240 0x7fa97c197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:22.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.047+0000 7fa9838bf700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa97c102440 0x7fa97c197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:22.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.047+0000 7fa9838bf700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa97c102de0 0x7fa97c19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.047+0000 7fa9838bf700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c10ce50 con 0x7fa97c102440 2024-03-19T01:12:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.047+0000 7fa9838bf700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c10ccd0 con 0x7fa97c101240 2024-03-19T01:12:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.047+0000 7fa9838bf700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa97c10cfd0 con 0x7fa97c102de0 2024-03-19T01:12:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.047+0000 7fa980e5a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa97c102440 0x7fa97c197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa98165b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa97c101240 0x7fa97c197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa980e5a700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa97c102440 0x7fa97c197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35126/0 (socket says 172.21.15.116:35126) 2024-03-19T01:12:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa980e5a700 1 -- 172.21.15.116:0/403352387 learned_addr learned my addr 172.21.15.116:0/403352387 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa981e5c700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa97c102de0 0x7fa97c19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa98165b700 1 -- 172.21.15.116:0/403352387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa97c102de0 msgr2=0x7fa97c19ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa98165b700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa97c102de0 0x7fa97c19ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa98165b700 1 -- 172.21.15.116:0/403352387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa97c102440 msgr2=0x7fa97c197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa98165b700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa97c102440 0x7fa97c197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:22.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa98165b700 1 -- 172.21.15.116:0/403352387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa97c19f170 con 0x7fa97c101240 2024-03-19T01:12:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa981e5c700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa97c102de0 0x7fa97c19ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa98165b700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa97c101240 0x7fa97c197510 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa978009990 tx=0x7fa97800f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa9727fc700 1 -- 172.21.15.116:0/403352387 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa978009e60 con 0x7fa97c101240 2024-03-19T01:12:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa9838bf700 1 -- 172.21.15.116:0/403352387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa97c19f400 con 0x7fa97c101240 2024-03-19T01:12:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa9727fc700 1 -- 172.21.15.116:0/403352387 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa978003960 con 0x7fa97c101240 2024-03-19T01:12:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa9727fc700 1 -- 172.21.15.116:0/403352387 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa978005640 con 0x7fa97c101240 2024-03-19T01:12:22.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.048+0000 7fa9838bf700 1 -- 172.21.15.116:0/403352387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa97c19f8b0 con 0x7fa97c101240 2024-03-19T01:12:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.049+0000 7fa9838bf700 1 -- 172.21.15.116:0/403352387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa97c068fd0 con 0x7fa97c101240 2024-03-19T01:12:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.050+0000 7fa9727fc700 1 -- 172.21.15.116:0/403352387 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fa978003b10 con 0x7fa97c101240 2024-03-19T01:12:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.051+0000 7fa9727fc700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa9680a2890 0x7fa9680a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.051+0000 7fa980e5a700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa9680a2890 0x7fa9680a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.051+0000 7fa9727fc700 1 -- 172.21.15.116:0/403352387 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa9780cf960 con 0x7fa97c101240 2024-03-19T01:12:22.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.051+0000 7fa980e5a700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa9680a2890 0x7fa9680a4d50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009d20 tx=0x7fa96c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:22.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.057+0000 7fa9727fc700 1 -- 172.21.15.116:0/403352387 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa97805d740 con 0x7fa97c101240 2024-03-19T01:12:22.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.263+0000 7fa9838bf700 1 -- 172.21.15.116:0/403352387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fa97c02dbe0 con 0x7fa97c101240 2024-03-19T01:12:22.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.263+0000 7fa9727fc700 1 -- 172.21.15.116:0/403352387 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) ==== 56+0+5361 (secure 0 0 0) 0x7fa978013070 con 0x7fa97c101240 2024-03-19T01:12:22.264 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 87 2024-03-19T01:12:22.264 INFO:teuthology.orchestra.run.smithi116.stdout:fsid 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:12:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-03-19T01:02:30.471059+0000 2024-03-19T01:12:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:modified 2024-03-19T01:11:28.144841+0000 2024-03-19T01:12:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-19T01:12:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:crush_version 26 2024-03-19T01:12:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:full_ratio 0.95 2024-03-19T01:12:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:backfillfull_ratio 0.9 2024-03-19T01:12:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:nearfull_ratio 0.85 2024-03-19T01:12:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:require_min_compat_client luminous 2024-03-19T01:12:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:min_compat_client luminous 2024-03-19T01:12:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:require_osd_release squid 2024-03-19T01:12:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:stretch_mode_enabled false 2024-03-19T01:12:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-19T01:12:22.265 INFO:teuthology.orchestra.run.smithi116.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-19T01:12:22.266 INFO:teuthology.orchestra.run.smithi116.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-19T01:12:22.266 INFO:teuthology.orchestra.run.smithi116.stdout:max_osd 12 2024-03-19T01:12:22.266 INFO:teuthology.orchestra.run.smithi116.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6802/1461395811,v1:172.21.15.116:6803/1461395811] [v2:172.21.15.116:6804/1461395811,v1:172.21.15.116:6805/1461395811] exists,up 95728d5b-46c8-41c0-830a-7602d8b251fe 2024-03-19T01:12:22.266 INFO:teuthology.orchestra.run.smithi116.stdout:osd.1 up in weight 1 up_from 13 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6800/2727091908,v1:172.21.15.156:6801/2727091908] [v2:172.21.15.156:6802/2727091908,v1:172.21.15.156:6803/2727091908] exists,up 6ab4837e-b52f-4696-9361-2395e835e740 2024-03-19T01:12:22.266 INFO:teuthology.orchestra.run.smithi116.stdout:osd.2 up in weight 1 up_from 18 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6800/2120908758,v1:172.21.15.162:6801/2120908758] [v2:172.21.15.162:6802/2120908758,v1:172.21.15.162:6803/2120908758] exists,up ef1c978e-270f-4b74-844d-2fe4c09f2ef5 2024-03-19T01:12:22.266 INFO:teuthology.orchestra.run.smithi116.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6810/2026775607,v1:172.21.15.116:6811/2026775607] [v2:172.21.15.116:6812/2026775607,v1:172.21.15.116:6813/2026775607] exists,up 944f6747-a95f-49ae-8b3b-d7c48dc15059 2024-03-19T01:12:22.266 INFO:teuthology.orchestra.run.smithi116.stdout:osd.4 up in weight 1 up_from 33 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6808/1262076382,v1:172.21.15.156:6809/1262076382] [v2:172.21.15.156:6810/1262076382,v1:172.21.15.156:6811/1262076382] exists,up 1bd47f0a-a0d1-4402-8de0-940cd4f6a69b 2024-03-19T01:12:22.266 INFO:teuthology.orchestra.run.smithi116.stdout:osd.5 up in weight 1 up_from 38 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6808/2786247633,v1:172.21.15.162:6809/2786247633] [v2:172.21.15.162:6810/2786247633,v1:172.21.15.162:6811/2786247633] exists,up 457a29a6-6e9d-406a-95f6-d34e1881e6ca 2024-03-19T01:12:22.266 INFO:teuthology.orchestra.run.smithi116.stdout:osd.6 up in weight 1 up_from 46 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6818/2037449241,v1:172.21.15.116:6819/2037449241] [v2:172.21.15.116:6820/2037449241,v1:172.21.15.116:6821/2037449241] exists,up 7db79fde-7c61-4d3e-adb5-78b906eb3173 2024-03-19T01:12:22.266 INFO:teuthology.orchestra.run.smithi116.stdout:osd.7 up in weight 1 up_from 51 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6816/3540949514,v1:172.21.15.156:6817/3540949514] [v2:172.21.15.156:6818/3540949514,v1:172.21.15.156:6819/3540949514] exists,up 8e27814d-4a85-4dcd-8640-a09b2e55df23 2024-03-19T01:12:22.266 INFO:teuthology.orchestra.run.smithi116.stdout:osd.8 up in weight 1 up_from 59 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6816/1669615340,v1:172.21.15.162:6817/1669615340] [v2:172.21.15.162:6818/1669615340,v1:172.21.15.162:6819/1669615340] exists,up 9660b3b0-ed99-4050-a339-1beea2450c79 2024-03-19T01:12:22.266 INFO:teuthology.orchestra.run.smithi116.stdout:osd.9 up in weight 1 up_from 67 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.116:6826/96304116,v1:172.21.15.116:6827/96304116] [v2:172.21.15.116:6828/96304116,v1:172.21.15.116:6829/96304116] exists,up 59d2e3c8-0173-48eb-8eed-303c1dc31822 2024-03-19T01:12:22.266 INFO:teuthology.orchestra.run.smithi116.stdout:osd.10 up in weight 1 up_from 72 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6824/3316972631,v1:172.21.15.156:6825/3316972631] [v2:172.21.15.156:6826/3316972631,v1:172.21.15.156:6827/3316972631] exists,up 7abd5362-d32e-48a0-9de5-d23ea5ba4391 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:osd.11 up in weight 1 up_from 80 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6824/246086941,v1:172.21.15.162:6825/246086941] [v2:172.21.15.162:6826/246086941,v1:172.21.15.162:6827/246086941] exists,up b4fafaa0-173c-4b54-b5b1-10b925873fa0 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 2.6 [8,11] 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 2.b [2,1] 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 2.c [2,4] 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 2.1d [8,11] 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.2 [3,9] 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.c [5,6] 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.11 [4,8] 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.21 [5,9] 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.24 [4,9] 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:pg_upmap_items 3.3a [4,7] 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6800/3676795487 expires 2024-03-20T01:03:00.584768+0000 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/1073894644 expires 2024-03-20T01:03:26.974893+0000 2024-03-19T01:12:22.267 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6800/1107242393 expires 2024-03-20T01:03:26.974893+0000 2024-03-19T01:12:22.268 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/3553505800 expires 2024-03-20T01:03:00.584768+0000 2024-03-19T01:12:22.268 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6801/1107242393 expires 2024-03-20T01:03:26.974893+0000 2024-03-19T01:12:22.268 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:6801/3676795487 expires 2024-03-20T01:03:00.584768+0000 2024-03-19T01:12:22.268 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/2950253911 expires 2024-03-20T01:03:26.974893+0000 2024-03-19T01:12:22.268 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/2689675916 expires 2024-03-20T01:03:00.584768+0000 2024-03-19T01:12:22.268 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/1322211514 expires 2024-03-20T01:03:00.584768+0000 2024-03-19T01:12:22.268 INFO:teuthology.orchestra.run.smithi116.stdout:blocklist 172.21.15.116:0/2178441515 expires 2024-03-20T01:03:26.974893+0000 2024-03-19T01:12:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.266+0000 7fa9838bf700 1 -- 172.21.15.116:0/403352387 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa9680a2890 msgr2=0x7fa9680a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.266+0000 7fa9838bf700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa9680a2890 0x7fa9680a4d50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009d20 tx=0x7fa96c009260 comp rx=0 tx=0).stop 2024-03-19T01:12:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.266+0000 7fa9838bf700 1 -- 172.21.15.116:0/403352387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa97c101240 msgr2=0x7fa97c197510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.266+0000 7fa9838bf700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa97c101240 0x7fa97c197510 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa978009990 tx=0x7fa97800f690 comp rx=0 tx=0).stop 2024-03-19T01:12:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.266+0000 7fa9838bf700 1 -- 172.21.15.116:0/403352387 shutdown_connections 2024-03-19T01:12:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.266+0000 7fa9838bf700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa9680a2890 0x7fa9680a4d50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.267+0000 7fa9838bf700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa97c102de0 0x7fa97c19ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.267+0000 7fa9838bf700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa97c101240 0x7fa97c197510 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.267+0000 7fa9838bf700 1 --2- 172.21.15.116:0/403352387 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa97c102440 0x7fa97c197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:22.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.267+0000 7fa9838bf700 1 -- 172.21.15.116:0/403352387 >> 172.21.15.116:0/403352387 conn(0x7fa97c0fc7e0 msgr2=0x7fa97c105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:22.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.267+0000 7fa9838bf700 1 -- 172.21.15.116:0/403352387 shutdown_connections 2024-03-19T01:12:22.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:22.267+0000 7fa9838bf700 1 -- 172.21.15.116:0/403352387 wait complete. 2024-03-19T01:12:22.448 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:22 smithi116 ceph-mon[62701]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T01:12:22.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:22 smithi156 ceph-mon[66799]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T01:12:22.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:22 smithi162 ceph-mon[67178]: pgmap v344: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T01:12:23.076 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-19T01:12:23.088 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:12:23.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 -- 172.21.15.116:0/1606401401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa20100760 msgr2=0x7faa20100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:23.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 --2- 172.21.15.116:0/1606401401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa20100760 0x7faa20100b80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faa18009ca0 tx=0x7faa1802c0f0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 -- 172.21.15.116:0/1606401401 shutdown_connections 2024-03-19T01:12:23.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 --2- 172.21.15.116:0/1606401401 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa20102310 0x7faa20108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 --2- 172.21.15.116:0/1606401401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa20101950 0x7faa20101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 --2- 172.21.15.116:0/1606401401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa20100760 0x7faa20100b80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 -- 172.21.15.116:0/1606401401 >> 172.21.15.116:0/1606401401 conn(0x7faa200fbcd0 msgr2=0x7faa200fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:23.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 -- 172.21.15.116:0/1606401401 shutdown_connections 2024-03-19T01:12:23.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 -- 172.21.15.116:0/1606401401 wait complete. 2024-03-19T01:12:23.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 Processor -- start 2024-03-19T01:12:23.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 -- start start 2024-03-19T01:12:23.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa20100760 0x7faa20199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:23.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa20101950 0x7faa2019a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:23.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa20102310 0x7faa201a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:23.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faa2010c3b0 con 0x7faa20100760 2024-03-19T01:12:23.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faa2010c230 con 0x7faa20102310 2024-03-19T01:12:23.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.195+0000 7faa27627640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faa2010c530 con 0x7faa20101950 2024-03-19T01:12:23.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa253c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa20100760 0x7faa20199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:23.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa253c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa20100760 0x7faa20199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35152/0 (socket says 172.21.15.116:35152) 2024-03-19T01:12:23.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa253c3640 1 -- 172.21.15.116:0/4054278446 learned_addr learned my addr 172.21.15.116:0/4054278446 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:23.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa25bc4640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa20102310 0x7faa201a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:23.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa24bc2640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa20101950 0x7faa2019a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:23.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa253c3640 1 -- 172.21.15.116:0/4054278446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa20101950 msgr2=0x7faa2019a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:23.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa253c3640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa20101950 0x7faa2019a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa253c3640 1 -- 172.21.15.116:0/4054278446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa20102310 msgr2=0x7faa201a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:23.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa253c3640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa20102310 0x7faa201a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa253c3640 1 -- 172.21.15.116:0/4054278446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa201a1a10 con 0x7faa20100760 2024-03-19T01:12:23.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa253c3640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa20100760 0x7faa20199e40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7faa18006010 tx=0x7faa18032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:23.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa127fc640 1 -- 172.21.15.116:0/4054278446 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18041020 con 0x7faa20100760 2024-03-19T01:12:23.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa127fc640 1 -- 172.21.15.116:0/4054278446 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa18032340 con 0x7faa20100760 2024-03-19T01:12:23.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa27627640 1 -- 172.21.15.116:0/4054278446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa201a1ca0 con 0x7faa20100760 2024-03-19T01:12:23.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa127fc640 1 -- 172.21.15.116:0/4054278446 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa18039040 con 0x7faa20100760 2024-03-19T01:12:23.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.196+0000 7faa27627640 1 -- 172.21.15.116:0/4054278446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa201a2190 con 0x7faa20100760 2024-03-19T01:12:23.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.197+0000 7faa27627640 1 -- 172.21.15.116:0/4054278446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa201a2360 con 0x7faa20100760 2024-03-19T01:12:23.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.199+0000 7faa127fc640 1 -- 172.21.15.116:0/4054278446 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7faa18044020 con 0x7faa20100760 2024-03-19T01:12:23.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.199+0000 7faa127fc640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7faa0c0a2800 0x7faa0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:23.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.200+0000 7faa127fc640 1 -- 172.21.15.116:0/4054278446 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7faa180f0800 con 0x7faa20100760 2024-03-19T01:12:23.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.200+0000 7faa24bc2640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7faa0c0a2800 0x7faa0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:23.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.200+0000 7faa24bc2640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7faa0c0a2800 0x7faa0c0a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faa2019b300 tx=0x7faa14005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:23.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.204+0000 7faa127fc640 1 -- 172.21.15.116:0/4054278446 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faa1804b020 con 0x7faa20100760 2024-03-19T01:12:23.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.394+0000 7faa27627640 1 -- 172.21.15.116:0/4054278446 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa201a2530 con 0x7faa20100760 2024-03-19T01:12:23.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.396+0000 7faa127fc640 1 -- 172.21.15.116:0/4054278446 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22804 (secure 0 0 0) 0x7faa180bd880 con 0x7faa20100760 2024-03-19T01:12:23.397 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:23.397 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:11:28.144841+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:12:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.398+0000 7faa27627640 1 -- 172.21.15.116:0/4054278446 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7faa0c0a2800 msgr2=0x7faa0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.398+0000 7faa27627640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7faa0c0a2800 0x7faa0c0a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faa2019b300 tx=0x7faa14005f90 comp rx=0 tx=0).stop 2024-03-19T01:12:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.398+0000 7faa27627640 1 -- 172.21.15.116:0/4054278446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa20100760 msgr2=0x7faa20199e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.398+0000 7faa27627640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa20100760 0x7faa20199e40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7faa18006010 tx=0x7faa18032f20 comp rx=0 tx=0).stop 2024-03-19T01:12:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.398+0000 7faa27627640 1 -- 172.21.15.116:0/4054278446 shutdown_connections 2024-03-19T01:12:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.398+0000 7faa27627640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7faa0c0a2800 0x7faa0c0a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.398+0000 7faa27627640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa20102310 0x7faa201a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.398+0000 7faa27627640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7faa20101950 0x7faa2019a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.398+0000 7faa27627640 1 --2- 172.21.15.116:0/4054278446 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa20100760 0x7faa20199e40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.398+0000 7faa27627640 1 -- 172.21.15.116:0/4054278446 >> 172.21.15.116:0/4054278446 conn(0x7faa200fbcd0 msgr2=0x7faa20069b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.398+0000 7faa27627640 1 -- 172.21.15.116:0/4054278446 shutdown_connections 2024-03-19T01:12:23.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.398+0000 7faa27627640 1 -- 172.21.15.116:0/4054278446 wait complete. 2024-03-19T01:12:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:23 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/403352387' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-19T01:12:23.410 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:05:53.301933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:10:31.028304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:10:31.704469+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:12:23.410 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:12:23.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:23 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/403352387' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-19T01:12:23.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.518+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2348644391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1241033e0 msgr2=0x7fd124109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.518+0000 7fd12b7f1640 1 --2- 172.21.15.116:0/2348644391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1241033e0 0x7fd124109c70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd11c009ca0 tx=0x7fd11c02c0f0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.518+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2348644391 shutdown_connections 2024-03-19T01:12:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.518+0000 7fd12b7f1640 1 --2- 172.21.15.116:0/2348644391 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd1241033e0 0x7fd124109c70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.518+0000 7fd12b7f1640 1 --2- 172.21.15.116:0/2348644391 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd124102a20 0x7fd124102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.518+0000 7fd12b7f1640 1 --2- 172.21.15.116:0/2348644391 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd124101830 0x7fd124101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.518+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2348644391 >> 172.21.15.116:0/2348644391 conn(0x7fd1240fcdc0 msgr2=0x7fd1240ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2348644391 shutdown_connections 2024-03-19T01:12:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2348644391 wait complete. 2024-03-19T01:12:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd12b7f1640 1 Processor -- start 2024-03-19T01:12:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd12b7f1640 1 -- start start 2024-03-19T01:12:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd12b7f1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd124101830 0x7fd124197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:23.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd12b7f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd124102a20 0x7fd124198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd12b7f1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1241033e0 0x7fd12419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd12b7f1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd12410d4b0 con 0x7fd124102a20 2024-03-19T01:12:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd12b7f1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd12410d330 con 0x7fd124101830 2024-03-19T01:12:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd12b7f1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd12410d630 con 0x7fd1241033e0 2024-03-19T01:12:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd128d8c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd124102a20 0x7fd124198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd128d8c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd124102a20 0x7fd124198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35180/0 (socket says 172.21.15.116:35180) 2024-03-19T01:12:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd128d8c640 1 -- 172.21.15.116:0/2731388654 learned_addr learned my addr 172.21.15.116:0/2731388654 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:23.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd12958d640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd124101830 0x7fd124197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.519+0000 7fd129d8e640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1241033e0 0x7fd12419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.520+0000 7fd128d8c640 1 -- 172.21.15.116:0/2731388654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1241033e0 msgr2=0x7fd12419f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.520+0000 7fd128d8c640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1241033e0 0x7fd12419f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.520+0000 7fd128d8c640 1 -- 172.21.15.116:0/2731388654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd124101830 msgr2=0x7fd124197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.520+0000 7fd128d8c640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd124101830 0x7fd124197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.520+0000 7fd128d8c640 1 -- 172.21.15.116:0/2731388654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12419f710 con 0x7fd124102a20 2024-03-19T01:12:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.520+0000 7fd128d8c640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd124102a20 0x7fd124198030 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd11400e9f0 tx=0x7fd11400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:23.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.520+0000 7fd11a7fc640 1 -- 172.21.15.116:0/2731388654 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11400cd80 con 0x7fd124102a20 2024-03-19T01:12:23.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.520+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2731388654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd12419fa00 con 0x7fd124102a20 2024-03-19T01:12:23.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.520+0000 7fd11a7fc640 1 -- 172.21.15.116:0/2731388654 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd114005500 con 0x7fd124102a20 2024-03-19T01:12:23.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.520+0000 7fd11a7fc640 1 -- 172.21.15.116:0/2731388654 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd114010690 con 0x7fd124102a20 2024-03-19T01:12:23.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.520+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2731388654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd12419ff50 con 0x7fd124102a20 2024-03-19T01:12:23.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.521+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2731388654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd124108b50 con 0x7fd124102a20 2024-03-19T01:12:23.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.523+0000 7fd11a7fc640 1 -- 172.21.15.116:0/2731388654 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fd1140046c0 con 0x7fd124102a20 2024-03-19T01:12:23.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.523+0000 7fd11a7fc640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd1100a2800 0x7fd1100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:23.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.523+0000 7fd12958d640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd1100a2800 0x7fd1100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:23.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.523+0000 7fd11a7fc640 1 -- 172.21.15.116:0/2731388654 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd1140cee70 con 0x7fd124102a20 2024-03-19T01:12:23.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.524+0000 7fd12958d640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd1100a2800 0x7fd1100a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd124102880 tx=0x7fd120005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:23.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.528+0000 7fd11a7fc640 1 -- 172.21.15.116:0/2731388654 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd11409bef0 con 0x7fd124102a20 2024-03-19T01:12:23.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:23 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/403352387' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-19T01:12:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.705+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2731388654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd124062420 con 0x7fd124102a20 2024-03-19T01:12:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.706+0000 7fd11a7fc640 1 -- 172.21.15.116:0/2731388654 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fd11409b7f0 con 0x7fd124102a20 2024-03-19T01:12:23.706 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T01:12:23.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.707+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2731388654 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd1100a2800 msgr2=0x7fd1100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:23.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.707+0000 7fd12b7f1640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd1100a2800 0x7fd1100a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd124102880 tx=0x7fd120005f90 comp rx=0 tx=0).stop 2024-03-19T01:12:23.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.707+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2731388654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd124102a20 msgr2=0x7fd124198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:23.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.707+0000 7fd12b7f1640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd124102a20 0x7fd124198030 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd11400e9f0 tx=0x7fd11400edf0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.708+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2731388654 shutdown_connections 2024-03-19T01:12:23.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.708+0000 7fd12b7f1640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd1241033e0 0x7fd12419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.708+0000 7fd12b7f1640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd124102a20 0x7fd124198030 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.708+0000 7fd12b7f1640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd1100a2800 0x7fd1100a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.708+0000 7fd12b7f1640 1 --2- 172.21.15.116:0/2731388654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd124101830 0x7fd124197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.708+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2731388654 >> 172.21.15.116:0/2731388654 conn(0x7fd1240fcdc0 msgr2=0x7fd124105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:23.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.708+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2731388654 shutdown_connections 2024-03-19T01:12:23.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.708+0000 7fd12b7f1640 1 -- 172.21.15.116:0/2731388654 wait complete. 2024-03-19T01:12:23.721 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:12:23.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.831+0000 7f5399574640 1 -- 172.21.15.116:0/1408222146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5394100720 msgr2=0x7f5394100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:23.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.831+0000 7f5399574640 1 --2- 172.21.15.116:0/1408222146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5394100720 0x7f5394100b40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5384009900 tx=0x7f538402bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:23.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.831+0000 7f5399574640 1 -- 172.21.15.116:0/1408222146 shutdown_connections 2024-03-19T01:12:23.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.831+0000 7f5399574640 1 --2- 172.21.15.116:0/1408222146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53941022d0 0x7f5394108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.831+0000 7f5399574640 1 --2- 172.21.15.116:0/1408222146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5394101910 0x7f5394101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.831+0000 7f5399574640 1 --2- 172.21.15.116:0/1408222146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5394100720 0x7f5394100b40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.831+0000 7f5399574640 1 -- 172.21.15.116:0/1408222146 >> 172.21.15.116:0/1408222146 conn(0x7f53940fbcb0 msgr2=0x7f53940fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:23.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.831+0000 7f5399574640 1 -- 172.21.15.116:0/1408222146 shutdown_connections 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.831+0000 7f5399574640 1 -- 172.21.15.116:0/1408222146 wait complete. 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.832+0000 7f5399574640 1 Processor -- start 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.832+0000 7f5399574640 1 -- start start 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.832+0000 7f5399574640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5394100720 0x7f5394196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.832+0000 7f5399574640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5394101910 0x7f5394196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.832+0000 7f5399574640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53941022d0 0x7f539419e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.832+0000 7f5399574640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f539410c370 con 0x7f5394101910 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.832+0000 7f5399574640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f539410c1f0 con 0x7f5394100720 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.832+0000 7f5399574640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f539410c4f0 con 0x7f53941022d0 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.832+0000 7f538a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5394101910 0x7f5394196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.832+0000 7f5392ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5394100720 0x7f5394196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.832+0000 7f538a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5394101910 0x7f5394196fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35186/0 (socket says 172.21.15.116:35186) 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.832+0000 7f538a7fc640 1 -- 172.21.15.116:0/849432574 learned_addr learned my addr 172.21.15.116:0/849432574 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:23.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.832+0000 7f53937fe640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53941022d0 0x7f539419e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.833+0000 7f538a7fc640 1 -- 172.21.15.116:0/849432574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53941022d0 msgr2=0x7f539419e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.833+0000 7f538a7fc640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53941022d0 0x7f539419e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.833+0000 7f538a7fc640 1 -- 172.21.15.116:0/849432574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5394100720 msgr2=0x7f5394196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.833+0000 7f538a7fc640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5394100720 0x7f5394196aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.833+0000 7f538a7fc640 1 -- 172.21.15.116:0/849432574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f539419e6c0 con 0x7f5394101910 2024-03-19T01:12:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.833+0000 7f5392ffd640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5394100720 0x7f5394196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.833+0000 7f538a7fc640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5394101910 0x7f5394196fe0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f537c00e9f0 tx=0x7f537c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.833+0000 7f5390ff9640 1 -- 172.21.15.116:0/849432574 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c00ccc0 con 0x7f5394101910 2024-03-19T01:12:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.833+0000 7f5390ff9640 1 -- 172.21.15.116:0/849432574 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f537c010910 con 0x7f5394101910 2024-03-19T01:12:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.833+0000 7f5399574640 1 -- 172.21.15.116:0/849432574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f539419e9b0 con 0x7f5394101910 2024-03-19T01:12:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.833+0000 7f5390ff9640 1 -- 172.21.15.116:0/849432574 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c0188a0 con 0x7f5394101910 2024-03-19T01:12:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.833+0000 7f5399574640 1 -- 172.21.15.116:0/849432574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f539419ef00 con 0x7f5394101910 2024-03-19T01:12:23.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.834+0000 7f5399574640 1 -- 172.21.15.116:0/849432574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5394107b60 con 0x7f5394101910 2024-03-19T01:12:23.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.835+0000 7f5390ff9640 1 -- 172.21.15.116:0/849432574 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f537c010ac0 con 0x7f5394101910 2024-03-19T01:12:23.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.836+0000 7f5390ff9640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53800a2800 0x7f53800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:23.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.836+0000 7f5392ffd640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53800a2800 0x7f53800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:23.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.836+0000 7f5390ff9640 1 -- 172.21.15.116:0/849432574 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f537c0ceee0 con 0x7f5394101910 2024-03-19T01:12:23.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.836+0000 7f5392ffd640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53800a2800 0x7f53800a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5384000c00 tx=0x7f5384005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:23.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:23.840+0000 7f5390ff9640 1 -- 172.21.15.116:0/849432574 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f537c09bf60 con 0x7f5394101910 2024-03-19T01:12:24.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.018+0000 7f5399574640 1 -- 172.21.15.116:0/849432574 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5394062420 con 0x7f5394101910 2024-03-19T01:12:24.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.018+0000 7f5390ff9640 1 -- 172.21.15.116:0/849432574 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f537c09b860 con 0x7f5394101910 2024-03-19T01:12:24.020 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T01:12:24.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.020+0000 7f5399574640 1 -- 172.21.15.116:0/849432574 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53800a2800 msgr2=0x7f53800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:24.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.020+0000 7f5399574640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53800a2800 0x7f53800a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5384000c00 tx=0x7f5384005fd0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.020+0000 7f5399574640 1 -- 172.21.15.116:0/849432574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5394101910 msgr2=0x7f5394196fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:24.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.020+0000 7f5399574640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5394101910 0x7f5394196fe0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f537c00e9f0 tx=0x7f537c00edf0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.020+0000 7f5399574640 1 -- 172.21.15.116:0/849432574 shutdown_connections 2024-03-19T01:12:24.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.020+0000 7f5399574640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f53941022d0 0x7f539419e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.020+0000 7f5399574640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5394101910 0x7f5394196fe0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.020+0000 7f5399574640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f53800a2800 0x7f53800a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.020+0000 7f5399574640 1 --2- 172.21.15.116:0/849432574 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5394100720 0x7f5394196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.020+0000 7f5399574640 1 -- 172.21.15.116:0/849432574 >> 172.21.15.116:0/849432574 conn(0x7f53940fbcb0 msgr2=0x7f5394104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:24.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.020+0000 7f5399574640 1 -- 172.21.15.116:0/849432574 shutdown_connections 2024-03-19T01:12:24.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.020+0000 7f5399574640 1 -- 172.21.15.116:0/849432574 wait complete. 2024-03-19T01:12:24.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:12:24.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.164+0000 7f9dfad62640 1 -- 172.21.15.116:0/3724351221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df4101910 msgr2=0x7f9df4101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:24.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.164+0000 7f9dfad62640 1 --2- 172.21.15.116:0/3724351221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df4101910 0x7f9df4101d90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f9de0009900 tx=0x7f9de002bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:24.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.164+0000 7f9dfad62640 1 -- 172.21.15.116:0/3724351221 shutdown_connections 2024-03-19T01:12:24.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.164+0000 7f9dfad62640 1 --2- 172.21.15.116:0/3724351221 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9df41022d0 0x7f9df4108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.164+0000 7f9dfad62640 1 --2- 172.21.15.116:0/3724351221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df4101910 0x7f9df4101d90 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.164+0000 7f9dfad62640 1 --2- 172.21.15.116:0/3724351221 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9df4100720 0x7f9df4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.164+0000 7f9dfad62640 1 -- 172.21.15.116:0/3724351221 >> 172.21.15.116:0/3724351221 conn(0x7f9df40fbcb0 msgr2=0x7f9df40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:24.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.164+0000 7f9dfad62640 1 -- 172.21.15.116:0/3724351221 shutdown_connections 2024-03-19T01:12:24.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.164+0000 7f9dfad62640 1 -- 172.21.15.116:0/3724351221 wait complete. 2024-03-19T01:12:24.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9dfad62640 1 Processor -- start 2024-03-19T01:12:24.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9dfad62640 1 -- start start 2024-03-19T01:12:24.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9dfad62640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9df4100720 0x7f9df40694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9dfad62640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9df4101910 0x7f9df419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9dfad62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df41022d0 0x7f9df419eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9dfad62640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9df410c3a0 con 0x7f9df41022d0 2024-03-19T01:12:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9dfad62640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9df410c220 con 0x7f9df4101910 2024-03-19T01:12:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9dfad62640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9df410c520 con 0x7f9df4100720 2024-03-19T01:12:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9df92ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df41022d0 0x7f9df419eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9df8afe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9df4100720 0x7f9df40694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9df8afe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9df4100720 0x7f9df40694f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42092/0 (socket says 172.21.15.116:42092) 2024-03-19T01:12:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9df3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9df4101910 0x7f9df419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9df92ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df41022d0 0x7f9df419eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35216/0 (socket says 172.21.15.116:35216) 2024-03-19T01:12:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9df8afe640 1 -- 172.21.15.116:0/1516269471 learned_addr learned my addr 172.21.15.116:0/1516269471 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9df8afe640 1 -- 172.21.15.116:0/1516269471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9df4101910 msgr2=0x7f9df419c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9df8afe640 1 --2- 172.21.15.116:0/1516269471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9df4101910 0x7f9df419c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9df8afe640 1 -- 172.21.15.116:0/1516269471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df41022d0 msgr2=0x7f9df419eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9df8afe640 1 --2- 172.21.15.116:0/1516269471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df41022d0 0x7f9df419eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9df8afe640 1 -- 172.21.15.116:0/1516269471 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df419f7e0 con 0x7f9df4100720 2024-03-19T01:12:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9df8afe640 1 --2- 172.21.15.116:0/1516269471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9df4100720 0x7f9df40694f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9de800e9c0 tx=0x7f9de800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.165+0000 7f9df1ffb640 1 -- 172.21.15.116:0/1516269471 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de800cd80 con 0x7f9df4100720 2024-03-19T01:12:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.166+0000 7f9df1ffb640 1 -- 172.21.15.116:0/1516269471 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9de80035f0 con 0x7f9df4100720 2024-03-19T01:12:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.166+0000 7f9dfad62640 1 -- 172.21.15.116:0/1516269471 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df419fa70 con 0x7f9df4100720 2024-03-19T01:12:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.166+0000 7f9df1ffb640 1 -- 172.21.15.116:0/1516269471 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8010690 con 0x7f9df4100720 2024-03-19T01:12:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.166+0000 7f9dfad62640 1 -- 172.21.15.116:0/1516269471 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df419ff40 con 0x7f9df4100720 2024-03-19T01:12:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.167+0000 7f9dfad62640 1 -- 172.21.15.116:0/1516269471 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9df4068610 con 0x7f9df4100720 2024-03-19T01:12:24.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.167+0000 7f9df1ffb640 1 -- 172.21.15.116:0/1516269471 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f9de80037a0 con 0x7f9df4100720 2024-03-19T01:12:24.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.168+0000 7f9df1ffb640 1 --2- 172.21.15.116:0/1516269471 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9de40a2800 0x7f9de40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:24.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.168+0000 7f9df3fff640 1 --2- 172.21.15.116:0/1516269471 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9de40a2800 0x7f9de40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:24.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.168+0000 7f9df1ffb640 1 -- 172.21.15.116:0/1516269471 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f9de80cede0 con 0x7f9df4100720 2024-03-19T01:12:24.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.168+0000 7f9df3fff640 1 --2- 172.21.15.116:0/1516269471 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9de40a2800 0x7f9de40a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9de002c7b0 tx=0x7f9de0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:24.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.172+0000 7f9df1ffb640 1 -- 172.21.15.116:0/1516269471 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9de809be60 con 0x7f9df4100720 2024-03-19T01:12:24.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.352+0000 7f9dfad62640 1 -- 172.21.15.116:0/1516269471 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9df4062420 con 0x7f9df4100720 2024-03-19T01:12:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.353+0000 7f9df1ffb640 1 -- 172.21.15.116:0/1516269471 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f9de809b760 con 0x7f9df4100720 2024-03-19T01:12:24.354 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T01:12:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.355+0000 7f9dfad62640 1 -- 172.21.15.116:0/1516269471 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9de40a2800 msgr2=0x7f9de40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.355+0000 7f9dfad62640 1 --2- 172.21.15.116:0/1516269471 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9de40a2800 0x7f9de40a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9de002c7b0 tx=0x7f9de0005fd0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.355+0000 7f9dfad62640 1 -- 172.21.15.116:0/1516269471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9df4100720 msgr2=0x7f9df40694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.355+0000 7f9dfad62640 1 --2- 172.21.15.116:0/1516269471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9df4100720 0x7f9df40694f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9de800e9c0 tx=0x7f9de800ee80 comp rx=0 tx=0).stop 2024-03-19T01:12:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.355+0000 7f9dfad62640 1 -- 172.21.15.116:0/1516269471 shutdown_connections 2024-03-19T01:12:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.355+0000 7f9dfad62640 1 --2- 172.21.15.116:0/1516269471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9df41022d0 0x7f9df419eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.355+0000 7f9dfad62640 1 --2- 172.21.15.116:0/1516269471 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9de40a2800 0x7f9de40a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.355+0000 7f9dfad62640 1 --2- 172.21.15.116:0/1516269471 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9df4101910 0x7f9df419c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.355+0000 7f9dfad62640 1 --2- 172.21.15.116:0/1516269471 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9df4100720 0x7f9df40694f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.355+0000 7f9dfad62640 1 -- 172.21.15.116:0/1516269471 >> 172.21.15.116:0/1516269471 conn(0x7f9df40fbcb0 msgr2=0x7f9df4104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.355+0000 7f9dfad62640 1 -- 172.21.15.116:0/1516269471 shutdown_connections 2024-03-19T01:12:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.355+0000 7f9dfad62640 1 -- 172.21.15.116:0/1516269471 wait complete. 2024-03-19T01:12:24.366 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:24 smithi116 ceph-mon[62701]: pgmap v345: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:12:24.366 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:24 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4054278446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:12:24.366 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:24 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2731388654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:12:24.367 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:24 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/849432574' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:12:24.368 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:12:24.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.469+0000 7f977dac0640 1 -- 172.21.15.116:0/183326606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9778101930 msgr2=0x7f9778101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.469+0000 7f977dac0640 1 --2- 172.21.15.116:0/183326606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9778101930 0x7f9778101db0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f976c009860 tx=0x7f976c02bb70 comp rx=0 tx=0).stop 2024-03-19T01:12:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.469+0000 7f977dac0640 1 -- 172.21.15.116:0/183326606 shutdown_connections 2024-03-19T01:12:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.469+0000 7f977dac0640 1 --2- 172.21.15.116:0/183326606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97781022f0 0x7f9778108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.469+0000 7f977dac0640 1 --2- 172.21.15.116:0/183326606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9778101930 0x7f9778101db0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.469+0000 7f977dac0640 1 --2- 172.21.15.116:0/183326606 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9778100740 0x7f9778100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.469+0000 7f977dac0640 1 -- 172.21.15.116:0/183326606 >> 172.21.15.116:0/183326606 conn(0x7f97780fbcb0 msgr2=0x7f97780fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.469+0000 7f977dac0640 1 -- 172.21.15.116:0/183326606 shutdown_connections 2024-03-19T01:12:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.469+0000 7f977dac0640 1 -- 172.21.15.116:0/183326606 wait complete. 2024-03-19T01:12:24.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.469+0000 7f977dac0640 1 Processor -- start 2024-03-19T01:12:24.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.469+0000 7f977dac0640 1 -- start start 2024-03-19T01:12:24.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f977dac0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9778100740 0x7f9778196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:24.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f977dac0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9778101930 0x7f9778196f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:24.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f977dac0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97781022f0 0x7f977819dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:24.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f977dac0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f977810c3a0 con 0x7f97781022f0 2024-03-19T01:12:24.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f977dac0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f977810c220 con 0x7f9778101930 2024-03-19T01:12:24.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f977dac0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f977810c520 con 0x7f9778100740 2024-03-19T01:12:24.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9777fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97781022f0 0x7f977819dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:24.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f97777fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9778100740 0x7f9778196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:24.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9777fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97781022f0 0x7f977819dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35242/0 (socket says 172.21.15.116:35242) 2024-03-19T01:12:24.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9776ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9778101930 0x7f9778196f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:24.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9777fff640 1 -- 172.21.15.116:0/892333803 learned_addr learned my addr 172.21.15.116:0/892333803 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:24.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f97777fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9778100740 0x7f9778196a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42098/0 (socket says 172.21.15.116:42098) 2024-03-19T01:12:24.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9777fff640 1 -- 172.21.15.116:0/892333803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9778100740 msgr2=0x7f9778196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:24.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9777fff640 1 --2- 172.21.15.116:0/892333803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9778100740 0x7f9778196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9777fff640 1 -- 172.21.15.116:0/892333803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9778101930 msgr2=0x7f9778196f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:24.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9777fff640 1 --2- 172.21.15.116:0/892333803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9778101930 0x7f9778196f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9777fff640 1 -- 172.21.15.116:0/892333803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977819e660 con 0x7f97781022f0 2024-03-19T01:12:24.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9776ffd640 1 --2- 172.21.15.116:0/892333803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9778101930 0x7f9778196f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:24.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9777fff640 1 --2- 172.21.15.116:0/892333803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97781022f0 0x7f977819dff0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f977000eca0 tx=0x7f977000b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:24.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9774ff9640 1 -- 172.21.15.116:0/892333803 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977000be40 con 0x7f97781022f0 2024-03-19T01:12:24.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f977dac0640 1 -- 172.21.15.116:0/892333803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97780689e0 con 0x7f97781022f0 2024-03-19T01:12:24.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9774ff9640 1 -- 172.21.15.116:0/892333803 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9770005020 con 0x7f97781022f0 2024-03-19T01:12:24.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f9774ff9640 1 -- 172.21.15.116:0/892333803 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9770017650 con 0x7f97781022f0 2024-03-19T01:12:24.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.470+0000 7f977dac0640 1 -- 172.21.15.116:0/892333803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9778068f00 con 0x7f97781022f0 2024-03-19T01:12:24.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.472+0000 7f977dac0640 1 -- 172.21.15.116:0/892333803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9778068610 con 0x7f97781022f0 2024-03-19T01:12:24.476 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:24 smithi156 ceph-mon[66799]: pgmap v345: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:12:24.477 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:24 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/4054278446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:12:24.477 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:24 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2731388654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:12:24.477 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:24 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/849432574' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:12:24.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.473+0000 7f9774ff9640 1 -- 172.21.15.116:0/892333803 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f97700051d0 con 0x7f97781022f0 2024-03-19T01:12:24.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.474+0000 7f9774ff9640 1 --2- 172.21.15.116:0/892333803 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f97600a2800 0x7f97600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:24.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.474+0000 7f97777fe640 1 --2- 172.21.15.116:0/892333803 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f97600a2800 0x7f97600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:24.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.474+0000 7f9774ff9640 1 -- 172.21.15.116:0/892333803 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f97700d04a0 con 0x7f97781022f0 2024-03-19T01:12:24.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.474+0000 7f97777fe640 1 --2- 172.21.15.116:0/892333803 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f97600a2800 0x7f97600a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9778101790 tx=0x7f9768009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:24.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.478+0000 7f9774ff9640 1 -- 172.21.15.116:0/892333803 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f977009d520 con 0x7f97781022f0 2024-03-19T01:12:24.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:24 smithi162 ceph-mon[67178]: pgmap v345: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:12:24.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:24 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/4054278446' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:12:24.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:24 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2731388654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:12:24.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:24 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/849432574' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:12:24.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.877+0000 7f977dac0640 1 -- 172.21.15.116:0/892333803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f977819e8f0 con 0x7f97781022f0 2024-03-19T01:12:24.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.877+0000 7f9774ff9640 1 -- 172.21.15.116:0/892333803 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11938 (secure 0 0 0) 0x7f977009ce20 con 0x7f97781022f0 2024-03-19T01:12:24.878 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:24.879 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T01:10:32.487607+0000","modified":"2024-03-19T01:10:38.982004+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24488,"mds_2":24473,"mds_3":24460,"mds_4":24445},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1242639081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1242639081},{"type":"v1","addr":"172.21.15.116:6839","nonce":1242639081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24394":{"gid":24394,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":3,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.162:6839/441486392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":441486392},{"type":"v1","addr":"172.21.15.162:6839","nonce":441486392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"a","rank":1,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6841/3343188504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3343188504},{"type":"v1","addr":"172.21.15.116:6841","nonce":3343188504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1601537427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1601537427},{"type":"v1","addr":"172.21.15.156:6839","nonce":1601537427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24473,24445,24500,24460,24488]},"id":1}]} 2024-03-19T01:12:24.879 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2024-03-19T01:12:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.880+0000 7f977dac0640 1 -- 172.21.15.116:0/892333803 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f97600a2800 msgr2=0x7f97600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.880+0000 7f977dac0640 1 --2- 172.21.15.116:0/892333803 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f97600a2800 0x7f97600a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9778101790 tx=0x7f9768009260 comp rx=0 tx=0).stop 2024-03-19T01:12:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.880+0000 7f977dac0640 1 -- 172.21.15.116:0/892333803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97781022f0 msgr2=0x7f977819dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.880+0000 7f977dac0640 1 --2- 172.21.15.116:0/892333803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97781022f0 0x7f977819dff0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f977000eca0 tx=0x7f977000b690 comp rx=0 tx=0).stop 2024-03-19T01:12:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.880+0000 7f977dac0640 1 -- 172.21.15.116:0/892333803 shutdown_connections 2024-03-19T01:12:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.880+0000 7f977dac0640 1 --2- 172.21.15.116:0/892333803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97781022f0 0x7f977819dff0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.880+0000 7f977dac0640 1 --2- 172.21.15.116:0/892333803 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f97600a2800 0x7f97600a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.880+0000 7f977dac0640 1 --2- 172.21.15.116:0/892333803 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9778101930 0x7f9778196f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.880+0000 7f977dac0640 1 --2- 172.21.15.116:0/892333803 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9778100740 0x7f9778196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:24.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.880+0000 7f977dac0640 1 -- 172.21.15.116:0/892333803 >> 172.21.15.116:0/892333803 conn(0x7f97780fbcb0 msgr2=0x7f9778104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:24.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.880+0000 7f977dac0640 1 -- 172.21.15.116:0/892333803 shutdown_connections 2024-03-19T01:12:24.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.880+0000 7f977dac0640 1 -- 172.21.15.116:0/892333803 wait complete. 2024-03-19T01:12:24.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:12:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.997+0000 7f9fe38c6640 1 -- 172.21.15.116:0/1319224587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fdc107580 msgr2=0x7f9fdc109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.997+0000 7f9fe38c6640 1 --2- 172.21.15.116:0/1319224587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fdc107580 0x7f9fdc109970 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009ca0 tx=0x7f9fd402c0f0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.997+0000 7f9fe38c6640 1 -- 172.21.15.116:0/1319224587 shutdown_connections 2024-03-19T01:12:25.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.997+0000 7f9fe38c6640 1 --2- 172.21.15.116:0/1319224587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fdc107580 0x7f9fdc109970 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.997+0000 7f9fe38c6640 1 --2- 172.21.15.116:0/1319224587 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fdc104c50 0x7f9fdc107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.997+0000 7f9fe38c6640 1 --2- 172.21.15.116:0/1319224587 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fdc102320 0x7f9fdc104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.997+0000 7f9fe38c6640 1 -- 172.21.15.116:0/1319224587 >> 172.21.15.116:0/1319224587 conn(0x7f9fdc0fbcd0 msgr2=0x7f9fdc0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.997+0000 7f9fe38c6640 1 -- 172.21.15.116:0/1319224587 shutdown_connections 2024-03-19T01:12:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.997+0000 7f9fe38c6640 1 -- 172.21.15.116:0/1319224587 wait complete. 2024-03-19T01:12:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe38c6640 1 Processor -- start 2024-03-19T01:12:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe38c6640 1 -- start start 2024-03-19T01:12:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe38c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fdc102320 0x7f9fdc197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe38c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fdc104c50 0x7f9fdc198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe38c6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fdc107580 0x7f9fdc19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe38c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc10d4c0 con 0x7f9fdc102320 2024-03-19T01:12:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe38c6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc10d340 con 0x7f9fdc107580 2024-03-19T01:12:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe38c6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9fdc10d640 con 0x7f9fdc104c50 2024-03-19T01:12:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe1662640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fdc102320 0x7f9fdc197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe1662640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fdc102320 0x7f9fdc197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35258/0 (socket says 172.21.15.116:35258) 2024-03-19T01:12:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe1662640 1 -- 172.21.15.116:0/756213438 learned_addr learned my addr 172.21.15.116:0/756213438 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe1e63640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fdc107580 0x7f9fdc19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe0e61640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fdc104c50 0x7f9fdc198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe1662640 1 -- 172.21.15.116:0/756213438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fdc104c50 msgr2=0x7f9fdc198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe1662640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fdc104c50 0x7f9fdc198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe1662640 1 -- 172.21.15.116:0/756213438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fdc107580 msgr2=0x7f9fdc19f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe1662640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fdc107580 0x7f9fdc19f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.998+0000 7f9fe1662640 1 -- 172.21.15.116:0/756213438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fdc19f720 con 0x7f9fdc102320 2024-03-19T01:12:25.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.999+0000 7f9fe1662640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fdc102320 0x7f9fdc197b00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9fd800e9f0 tx=0x7f9fd800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.999+0000 7f9fd27fc640 1 -- 172.21.15.116:0/756213438 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd800cd80 con 0x7f9fdc102320 2024-03-19T01:12:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.999+0000 7f9fd27fc640 1 -- 172.21.15.116:0/756213438 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd8010910 con 0x7f9fdc102320 2024-03-19T01:12:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.999+0000 7f9fd27fc640 1 -- 172.21.15.116:0/756213438 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8018990 con 0x7f9fdc102320 2024-03-19T01:12:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.999+0000 7f9fe38c6640 1 -- 172.21.15.116:0/756213438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fdc19fa10 con 0x7f9fdc102320 2024-03-19T01:12:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:24.999+0000 7f9fe38c6640 1 -- 172.21.15.116:0/756213438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fdc19ff60 con 0x7f9fdc102320 2024-03-19T01:12:25.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.000+0000 7f9fe38c6640 1 -- 172.21.15.116:0/756213438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fdc068a60 con 0x7f9fdc102320 2024-03-19T01:12:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.001+0000 7f9fd27fc640 1 -- 172.21.15.116:0/756213438 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f9fd8010ac0 con 0x7f9fdc102320 2024-03-19T01:12:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.001+0000 7f9fd27fc640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9fc80a2800 0x7f9fc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.001+0000 7f9fe0e61640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9fc80a2800 0x7f9fc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.001+0000 7f9fd27fc640 1 -- 172.21.15.116:0/756213438 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f9fd80cedd0 con 0x7f9fdc102320 2024-03-19T01:12:25.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.002+0000 7f9fe0e61640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9fc80a2800 0x7f9fc80a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc199010 tx=0x7f9fcc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:25.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.006+0000 7f9fd27fc640 1 -- 172.21.15.116:0/756213438 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9fd809b800 con 0x7f9fdc102320 2024-03-19T01:12:25.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.197+0000 7f9fe38c6640 1 -- 172.21.15.116:0/756213438 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9fdc062420 con 0x7f9fdc102320 2024-03-19T01:12:25.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.200+0000 7f9fd27fc640 1 -- 172.21.15.116:0/756213438 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22804 (secure 0 0 0) 0x7f9fd809f8a0 con 0x7f9fdc102320 2024-03-19T01:12:25.201 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:25.201 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:11:28.144841+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:12:25.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.202+0000 7f9fe38c6640 1 -- 172.21.15.116:0/756213438 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9fc80a2800 msgr2=0x7f9fc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.202+0000 7f9fe38c6640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9fc80a2800 0x7f9fc80a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc199010 tx=0x7f9fcc00b410 comp rx=0 tx=0).stop 2024-03-19T01:12:25.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.202+0000 7f9fe38c6640 1 -- 172.21.15.116:0/756213438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fdc102320 msgr2=0x7f9fdc197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.202+0000 7f9fe38c6640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fdc102320 0x7f9fdc197b00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9fd800e9f0 tx=0x7f9fd800eeb0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.202+0000 7f9fe38c6640 1 -- 172.21.15.116:0/756213438 shutdown_connections 2024-03-19T01:12:25.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.202+0000 7f9fe38c6640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9fc80a2800 0x7f9fc80a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.202+0000 7f9fe38c6640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9fdc107580 0x7f9fdc19f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.202+0000 7f9fe38c6640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9fdc104c50 0x7f9fdc198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.202+0000 7f9fe38c6640 1 --2- 172.21.15.116:0/756213438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fdc102320 0x7f9fdc197b00 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.202+0000 7f9fe38c6640 1 -- 172.21.15.116:0/756213438 >> 172.21.15.116:0/756213438 conn(0x7f9fdc0fbcd0 msgr2=0x7f9fdc102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:25.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.202+0000 7f9fe38c6640 1 -- 172.21.15.116:0/756213438 shutdown_connections 2024-03-19T01:12:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.202+0000 7f9fe38c6640 1 -- 172.21.15.116:0/756213438 wait complete. 2024-03-19T01:12:25.215 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:05:53.301933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:10:31.028304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:10:31.704469+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:12:25.215 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:12:25.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.330+0000 7fad2c7df640 1 -- 172.21.15.116:0/994059770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad241022d0 msgr2=0x7fad24108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.330+0000 7fad2c7df640 1 --2- 172.21.15.116:0/994059770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad241022d0 0x7fad24108b60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fad20009ca0 tx=0x7fad2002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 -- 172.21.15.116:0/994059770 shutdown_connections 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 --2- 172.21.15.116:0/994059770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad241022d0 0x7fad24108b60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 --2- 172.21.15.116:0/994059770 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad24101910 0x7fad24101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 --2- 172.21.15.116:0/994059770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad24100720 0x7fad24100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 -- 172.21.15.116:0/994059770 >> 172.21.15.116:0/994059770 conn(0x7fad240fbcb0 msgr2=0x7fad240fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 -- 172.21.15.116:0/994059770 shutdown_connections 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 -- 172.21.15.116:0/994059770 wait complete. 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 Processor -- start 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 -- start start 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad24100720 0x7fad24196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad24101910 0x7fad24196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad241022d0 0x7fad2419dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad2410c3a0 con 0x7fad241022d0 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad2410c220 con 0x7fad24101910 2024-03-19T01:12:25.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2c7df640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad2410c520 con 0x7fad24100720 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2ad7c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad241022d0 0x7fad2419dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2a57b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad24100720 0x7fad24196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2ad7c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad241022d0 0x7fad2419dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35266/0 (socket says 172.21.15.116:35266) 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad2ad7c640 1 -- 172.21.15.116:0/78093895 learned_addr learned my addr 172.21.15.116:0/78093895 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.331+0000 7fad29d7a640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad24101910 0x7fad24196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.332+0000 7fad2a57b640 1 -- 172.21.15.116:0/78093895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad24101910 msgr2=0x7fad24196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.332+0000 7fad2a57b640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad24101910 0x7fad24196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.332+0000 7fad2a57b640 1 -- 172.21.15.116:0/78093895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad241022d0 msgr2=0x7fad2419dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.332+0000 7fad2a57b640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad241022d0 0x7fad2419dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.332+0000 7fad2a57b640 1 -- 172.21.15.116:0/78093895 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2419e630 con 0x7fad24100720 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.332+0000 7fad2ad7c640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad241022d0 0x7fad2419dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.332+0000 7fad2a57b640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad24100720 0x7fad24196a10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00e9c0 tx=0x7fad1c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.332+0000 7fad1b7fe640 1 -- 172.21.15.116:0/78093895 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c00ccc0 con 0x7fad24100720 2024-03-19T01:12:25.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.332+0000 7fad1b7fe640 1 -- 172.21.15.116:0/78093895 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad1c0035f0 con 0x7fad24100720 2024-03-19T01:12:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.332+0000 7fad2c7df640 1 -- 172.21.15.116:0/78093895 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2419e920 con 0x7fad24100720 2024-03-19T01:12:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.332+0000 7fad1b7fe640 1 -- 172.21.15.116:0/78093895 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1c00ce60 con 0x7fad24100720 2024-03-19T01:12:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.332+0000 7fad2c7df640 1 -- 172.21.15.116:0/78093895 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad2419ee70 con 0x7fad24100720 2024-03-19T01:12:25.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.333+0000 7fad2c7df640 1 -- 172.21.15.116:0/78093895 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad24107a40 con 0x7fad24100720 2024-03-19T01:12:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.334+0000 7fad1b7fe640 1 -- 172.21.15.116:0/78093895 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fad1c0037a0 con 0x7fad24100720 2024-03-19T01:12:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.334+0000 7fad1b7fe640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fad100a2880 0x7fad100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.335+0000 7fad29d7a640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fad100a2880 0x7fad100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.335+0000 7fad1b7fe640 1 -- 172.21.15.116:0/78093895 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fad1c0cee80 con 0x7fad24100720 2024-03-19T01:12:25.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.335+0000 7fad29d7a640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fad100a2880 0x7fad100a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fad24197f20 tx=0x7fad14005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:25.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.339+0000 7fad1b7fe640 1 -- 172.21.15.116:0/78093895 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fad1c09bea0 con 0x7fad24100720 2024-03-19T01:12:25.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:25 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1516269471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:12:25.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:25 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/892333803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:12:25.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.518+0000 7fad2c7df640 1 -- 172.21.15.116:0/78093895 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fad24062420 con 0x7fad24100720 2024-03-19T01:12:25.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.519+0000 7fad1b7fe640 1 -- 172.21.15.116:0/78093895 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7fad1c09b7a0 con 0x7fad24100720 2024-03-19T01:12:25.519 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T01:12:25.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.521+0000 7fad2c7df640 1 -- 172.21.15.116:0/78093895 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fad100a2880 msgr2=0x7fad100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.521+0000 7fad2c7df640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fad100a2880 0x7fad100a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fad24197f20 tx=0x7fad14005f90 comp rx=0 tx=0).stop 2024-03-19T01:12:25.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.521+0000 7fad2c7df640 1 -- 172.21.15.116:0/78093895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad24100720 msgr2=0x7fad24196a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.521+0000 7fad2c7df640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad24100720 0x7fad24196a10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fad1c00e9c0 tx=0x7fad1c00ee80 comp rx=0 tx=0).stop 2024-03-19T01:12:25.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.521+0000 7fad2c7df640 1 -- 172.21.15.116:0/78093895 shutdown_connections 2024-03-19T01:12:25.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.521+0000 7fad2c7df640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad241022d0 0x7fad2419dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.521+0000 7fad2c7df640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fad100a2880 0x7fad100a4d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.521+0000 7fad2c7df640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad24101910 0x7fad24196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.521+0000 7fad2c7df640 1 --2- 172.21.15.116:0/78093895 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad24100720 0x7fad24196a10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.521+0000 7fad2c7df640 1 -- 172.21.15.116:0/78093895 >> 172.21.15.116:0/78093895 conn(0x7fad240fbcb0 msgr2=0x7fad24104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:25.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.521+0000 7fad2c7df640 1 -- 172.21.15.116:0/78093895 shutdown_connections 2024-03-19T01:12:25.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.521+0000 7fad2c7df640 1 -- 172.21.15.116:0/78093895 wait complete. 2024-03-19T01:12:25.532 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:25 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1516269471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:12:25.532 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:25 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/892333803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:12:25.533 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:12:25.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.639+0000 7f939a881640 1 -- 172.21.15.116:0/1530741038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9394100720 msgr2=0x7f9394100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.639+0000 7f939a881640 1 --2- 172.21.15.116:0/1530741038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9394100720 0x7f9394100b40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9388009900 tx=0x7f938802bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:25.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.639+0000 7f939a881640 1 -- 172.21.15.116:0/1530741038 shutdown_connections 2024-03-19T01:12:25.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.639+0000 7f939a881640 1 --2- 172.21.15.116:0/1530741038 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93941022d0 0x7f9394108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.639+0000 7f939a881640 1 --2- 172.21.15.116:0/1530741038 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9394101910 0x7f9394101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.639+0000 7f939a881640 1 --2- 172.21.15.116:0/1530741038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9394100720 0x7f9394100b40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.639+0000 7f939a881640 1 -- 172.21.15.116:0/1530741038 >> 172.21.15.116:0/1530741038 conn(0x7f93940fbcb0 msgr2=0x7f93940fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:25.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.639+0000 7f939a881640 1 -- 172.21.15.116:0/1530741038 shutdown_connections 2024-03-19T01:12:25.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.639+0000 7f939a881640 1 -- 172.21.15.116:0/1530741038 wait complete. 2024-03-19T01:12:25.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.640+0000 7f939a881640 1 Processor -- start 2024-03-19T01:12:25.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.640+0000 7f939a881640 1 -- start start 2024-03-19T01:12:25.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.640+0000 7f939a881640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9394100720 0x7f9394197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.640+0000 7f939a881640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9394101910 0x7f9394198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.640+0000 7f939a881640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93941022d0 0x7f939419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.640+0000 7f939a881640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f939410d4c0 con 0x7f9394101910 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.640+0000 7f939a881640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f939410d340 con 0x7f9394100720 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.640+0000 7f939a881640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f939410d640 con 0x7f93941022d0 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.640+0000 7f93937fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9394101910 0x7f9394198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.640+0000 7f9393fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9394100720 0x7f9394197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.640+0000 7f93937fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9394101910 0x7f9394198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35282/0 (socket says 172.21.15.116:35282) 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.640+0000 7f93937fe640 1 -- 172.21.15.116:0/1415398706 learned_addr learned my addr 172.21.15.116:0/1415398706 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.640+0000 7f9398e1e640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93941022d0 0x7f939419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.641+0000 7f9393fff640 1 -- 172.21.15.116:0/1415398706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93941022d0 msgr2=0x7f939419f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.641+0000 7f9393fff640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93941022d0 0x7f939419f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.641+0000 7f9393fff640 1 -- 172.21.15.116:0/1415398706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9394101910 msgr2=0x7f9394198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.641+0000 7f9393fff640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9394101910 0x7f9394198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.641+0000 7f9393fff640 1 -- 172.21.15.116:0/1415398706 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f939419f7e0 con 0x7f9394100720 2024-03-19T01:12:25.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.641+0000 7f93937fe640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9394101910 0x7f9394198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:25.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.641+0000 7f9393fff640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9394100720 0x7f9394197bc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f938802c170 tx=0x7f938802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:25.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.641+0000 7f93917fa640 1 -- 172.21.15.116:0/1415398706 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9388041070 con 0x7f9394100720 2024-03-19T01:12:25.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.641+0000 7f939a881640 1 -- 172.21.15.116:0/1415398706 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f939419fa70 con 0x7f9394100720 2024-03-19T01:12:25.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.641+0000 7f93917fa640 1 -- 172.21.15.116:0/1415398706 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9388004450 con 0x7f9394100720 2024-03-19T01:12:25.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.641+0000 7f939a881640 1 -- 172.21.15.116:0/1415398706 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f939419ff80 con 0x7f9394100720 2024-03-19T01:12:25.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.641+0000 7f93917fa640 1 -- 172.21.15.116:0/1415398706 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9388003e50 con 0x7f9394100720 2024-03-19T01:12:25.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.642+0000 7f939a881640 1 -- 172.21.15.116:0/1415398706 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9394107b60 con 0x7f9394100720 2024-03-19T01:12:25.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.643+0000 7f93917fa640 1 -- 172.21.15.116:0/1415398706 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f9388044070 con 0x7f9394100720 2024-03-19T01:12:25.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.644+0000 7f93917fa640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f937c0a29a0 0x7f937c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.644+0000 7f93937fe640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f937c0a29a0 0x7f937c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.644+0000 7f93917fa640 1 -- 172.21.15.116:0/1415398706 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f93880f0760 con 0x7f9394100720 2024-03-19T01:12:25.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.644+0000 7f93937fe640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f937c0a29a0 0x7f937c0a4e60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f93941990d0 tx=0x7f9384005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:25.648 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:25 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1516269471' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:12:25.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:25 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/892333803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:12:25.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.652+0000 7f93917fa640 1 -- 172.21.15.116:0/1415398706 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f93880bd190 con 0x7f9394100720 2024-03-19T01:12:25.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.831+0000 7f939a881640 1 -- 172.21.15.116:0/1415398706 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f939402d1f0 con 0x7f9394100720 2024-03-19T01:12:25.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.831+0000 7f93917fa640 1 -- 172.21.15.116:0/1415398706 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f93880c1230 con 0x7f9394100720 2024-03-19T01:12:25.832 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T01:12:25.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.833+0000 7f939a881640 1 -- 172.21.15.116:0/1415398706 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f937c0a29a0 msgr2=0x7f937c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.833+0000 7f939a881640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f937c0a29a0 0x7f937c0a4e60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f93941990d0 tx=0x7f9384005f90 comp rx=0 tx=0).stop 2024-03-19T01:12:25.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.833+0000 7f939a881640 1 -- 172.21.15.116:0/1415398706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9394100720 msgr2=0x7f9394197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.833+0000 7f939a881640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9394100720 0x7f9394197bc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f938802c170 tx=0x7f938802c650 comp rx=0 tx=0).stop 2024-03-19T01:12:25.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.834+0000 7f939a881640 1 -- 172.21.15.116:0/1415398706 shutdown_connections 2024-03-19T01:12:25.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.834+0000 7f939a881640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f93941022d0 0x7f939419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.834+0000 7f939a881640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9394101910 0x7f9394198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.834+0000 7f939a881640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f937c0a29a0 0x7f937c0a4e60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.834+0000 7f939a881640 1 --2- 172.21.15.116:0/1415398706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9394100720 0x7f9394197bc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.834+0000 7f939a881640 1 -- 172.21.15.116:0/1415398706 >> 172.21.15.116:0/1415398706 conn(0x7f93940fbcb0 msgr2=0x7f9394104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:25.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.834+0000 7f939a881640 1 -- 172.21.15.116:0/1415398706 shutdown_connections 2024-03-19T01:12:25.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.834+0000 7f939a881640 1 -- 172.21.15.116:0/1415398706 wait complete. 2024-03-19T01:12:25.845 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:12:25.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.949+0000 7fd53c49a640 1 -- 172.21.15.116:0/4255465445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd534102a20 msgr2=0x7fd534102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.949+0000 7fd53c49a640 1 --2- 172.21.15.116:0/4255465445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd534102a20 0x7fd534102ea0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd524009900 tx=0x7fd52402bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:25.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.949+0000 7fd53c49a640 1 -- 172.21.15.116:0/4255465445 shutdown_connections 2024-03-19T01:12:25.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.949+0000 7fd53c49a640 1 --2- 172.21.15.116:0/4255465445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5341033e0 0x7fd534109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.949+0000 7fd53c49a640 1 --2- 172.21.15.116:0/4255465445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd534102a20 0x7fd534102ea0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.949+0000 7fd53c49a640 1 --2- 172.21.15.116:0/4255465445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd534101830 0x7fd534101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.949+0000 7fd53c49a640 1 -- 172.21.15.116:0/4255465445 >> 172.21.15.116:0/4255465445 conn(0x7fd5340fcdc0 msgr2=0x7fd5340ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.949+0000 7fd53c49a640 1 -- 172.21.15.116:0/4255465445 shutdown_connections 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.949+0000 7fd53c49a640 1 -- 172.21.15.116:0/4255465445 wait complete. 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.949+0000 7fd53c49a640 1 Processor -- start 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53c49a640 1 -- start start 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53c49a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd534101830 0x7fd534197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53c49a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd534102a20 0x7fd534198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53c49a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5341033e0 0x7fd53419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53c49a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd53410d500 con 0x7fd534101830 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53c49a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd53410d380 con 0x7fd534102a20 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53c49a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd53410d680 con 0x7fd5341033e0 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53a236640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd534101830 0x7fd534197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd539a35640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd534102a20 0x7fd534198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53aa37640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5341033e0 0x7fd53419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53a236640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd534101830 0x7fd534197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35298/0 (socket says 172.21.15.116:35298) 2024-03-19T01:12:25.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53a236640 1 -- 172.21.15.116:0/842327880 learned_addr learned my addr 172.21.15.116:0/842327880 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:25.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd539a35640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd534102a20 0x7fd534198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:48168/0 (socket says 172.21.15.116:48168) 2024-03-19T01:12:25.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53aa37640 1 -- 172.21.15.116:0/842327880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd534102a20 msgr2=0x7fd534198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53aa37640 1 --2- 172.21.15.116:0/842327880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd534102a20 0x7fd534198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53aa37640 1 -- 172.21.15.116:0/842327880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd534101830 msgr2=0x7fd534197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:25.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53aa37640 1 --2- 172.21.15.116:0/842327880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd534101830 0x7fd534197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:25.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53aa37640 1 -- 172.21.15.116:0/842327880 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53419f740 con 0x7fd5341033e0 2024-03-19T01:12:25.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53a236640 1 --2- 172.21.15.116:0/842327880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd534101830 0x7fd534197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:25.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53aa37640 1 --2- 172.21.15.116:0/842327880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5341033e0 0x7fd53419f0d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd52c009fd0 tx=0x7fd52c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:25.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd52b7fe640 1 -- 172.21.15.116:0/842327880 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c00ce20 con 0x7fd5341033e0 2024-03-19T01:12:25.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd53c49a640 1 -- 172.21.15.116:0/842327880 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd53419fa30 con 0x7fd5341033e0 2024-03-19T01:12:25.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.950+0000 7fd52b7fe640 1 -- 172.21.15.116:0/842327880 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd52c005020 con 0x7fd5341033e0 2024-03-19T01:12:25.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.951+0000 7fd52b7fe640 1 -- 172.21.15.116:0/842327880 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c010720 con 0x7fd5341033e0 2024-03-19T01:12:25.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.951+0000 7fd53c49a640 1 -- 172.21.15.116:0/842327880 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd53419ff50 con 0x7fd5341033e0 2024-03-19T01:12:25.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.952+0000 7fd53c49a640 1 -- 172.21.15.116:0/842327880 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd534108b50 con 0x7fd5341033e0 2024-03-19T01:12:25.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.952+0000 7fd52b7fe640 1 -- 172.21.15.116:0/842327880 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fd52c005540 con 0x7fd5341033e0 2024-03-19T01:12:25.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.953+0000 7fd52b7fe640 1 --2- 172.21.15.116:0/842327880 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd5200a2800 0x7fd5200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:25.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.953+0000 7fd53a236640 1 --2- 172.21.15.116:0/842327880 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd5200a2800 0x7fd5200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:25.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.953+0000 7fd52b7fe640 1 -- 172.21.15.116:0/842327880 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd52c0cf1c0 con 0x7fd5341033e0 2024-03-19T01:12:25.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.953+0000 7fd53a236640 1 --2- 172.21.15.116:0/842327880 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd5200a2800 0x7fd5200a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd5300058c0 tx=0x7fd530005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:25.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:25.957+0000 7fd52b7fe640 1 -- 172.21.15.116:0/842327880 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd52c09c1e0 con 0x7fd5341033e0 2024-03-19T01:12:26.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.210+0000 7fd53c49a640 1 -- 172.21.15.116:0/842327880 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd53402d1f0 con 0x7fd5341033e0 2024-03-19T01:12:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.211+0000 7fd52b7fe640 1 -- 172.21.15.116:0/842327880 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fd52c09bae0 con 0x7fd5341033e0 2024-03-19T01:12:26.212 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T01:12:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.214+0000 7fd53c49a640 1 -- 172.21.15.116:0/842327880 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd5200a2800 msgr2=0x7fd5200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.214+0000 7fd53c49a640 1 --2- 172.21.15.116:0/842327880 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd5200a2800 0x7fd5200a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd5300058c0 tx=0x7fd530005f90 comp rx=0 tx=0).stop 2024-03-19T01:12:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.214+0000 7fd53c49a640 1 -- 172.21.15.116:0/842327880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5341033e0 msgr2=0x7fd53419f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.214+0000 7fd53c49a640 1 --2- 172.21.15.116:0/842327880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5341033e0 0x7fd53419f0d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd52c009fd0 tx=0x7fd52c00ef10 comp rx=0 tx=0).stop 2024-03-19T01:12:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.214+0000 7fd53c49a640 1 -- 172.21.15.116:0/842327880 shutdown_connections 2024-03-19T01:12:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.214+0000 7fd53c49a640 1 --2- 172.21.15.116:0/842327880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd5341033e0 0x7fd53419f0d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.214+0000 7fd53c49a640 1 --2- 172.21.15.116:0/842327880 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd5200a2800 0x7fd5200a4cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.214+0000 7fd53c49a640 1 --2- 172.21.15.116:0/842327880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd534102a20 0x7fd534198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.214+0000 7fd53c49a640 1 --2- 172.21.15.116:0/842327880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd534101830 0x7fd534197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.214+0000 7fd53c49a640 1 -- 172.21.15.116:0/842327880 >> 172.21.15.116:0/842327880 conn(0x7fd5340fcdc0 msgr2=0x7fd534105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.214+0000 7fd53c49a640 1 -- 172.21.15.116:0/842327880 shutdown_connections 2024-03-19T01:12:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.214+0000 7fd53c49a640 1 -- 172.21.15.116:0/842327880 wait complete. 2024-03-19T01:12:26.226 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:12:26.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.327+0000 7fc67e177640 1 -- 172.21.15.116:0/3097722822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc678107520 msgr2=0x7fc678109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:26.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.327+0000 7fc67e177640 1 --2- 172.21.15.116:0/3097722822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc678107520 0x7fc678109970 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc670009ca0 tx=0x7fc67002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.327+0000 7fc67e177640 1 -- 172.21.15.116:0/3097722822 shutdown_connections 2024-03-19T01:12:26.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.327+0000 7fc67e177640 1 --2- 172.21.15.116:0/3097722822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc678107520 0x7fc678109970 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.327+0000 7fc67e177640 1 --2- 172.21.15.116:0/3097722822 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc678104bf0 0x7fc678106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.327+0000 7fc67e177640 1 --2- 172.21.15.116:0/3097722822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc6781022c0 0x7fc6781046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.327+0000 7fc67e177640 1 -- 172.21.15.116:0/3097722822 >> 172.21.15.116:0/3097722822 conn(0x7fc6780fbcb0 msgr2=0x7fc6780fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:26.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.328+0000 7fc67e177640 1 -- 172.21.15.116:0/3097722822 shutdown_connections 2024-03-19T01:12:26.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.328+0000 7fc67e177640 1 -- 172.21.15.116:0/3097722822 wait complete. 2024-03-19T01:12:26.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.328+0000 7fc67e177640 1 Processor -- start 2024-03-19T01:12:26.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.328+0000 7fc67e177640 1 -- start start 2024-03-19T01:12:26.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.328+0000 7fc67e177640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6781022c0 0x7fc678197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:26.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.328+0000 7fc67e177640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc678104bf0 0x7fc678198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:26.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.328+0000 7fc67e177640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc678107520 0x7fc67819f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:26.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.328+0000 7fc67e177640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc67810d4e0 con 0x7fc678107520 2024-03-19T01:12:26.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.328+0000 7fc67e177640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc67810d360 con 0x7fc678104bf0 2024-03-19T01:12:26.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.328+0000 7fc67e177640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc67810d660 con 0x7fc6781022c0 2024-03-19T01:12:26.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.328+0000 7fc677fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc678107520 0x7fc67819f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:26.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc677fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc678107520 0x7fc67819f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35310/0 (socket says 172.21.15.116:35310) 2024-03-19T01:12:26.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc676ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc678104bf0 0x7fc678198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:26.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc677fff640 1 -- 172.21.15.116:0/1175247167 learned_addr learned my addr 172.21.15.116:0/1175247167 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:26.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc6777fe640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6781022c0 0x7fc678197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:26.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc677fff640 1 -- 172.21.15.116:0/1175247167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6781022c0 msgr2=0x7fc678197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:26.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc677fff640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6781022c0 0x7fc678197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc677fff640 1 -- 172.21.15.116:0/1175247167 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc678104bf0 msgr2=0x7fc678198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:26.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc677fff640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc678104bf0 0x7fc678198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc677fff640 1 -- 172.21.15.116:0/1175247167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc67819f720 con 0x7fc678107520 2024-03-19T01:12:26.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc676ffd640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc678104bf0 0x7fc678198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:26.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc677fff640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc678107520 0x7fc67819f0b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc670037040 tx=0x7fc670032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:26.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc674ff9640 1 -- 172.21.15.116:0/1175247167 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670041020 con 0x7fc678107520 2024-03-19T01:12:26.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc67e177640 1 -- 172.21.15.116:0/1175247167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc67819f9b0 con 0x7fc678107520 2024-03-19T01:12:26.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc674ff9640 1 -- 172.21.15.116:0/1175247167 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc670034cd0 con 0x7fc678107520 2024-03-19T01:12:26.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc674ff9640 1 -- 172.21.15.116:0/1175247167 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670039040 con 0x7fc678107520 2024-03-19T01:12:26.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.329+0000 7fc67e177640 1 -- 172.21.15.116:0/1175247167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc67819fef0 con 0x7fc678107520 2024-03-19T01:12:26.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.330+0000 7fc67e177640 1 -- 172.21.15.116:0/1175247167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6780fd7a0 con 0x7fc678107520 2024-03-19T01:12:26.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.331+0000 7fc674ff9640 1 -- 172.21.15.116:0/1175247167 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fc670044020 con 0x7fc678107520 2024-03-19T01:12:26.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.332+0000 7fc674ff9640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6600a2800 0x7fc6600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:26.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.332+0000 7fc6777fe640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6600a2800 0x7fc6600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:26.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.332+0000 7fc674ff9640 1 -- 172.21.15.116:0/1175247167 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc6700f0940 con 0x7fc678107520 2024-03-19T01:12:26.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.332+0000 7fc6777fe640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6600a2800 0x7fc6600a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc668009bd0 tx=0x7fc668005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:26.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.336+0000 7fc674ff9640 1 -- 172.21.15.116:0/1175247167 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc67004b020 con 0x7fc678107520 2024-03-19T01:12:26.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:26 smithi156 ceph-mon[66799]: pgmap v346: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:12:26.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:26 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/756213438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:12:26.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:26 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/78093895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:12:26.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:26 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1415398706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:12:26.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.596+0000 7fc67e177640 1 -- 172.21.15.116:0/1175247167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc678062420 con 0x7fc678107520 2024-03-19T01:12:26.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.596+0000 7fc674ff9640 1 -- 172.21.15.116:0/1175247167 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11938 (secure 0 0 0) 0x7fc6700bda50 con 0x7fc678107520 2024-03-19T01:12:26.598 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:26.598 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T01:10:32.487607+0000","modified":"2024-03-19T01:10:38.982004+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24488,"mds_2":24473,"mds_3":24460,"mds_4":24445},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1242639081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1242639081},{"type":"v1","addr":"172.21.15.116:6839","nonce":1242639081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24394":{"gid":24394,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":3,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.162:6839/441486392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":441486392},{"type":"v1","addr":"172.21.15.162:6839","nonce":441486392}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"a","rank":1,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6841/3343188504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3343188504},{"type":"v1","addr":"172.21.15.116:6841","nonce":3343188504}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1601537427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1601537427},{"type":"v1","addr":"172.21.15.156:6839","nonce":1601537427}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24473,24445,24500,24460,24488]},"id":1}]} 2024-03-19T01:12:26.599 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2024-03-19T01:12:26.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.598+0000 7fc67e177640 1 -- 172.21.15.116:0/1175247167 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6600a2800 msgr2=0x7fc6600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:26.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.598+0000 7fc67e177640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6600a2800 0x7fc6600a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc668009bd0 tx=0x7fc668005f90 comp rx=0 tx=0).stop 2024-03-19T01:12:26.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.598+0000 7fc67e177640 1 -- 172.21.15.116:0/1175247167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc678107520 msgr2=0x7fc67819f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:26.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.598+0000 7fc67e177640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc678107520 0x7fc67819f0b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc670037040 tx=0x7fc670032f40 comp rx=0 tx=0).stop 2024-03-19T01:12:26.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.598+0000 7fc67e177640 1 -- 172.21.15.116:0/1175247167 shutdown_connections 2024-03-19T01:12:26.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.598+0000 7fc67e177640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc678107520 0x7fc67819f0b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.598+0000 7fc67e177640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6600a2800 0x7fc6600a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.598+0000 7fc67e177640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc678104bf0 0x7fc678198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.598+0000 7fc67e177640 1 --2- 172.21.15.116:0/1175247167 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc6781022c0 0x7fc678197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.598+0000 7fc67e177640 1 -- 172.21.15.116:0/1175247167 >> 172.21.15.116:0/1175247167 conn(0x7fc6780fbcb0 msgr2=0x7fc678102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:26.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.599+0000 7fc67e177640 1 -- 172.21.15.116:0/1175247167 shutdown_connections 2024-03-19T01:12:26.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.599+0000 7fc67e177640 1 -- 172.21.15.116:0/1175247167 wait complete. 2024-03-19T01:12:26.611 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:26 smithi116 ceph-mon[62701]: pgmap v346: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:12:26.611 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:26 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/756213438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:12:26.611 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:26 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/78093895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:12:26.611 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:26 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1415398706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:12:26.613 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:12:26.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:26 smithi162 ceph-mon[67178]: pgmap v346: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:12:26.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:26 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/756213438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:12:26.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:26 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/78093895' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:12:26.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:26 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1415398706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:12:26.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 -- 172.21.15.116:0/901885122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ee4101860 msgr2=0x7f0ee4101c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:26.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 --2- 172.21.15.116:0/901885122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ee4101860 0x7f0ee4101c80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0009860 tx=0x7f0ee002bb70 comp rx=0 tx=0).stop 2024-03-19T01:12:26.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 -- 172.21.15.116:0/901885122 shutdown_connections 2024-03-19T01:12:26.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 --2- 172.21.15.116:0/901885122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ee4103410 0x7f0ee4109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 --2- 172.21.15.116:0/901885122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee4102a50 0x7f0ee4102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 --2- 172.21.15.116:0/901885122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ee4101860 0x7f0ee4101c80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 -- 172.21.15.116:0/901885122 >> 172.21.15.116:0/901885122 conn(0x7f0ee40fcdd0 msgr2=0x7f0ee40ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:26.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 -- 172.21.15.116:0/901885122 shutdown_connections 2024-03-19T01:12:26.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 -- 172.21.15.116:0/901885122 wait complete. 2024-03-19T01:12:26.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 Processor -- start 2024-03-19T01:12:26.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 -- start start 2024-03-19T01:12:26.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ee4101860 0x7f0ee4197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:26.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ee4102a50 0x7f0ee4198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:26.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee4103410 0x7f0ee419f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:26.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee410d4b0 con 0x7f0ee4103410 2024-03-19T01:12:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee410d330 con 0x7f0ee4102a50 2024-03-19T01:12:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.717+0000 7f0eebff4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee410d630 con 0x7f0ee4101860 2024-03-19T01:12:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0eea591640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee4103410 0x7f0ee419f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0ee9d90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ee4101860 0x7f0ee4197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0ee958f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ee4102a50 0x7f0ee4198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0ee9d90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ee4101860 0x7f0ee4197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54390/0 (socket says 172.21.15.116:54390) 2024-03-19T01:12:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0ee9d90640 1 -- 172.21.15.116:0/1665202576 learned_addr learned my addr 172.21.15.116:0/1665202576 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0ee9d90640 1 -- 172.21.15.116:0/1665202576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ee4102a50 msgr2=0x7f0ee4198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0ee9d90640 1 --2- 172.21.15.116:0/1665202576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ee4102a50 0x7f0ee4198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0ee9d90640 1 -- 172.21.15.116:0/1665202576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee4103410 msgr2=0x7f0ee419f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0ee9d90640 1 --2- 172.21.15.116:0/1665202576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee4103410 0x7f0ee419f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0ee9d90640 1 -- 172.21.15.116:0/1665202576 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee419f7f0 con 0x7f0ee4101860 2024-03-19T01:12:26.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0eea591640 1 --2- 172.21.15.116:0/1665202576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee4103410 0x7f0ee419f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:26.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0ee9d90640 1 --2- 172.21.15.116:0/1665202576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ee4101860 0x7f0ee4197bd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0ee002c0b0 tx=0x7f0ee0032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:26.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0edaffd640 1 -- 172.21.15.116:0/1665202576 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0041020 con 0x7f0ee4101860 2024-03-19T01:12:26.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0eebff4640 1 -- 172.21.15.116:0/1665202576 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee419fa80 con 0x7f0ee4101860 2024-03-19T01:12:26.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0edaffd640 1 -- 172.21.15.116:0/1665202576 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ee002c9f0 con 0x7f0ee4101860 2024-03-19T01:12:26.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0edaffd640 1 -- 172.21.15.116:0/1665202576 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0039040 con 0x7f0ee4101860 2024-03-19T01:12:26.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.718+0000 7f0eebff4640 1 -- 172.21.15.116:0/1665202576 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee419ff70 con 0x7f0ee4101860 2024-03-19T01:12:26.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.719+0000 7f0eebff4640 1 -- 172.21.15.116:0/1665202576 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ee4108c50 con 0x7f0ee4101860 2024-03-19T01:12:26.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.720+0000 7f0edaffd640 1 -- 172.21.15.116:0/1665202576 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f0ee0044020 con 0x7f0ee4101860 2024-03-19T01:12:26.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.721+0000 7f0edaffd640 1 --2- 172.21.15.116:0/1665202576 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0ed00a2800 0x7f0ed00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:26.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.721+0000 7f0ee958f640 1 --2- 172.21.15.116:0/1665202576 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0ed00a2800 0x7f0ed00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:26.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.721+0000 7f0edaffd640 1 -- 172.21.15.116:0/1665202576 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0ee00f07a0 con 0x7f0ee4101860 2024-03-19T01:12:26.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.721+0000 7f0ee958f640 1 --2- 172.21.15.116:0/1665202576 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0ed00a2800 0x7f0ed00a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0ee41990e0 tx=0x7f0ed4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:26.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.724+0000 7f0edaffd640 1 -- 172.21.15.116:0/1665202576 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0ee004b020 con 0x7f0ee4101860 2024-03-19T01:12:26.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.929+0000 7f0eebff4640 1 -- 172.21.15.116:0/1665202576 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0ee402d440 con 0x7f0ee4101860 2024-03-19T01:12:26.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.930+0000 7f0edaffd640 1 -- 172.21.15.116:0/1665202576 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22804 (secure 0 0 0) 0x7f0ee00bd820 con 0x7f0ee4101860 2024-03-19T01:12:26.932 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:26.932 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:11:28.144841+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:12:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.932+0000 7f0eebff4640 1 -- 172.21.15.116:0/1665202576 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0ed00a2800 msgr2=0x7f0ed00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.932+0000 7f0eebff4640 1 --2- 172.21.15.116:0/1665202576 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0ed00a2800 0x7f0ed00a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0ee41990e0 tx=0x7f0ed4009260 comp rx=0 tx=0).stop 2024-03-19T01:12:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.932+0000 7f0eebff4640 1 -- 172.21.15.116:0/1665202576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ee4101860 msgr2=0x7f0ee4197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.932+0000 7f0eebff4640 1 --2- 172.21.15.116:0/1665202576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ee4101860 0x7f0ee4197bd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0ee002c0b0 tx=0x7f0ee0032430 comp rx=0 tx=0).stop 2024-03-19T01:12:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.932+0000 7f0eebff4640 1 -- 172.21.15.116:0/1665202576 shutdown_connections 2024-03-19T01:12:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.932+0000 7f0eebff4640 1 --2- 172.21.15.116:0/1665202576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee4103410 0x7f0ee419f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.932+0000 7f0eebff4640 1 --2- 172.21.15.116:0/1665202576 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0ed00a2800 0x7f0ed00a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.932+0000 7f0eebff4640 1 --2- 172.21.15.116:0/1665202576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ee4102a50 0x7f0ee4198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.932+0000 7f0eebff4640 1 --2- 172.21.15.116:0/1665202576 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0ee4101860 0x7f0ee4197bd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.932+0000 7f0eebff4640 1 -- 172.21.15.116:0/1665202576 >> 172.21.15.116:0/1665202576 conn(0x7f0ee40fcdd0 msgr2=0x7f0ee4105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.932+0000 7f0eebff4640 1 -- 172.21.15.116:0/1665202576 shutdown_connections 2024-03-19T01:12:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:26.932+0000 7f0eebff4640 1 -- 172.21.15.116:0/1665202576 wait complete. 2024-03-19T01:12:26.945 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-19T01:10:32.487607+0000', 'modified': '2024-03-19T01:10:38.982004+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24500, 'mds_1': 24488, 'mds_2': 24473, 'mds_3': 24460, 'mds_4': 24445}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14613': {'gid': 14613, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.116:6839/1242639081', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6838', 'nonce': 1242639081}, {'type': 'v1', 'addr': '172.21.15.116:6839', 'nonce': 1242639081}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24394': {'gid': 24394, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6833/3113382416', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6832', 'nonce': 3113382416}, {'type': 'v1', 'addr': '172.21.15.162:6833', 'nonce': 3113382416}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24410': {'gid': 24410, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6833/1647822466', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6832', 'nonce': 1647822466}, {'type': 'v1', 'addr': '172.21.15.156:6833', 'nonce': 1647822466}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24427': {'gid': 24427, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.162:6835/1592939032', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6834', 'nonce': 1592939032}, {'type': 'v1', 'addr': '172.21.15.162:6835', 'nonce': 1592939032}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24434': {'gid': 24434, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6835/1738017780', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6834', 'nonce': 1738017780}, {'type': 'v1', 'addr': '172.21.15.156:6835', 'nonce': 1738017780}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'c', 'rank': 4, 'incarnation': 22, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.162:6837/585664575', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6836', 'nonce': 585664575}, {'type': 'v1', 'addr': '172.21.15.162:6837', 'nonce': 585664575}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'i', 'rank': 3, 'incarnation': 20, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.162:6839/441486392', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6838', 'nonce': 441486392}, {'type': 'v1', 'addr': '172.21.15.162:6839', 'nonce': 441486392}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.156:6837/4147060396', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6836', 'nonce': 4147060396}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 4147060396}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'a', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.116:6841/3343188504', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6840', 'nonce': 3343188504}, {'type': 'v1', 'addr': '172.21.15.116:6841', 'nonce': 3343188504}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.156:6839/1601537427', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 1601537427}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 1601537427}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24500, 'qdb_cluster': [24473, 24445, 24500, 24460, 24488]} 2024-03-19T01:12:26.945 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-19T01:12:26.945 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-19T01:12:27.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.048+0000 7f7af2a3f640 1 -- 172.21.15.116:0/1647426597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec101830 msgr2=0x7f7aec101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.048+0000 7f7af2a3f640 1 --2- 172.21.15.116:0/1647426597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec101830 0x7f7aec101c50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7adc009900 tx=0x7f7adc02bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:27.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.048+0000 7f7af2a3f640 1 -- 172.21.15.116:0/1647426597 shutdown_connections 2024-03-19T01:12:27.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.048+0000 7f7af2a3f640 1 --2- 172.21.15.116:0/1647426597 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7aec1033e0 0x7f7aec109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.048+0000 7f7af2a3f640 1 --2- 172.21.15.116:0/1647426597 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aec102a20 0x7f7aec102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.048+0000 7f7af2a3f640 1 --2- 172.21.15.116:0/1647426597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec101830 0x7f7aec101c50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.048+0000 7f7af2a3f640 1 -- 172.21.15.116:0/1647426597 >> 172.21.15.116:0/1647426597 conn(0x7f7aec0fcdc0 msgr2=0x7f7aec0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.048+0000 7f7af2a3f640 1 -- 172.21.15.116:0/1647426597 shutdown_connections 2024-03-19T01:12:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.048+0000 7f7af2a3f640 1 -- 172.21.15.116:0/1647426597 wait complete. 2024-03-19T01:12:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af2a3f640 1 Processor -- start 2024-03-19T01:12:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af2a3f640 1 -- start start 2024-03-19T01:12:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af2a3f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aec101830 0x7f7aec199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af2a3f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7aec102a20 0x7f7aec19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af2a3f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec1033e0 0x7f7aec1a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af2a3f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec074060 con 0x7f7aec1033e0 2024-03-19T01:12:27.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af2a3f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec073ee0 con 0x7f7aec101830 2024-03-19T01:12:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af2a3f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec0741e0 con 0x7f7aec102a20 2024-03-19T01:12:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af0fdc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec1033e0 0x7f7aec1a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af0fdc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec1033e0 0x7f7aec1a1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52610/0 (socket says 172.21.15.116:52610) 2024-03-19T01:12:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af0fdc640 1 -- 172.21.15.116:0/1559969926 learned_addr learned my addr 172.21.15.116:0/1559969926 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7aebfff640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aec101830 0x7f7aec199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7aec102a20 0x7f7aec19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af0fdc640 1 -- 172.21.15.116:0/1559969926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7aec102a20 msgr2=0x7f7aec19a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af0fdc640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7aec102a20 0x7f7aec19a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af0fdc640 1 -- 172.21.15.116:0/1559969926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aec101830 msgr2=0x7f7aec199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af0fdc640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aec101830 0x7f7aec199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7af0fdc640 1 -- 172.21.15.116:0/1559969926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aec1a19e0 con 0x7f7aec1033e0 2024-03-19T01:12:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.049+0000 7f7aebfff640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aec101830 0x7f7aec199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.050+0000 7f7af0fdc640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec1033e0 0x7f7aec1a1370 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400eb10 tx=0x7f7ae400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.050+0000 7f7ae97fa640 1 -- 172.21.15.116:0/1559969926 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4004b20 con 0x7f7aec1033e0 2024-03-19T01:12:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.050+0000 7f7af2a3f640 1 -- 172.21.15.116:0/1559969926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aec1a1cd0 con 0x7f7aec1033e0 2024-03-19T01:12:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.050+0000 7f7ae97fa640 1 -- 172.21.15.116:0/1559969926 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ae4004cc0 con 0x7f7aec1033e0 2024-03-19T01:12:27.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.050+0000 7f7ae97fa640 1 -- 172.21.15.116:0/1559969926 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4010760 con 0x7f7aec1033e0 2024-03-19T01:12:27.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.050+0000 7f7af2a3f640 1 -- 172.21.15.116:0/1559969926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aec1a21c0 con 0x7f7aec1033e0 2024-03-19T01:12:27.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.052+0000 7f7af5620740 1 Processor -- start 2024-03-19T01:12:27.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7ae97fa640 1 -- 172.21.15.116:0/1559969926 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7ae4005020 con 0x7f7aec1033e0 2024-03-19T01:12:27.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7af5620740 1 -- start start 2024-03-19T01:12:27.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7af5620740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 0x55753d259410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7af5620740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d2599e0 0x55753d25e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7af5620740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d25eb50 0x55753d260fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7af5620740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55753d1bdde0 con 0x55753d258ff0 2024-03-19T01:12:27.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7af5620740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55753d166c80 con 0x55753d2599e0 2024-03-19T01:12:27.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7af5620740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55753d1b9e80 con 0x55753d25eb50 2024-03-19T01:12:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 0x55753d259410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 0x55753d259410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52626/0 (socket says 172.21.15.116:52626) 2024-03-19T01:12:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/3992090545 learned_addr learned my addr 172.21.15.116:0/3992090545 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7aebfff640 1 --2- 172.21.15.116:0/3992090545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d2599e0 0x55753d25e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/3992090545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d25eb50 0x55753d260fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7ae97fa640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ad40a2800 0x7f7ad40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/3992090545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d25eb50 msgr2=0x55753d260fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/3992090545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d25eb50 0x55753d260fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/3992090545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d2599e0 msgr2=0x55753d25e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/3992090545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d2599e0 0x55753d25e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.053+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/3992090545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55753d178fb0 con 0x55753d258ff0 2024-03-19T01:12:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.054+0000 7f7af0fdc640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ad40a2800 0x7f7ad40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.054+0000 7f7aebfff640 1 --2- 172.21.15.116:0/3992090545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d2599e0 0x55753d25e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.054+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/3992090545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 0x55753d259410 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0005bc0 tx=0x7f7ae0009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=453ea3a991dc591f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.054+0000 7f7ae97fa640 1 -- 172.21.15.116:0/1559969926 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7ae40cf140 con 0x7f7aec1033e0 2024-03-19T01:12:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.054+0000 7f7ad27fc640 1 -- 172.21.15.116:0/3992090545 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0013130 con 0x55753d258ff0 2024-03-19T01:12:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.054+0000 7f7ad27fc640 1 -- 172.21.15.116:0/3992090545 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ae0013e50 con 0x55753d258ff0 2024-03-19T01:12:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.054+0000 7f7ad27fc640 1 -- 172.21.15.116:0/3992090545 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae00047a0 con 0x55753d258ff0 2024-03-19T01:12:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 -- 172.21.15.116:0/3992090545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 msgr2=0x55753d259410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af0fdc640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ad40a2800 0x7f7ad40a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4097f20 tx=0x7f7ae4097640 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 --2- 172.21.15.116:0/3992090545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 0x55753d259410 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0005bc0 tx=0x7f7ae0009850 comp rx=0 tx=0).stop 2024-03-19T01:12:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 -- 172.21.15.116:0/3992090545 shutdown_connections 2024-03-19T01:12:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 --2- 172.21.15.116:0/3992090545 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d25eb50 0x55753d260fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 --2- 172.21.15.116:0/3992090545 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d2599e0 0x55753d25e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 --2- 172.21.15.116:0/3992090545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 0x55753d259410 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 -- 172.21.15.116:0/3992090545 >> 172.21.15.116:0/3992090545 conn(0x55753d254560 msgr2=0x55753d254960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 -- 172.21.15.116:0/3992090545 shutdown_connections 2024-03-19T01:12:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 -- 172.21.15.116:0/3992090545 wait complete. 2024-03-19T01:12:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 -- start start 2024-03-19T01:12:27.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 Processor -- start 2024-03-19T01:12:27.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d258ff0 0x55753d263420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 0x55753d263960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d25eb50 0x55753d26a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55753d261aa0 con 0x55753d25eb50 2024-03-19T01:12:27.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55753d179fc0 con 0x55753d258ff0 2024-03-19T01:12:27.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55753d261c20 con 0x55753d2599e0 2024-03-19T01:12:27.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.055+0000 7f7af5620740 10 client.0 _ll_register_callbacks cb 0x55753d1b5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:27.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d258ff0 0x55753d263420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d258ff0 0x55753d263420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:36652/0 (socket says 172.21.15.116:36652) 2024-03-19T01:12:27.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/2474740751 learned_addr learned my addr 172.21.15.116:0/2474740751 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7aebfff640 1 --2- 172.21.15.116:0/2474740751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 0x55753d263960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/2474740751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d25eb50 0x55753d26a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/2474740751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 msgr2=0x55753d263960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/2474740751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 0x55753d263960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/2474740751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d25eb50 msgr2=0x55753d26a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/2474740751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d25eb50 0x55753d26a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/2474740751 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55753ced22d0 con 0x55753d258ff0 2024-03-19T01:12:27.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/2474740751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d258ff0 0x55753d263420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00057b0 tx=0x7f7ae0005020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=779b49dc89b0f22b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7acb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:12:27.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7acb7fe640 1 -- 172.21.15.116:0/2474740751 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0004030 con 0x55753d258ff0 2024-03-19T01:12:27.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7acb7fe640 1 -- 172.21.15.116:0/2474740751 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7ae00041d0 con 0x55753d258ff0 2024-03-19T01:12:27.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7af5620740 1 -- 172.21.15.116:0/2474740751 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ae0013e50 con 0x55753d258ff0 2024-03-19T01:12:27.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7acb7fe640 1 -- 172.21.15.116:0/2474740751 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7ae00043d0 con 0x55753d258ff0 2024-03-19T01:12:27.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7af5620740 10 client.25091 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:27.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.056+0000 7f7af5620740 1 -- 172.21.15.116:0/2474740751 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55753cf9c6b0 con 0x55753d258ff0 2024-03-19T01:12:27.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.057+0000 7f7acb7fe640 1 -- 172.21.15.116:0/2474740751 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ae0013e50 con 0x55753d258ff0 2024-03-19T01:12:27.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.057+0000 7f7acb7fe640 1 -- 172.21.15.116:0/2474740751 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f7ae0004a80 con 0x55753d258ff0 2024-03-19T01:12:27.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.057+0000 7f7af5620740 10 client.25091 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:27.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.057+0000 7f7af5620740 10 client.25091 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-19T01:12:27.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.057+0000 7f7af5620740 20 client.25091 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:27.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.057+0000 7f7af5620740 1 --2- 172.21.15.116:0/2474740751 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x55753d26fdf0 0x55753d271260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.057+0000 7f7af5620740 4 client.25091 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:12:27.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.057+0000 7f7af5620740 1 -- 172.21.15.116:0/2474740751 --> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55753d1cb320 con 0x55753d26fdf0 2024-03-19T01:12:27.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.057+0000 7f7aebfff640 1 --2- 172.21.15.116:0/2474740751 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x55753d26fdf0 0x55753d271260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.057+0000 7f7acb7fe640 10 client.25091 ms_handle_connect on v2:172.21.15.162:6836/585664575 2024-03-19T01:12:27.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.057+0000 7f7aebfff640 1 --2- 172.21.15.116:0/2474740751 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x55753d26fdf0 0x55753d271260 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f182d4f7c2d0418e server_cookie=2aa7f875c8b2a184 in_seq=0 out_seq=0 2024-03-19T01:12:27.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7acb7fe640 1 -- 172.21.15.116:0/2474740751 <== mds.4 v2:172.21.15.162:6836/585664575 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55753d1cb320 con 0x55753d26fdf0 2024-03-19T01:12:27.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7acb7fe640 10 client.25091 handle_command_reply: tid=0 2024-03-19T01:12:27.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7af5620740 1 client.25091 shutdown 2024-03-19T01:12:27.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7af5620740 1 -- 172.21.15.116:0/2474740751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d258ff0 msgr2=0x55753d263420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7af5620740 1 --2- 172.21.15.116:0/2474740751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d258ff0 0x55753d263420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00057b0 tx=0x7f7ae0005020 comp rx=0 tx=0).stop 2024-03-19T01:12:27.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7af5620740 1 -- 172.21.15.116:0/2474740751 shutdown_connections 2024-03-19T01:12:27.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7af5620740 1 --2- 172.21.15.116:0/2474740751 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x55753d26fdf0 0x55753d271260 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7af5620740 1 --2- 172.21.15.116:0/2474740751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d25eb50 0x55753d26a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7acb7fe640 0 client.25091 ms_handle_reset on v2:172.21.15.162:6836/585664575 2024-03-19T01:12:27.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7af5620740 1 --2- 172.21.15.116:0/2474740751 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 0x55753d263960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7af5620740 1 --2- 172.21.15.116:0/2474740751 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d258ff0 0x55753d263420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7af5620740 1 -- 172.21.15.116:0/2474740751 >> 172.21.15.116:0/2474740751 conn(0x55753d254560 msgr2=0x55753d2537e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7af5620740 1 -- 172.21.15.116:0/2474740751 shutdown_connections 2024-03-19T01:12:27.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7af5620740 1 -- 172.21.15.116:0/2474740751 wait complete. 2024-03-19T01:12:27.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.058+0000 7f7af5620740 20 client.25091 trim_cache size 0 max 16384 2024-03-19T01:12:27.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7af5620740 1 Processor -- start 2024-03-19T01:12:27.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7af5620740 1 -- start start 2024-03-19T01:12:27.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7af5620740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 0x55753d262a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7af5620740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 0x55753d26a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7af5620740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d25eb50 0x55753d25ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7af5620740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55753d26f5a0 con 0x55753d258ff0 2024-03-19T01:12:27.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7af5620740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55753d26fc70 con 0x55753d25eb50 2024-03-19T01:12:27.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7af5620740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55753d270340 con 0x55753d2599e0 2024-03-19T01:12:27.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 0x55753d262a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 0x55753d262a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52648/0 (socket says 172.21.15.116:52648) 2024-03-19T01:12:27.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/422785756 learned_addr learned my addr 172.21.15.116:0/422785756 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7aebfff640 1 --2- 172.21.15.116:0/422785756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 0x55753d26a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/422785756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d25eb50 0x55753d25ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/422785756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 msgr2=0x55753d26a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/422785756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 0x55753d26a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/422785756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d25eb50 msgr2=0x55753d25ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/422785756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d25eb50 0x55753d25ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/422785756 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55753d259dc0 con 0x55753d258ff0 2024-03-19T01:12:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.080+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/422785756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 0x55753d262a40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0005130 tx=0x7f7ae0013130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a3fc12df670dfe server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.081+0000 7f7ad1ffb640 1 -- 172.21.15.116:0/422785756 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0005b90 con 0x55753d258ff0 2024-03-19T01:12:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.081+0000 7f7ad1ffb640 1 -- 172.21.15.116:0/422785756 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ae0036070 con 0x55753d258ff0 2024-03-19T01:12:27.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.081+0000 7f7ad1ffb640 1 -- 172.21.15.116:0/422785756 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae00048f0 con 0x55753d258ff0 2024-03-19T01:12:27.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.081+0000 7f7af5620740 1 -- 172.21.15.116:0/422785756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 msgr2=0x55753d262a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.081+0000 7f7af5620740 1 --2- 172.21.15.116:0/422785756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 0x55753d262a40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0005130 tx=0x7f7ae0013130 comp rx=0 tx=0).stop 2024-03-19T01:12:27.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.081+0000 7f7af5620740 1 -- 172.21.15.116:0/422785756 shutdown_connections 2024-03-19T01:12:27.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.081+0000 7f7af5620740 1 --2- 172.21.15.116:0/422785756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d25eb50 0x55753d25ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.081+0000 7f7af5620740 1 --2- 172.21.15.116:0/422785756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 0x55753d26a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.081+0000 7f7af5620740 1 --2- 172.21.15.116:0/422785756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d258ff0 0x55753d262a40 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.081+0000 7f7af5620740 1 -- 172.21.15.116:0/422785756 >> 172.21.15.116:0/422785756 conn(0x55753d254560 msgr2=0x55753d261fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.081+0000 7f7af5620740 1 -- 172.21.15.116:0/422785756 shutdown_connections 2024-03-19T01:12:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.081+0000 7f7af5620740 1 -- 172.21.15.116:0/422785756 wait complete. 2024-03-19T01:12:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7af5620740 1 -- start start 2024-03-19T01:12:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7af5620740 1 Processor -- start 2024-03-19T01:12:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7af5620740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d258ff0 0x55753d265df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7af5620740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 0x55753d266330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7af5620740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d25eb50 0x55753d269350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7af5620740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55753d2710e0 con 0x55753d25eb50 2024-03-19T01:12:27.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7af5620740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55753d270a10 con 0x55753d258ff0 2024-03-19T01:12:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7af5620740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55753d2717b0 con 0x55753d2599e0 2024-03-19T01:12:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7af5620740 10 client.0 _ll_register_callbacks cb 0x7f7ae40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d25eb50 0x55753d269350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d25eb50 0x55753d269350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52662/0 (socket says 172.21.15.116:52662) 2024-03-19T01:12:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/1725237237 learned_addr learned my addr 172.21.15.116:0/1725237237 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7aebfff640 1 --2- 172.21.15.116:0/1725237237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 0x55753d266330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/1725237237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d258ff0 0x55753d265df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/1725237237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 msgr2=0x55753d266330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/1725237237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 0x55753d266330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/1725237237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d258ff0 msgr2=0x55753d265df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/1725237237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d258ff0 0x55753d265df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.082+0000 7f7aeb7fe640 1 -- 172.21.15.116:0/1725237237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55753d269940 con 0x55753d25eb50 2024-03-19T01:12:27.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7aeb7fe640 1 --2- 172.21.15.116:0/1725237237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d25eb50 0x55753d269350 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00058c0 tx=0x7f7ae0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456f47f783ffc1dd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7acb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7acb7fe640 1 -- 172.21.15.116:0/1725237237 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0004a80 con 0x55753d25eb50 2024-03-19T01:12:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7acb7fe640 1 -- 172.21.15.116:0/1725237237 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7ae0020470 con 0x55753d25eb50 2024-03-19T01:12:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7af5620740 1 -- 172.21.15.116:0/1725237237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ae0036070 con 0x55753d25eb50 2024-03-19T01:12:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7acb7fe640 1 -- 172.21.15.116:0/1725237237 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7ae0005b90 con 0x55753d25eb50 2024-03-19T01:12:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7af5620740 10 client.15255 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7af5620740 1 -- 172.21.15.116:0/1725237237 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55753d26ac40 con 0x55753d25eb50 2024-03-19T01:12:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7acb7fe640 1 -- 172.21.15.116:0/1725237237 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ae0036070 con 0x55753d25eb50 2024-03-19T01:12:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7acb7fe640 1 -- 172.21.15.116:0/1725237237 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f7ae0010b70 con 0x55753d25eb50 2024-03-19T01:12:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7af5620740 10 client.15255 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7af5620740 10 client.15255 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-19T01:12:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7af5620740 20 client.15255 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:27.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7af5620740 1 --2- 172.21.15.116:0/1725237237 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x55753d2da2b0 0x55753d2d8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7af5620740 4 client.15255 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.083+0000 7f7af5620740 1 -- 172.21.15.116:0/1725237237 --> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] -- command(tid 0: {"prefix": "status"}) -- 0x55753d1c5670 con 0x55753d2da2b0 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7aebfff640 1 --2- 172.21.15.116:0/1725237237 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x55753d2da2b0 0x55753d2d8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7acb7fe640 10 client.15255 ms_handle_connect on v2:172.21.15.162:6836/585664575 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7aebfff640 1 --2- 172.21.15.116:0/1725237237 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x55753d2da2b0 0x55753d2d8fc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7407877d6484220a server_cookie=7041ea87ae12dd0f in_seq=0 out_seq=0 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7acb7fe640 1 -- 172.21.15.116:0/1725237237 <== mds.4 v2:172.21.15.162:6836/585664575 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55753d1c5670 con 0x55753d2da2b0 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7acb7fe640 10 client.15255 handle_command_reply: tid=0 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7af5620740 1 client.15255 shutdown 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7af5620740 1 -- 172.21.15.116:0/1725237237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d25eb50 msgr2=0x55753d269350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7af5620740 1 --2- 172.21.15.116:0/1725237237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d25eb50 0x55753d269350 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7ae00058c0 tx=0x7f7ae0005f90 comp rx=0 tx=0).stop 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7af5620740 1 -- 172.21.15.116:0/1725237237 shutdown_connections 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7af5620740 1 --2- 172.21.15.116:0/1725237237 >> [v2:172.21.15.162:6836/585664575,v1:172.21.15.162:6837/585664575] conn(0x55753d2da2b0 0x55753d2d8fc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7af5620740 1 --2- 172.21.15.116:0/1725237237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55753d25eb50 0x55753d269350 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7af5620740 1 --2- 172.21.15.116:0/1725237237 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55753d2599e0 0x55753d266330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7acb7fe640 0 client.15255 ms_handle_reset on v2:172.21.15.162:6836/585664575 2024-03-19T01:12:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7af5620740 1 --2- 172.21.15.116:0/1725237237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55753d258ff0 0x55753d265df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7af5620740 1 -- 172.21.15.116:0/1725237237 >> 172.21.15.116:0/1725237237 conn(0x55753d254560 msgr2=0x55753d263030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7af5620740 1 -- 172.21.15.116:0/1725237237 shutdown_connections 2024-03-19T01:12:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.084+0000 7f7af5620740 1 -- 172.21.15.116:0/1725237237 wait complete. 2024-03-19T01:12:27.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.085+0000 7f7af5620740 20 client.15255 trim_cache size 0 max 16384 2024-03-19T01:12:27.097 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:12:27.097 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "35912c14-e58c-11ee-95c9-87774f69a715", 2024-03-19T01:12:27.098 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 4, 2024-03-19T01:12:27.098 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24445, 2024-03-19T01:12:27.098 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T01:12:27.098 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T01:12:27.098 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T01:12:27.098 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 109.105975117, 2024-03-19T01:12:27.098 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 24, 2024-03-19T01:12:27.098 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 87, 2024-03-19T01:12:27.098 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 85, 2024-03-19T01:12:27.098 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 134.12591926600001 2024-03-19T01:12:27.098 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:12:27.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.085+0000 7f7af2a3f640 1 -- 172.21.15.116:0/1559969926 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ad40a2800 msgr2=0x7f7ad40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.085+0000 7f7af2a3f640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ad40a2800 0x7f7ad40a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4097f20 tx=0x7f7ae4097640 comp rx=0 tx=0).stop 2024-03-19T01:12:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.085+0000 7f7af2a3f640 1 -- 172.21.15.116:0/1559969926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec1033e0 msgr2=0x7f7aec1a1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.085+0000 7f7af2a3f640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec1033e0 0x7f7aec1a1370 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400eb10 tx=0x7f7ae400c2d0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.085+0000 7f7af2a3f640 1 -- 172.21.15.116:0/1559969926 shutdown_connections 2024-03-19T01:12:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.085+0000 7f7af2a3f640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7aec1033e0 0x7f7aec1a1370 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.085+0000 7f7af2a3f640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7aec102a20 0x7f7aec19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.085+0000 7f7af2a3f640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ad40a2800 0x7f7ad40a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.085+0000 7f7af2a3f640 1 --2- 172.21.15.116:0/1559969926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7aec101830 0x7f7aec199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.085+0000 7f7af2a3f640 1 -- 172.21.15.116:0/1559969926 >> 172.21.15.116:0/1559969926 conn(0x7f7aec0fcdc0 msgr2=0x7f7aec0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.085+0000 7f7af2a3f640 1 -- 172.21.15.116:0/1559969926 shutdown_connections 2024-03-19T01:12:27.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.085+0000 7f7af2a3f640 1 -- 172.21.15.116:0/1559969926 wait complete. 2024-03-19T01:12:27.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-19T01:12:27.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.207+0000 7fdfa353b640 1 -- 172.21.15.116:0/654113859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9c100720 msgr2=0x7fdf9c100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.207+0000 7fdfa353b640 1 --2- 172.21.15.116:0/654113859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9c100720 0x7fdf9c100b40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fdf98009900 tx=0x7fdf9802bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.207+0000 7fdfa353b640 1 -- 172.21.15.116:0/654113859 shutdown_connections 2024-03-19T01:12:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.207+0000 7fdfa353b640 1 --2- 172.21.15.116:0/654113859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf9c1022d0 0x7fdf9c108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.207+0000 7fdfa353b640 1 --2- 172.21.15.116:0/654113859 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf9c101910 0x7fdf9c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.207+0000 7fdfa353b640 1 --2- 172.21.15.116:0/654113859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9c100720 0x7fdf9c100b40 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.207+0000 7fdfa353b640 1 -- 172.21.15.116:0/654113859 >> 172.21.15.116:0/654113859 conn(0x7fdf9c0fbcb0 msgr2=0x7fdf9c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa353b640 1 -- 172.21.15.116:0/654113859 shutdown_connections 2024-03-19T01:12:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa353b640 1 -- 172.21.15.116:0/654113859 wait complete. 2024-03-19T01:12:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa353b640 1 Processor -- start 2024-03-19T01:12:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa353b640 1 -- start start 2024-03-19T01:12:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa353b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9c100720 0x7fdf9c197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa353b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf9c101910 0x7fdf9c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa353b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf9c1022d0 0x7fdf9c19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa353b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c10d4c0 con 0x7fdf9c100720 2024-03-19T01:12:27.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa353b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c10d340 con 0x7fdf9c1022d0 2024-03-19T01:12:27.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa353b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9c10d640 con 0x7fdf9c101910 2024-03-19T01:12:27.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa12d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9c100720 0x7fdf9c197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa12d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9c100720 0x7fdf9c197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52680/0 (socket says 172.21.15.116:52680) 2024-03-19T01:12:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa12d7640 1 -- 172.21.15.116:0/3539696451 learned_addr learned my addr 172.21.15.116:0/3539696451 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa1ad8640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf9c1022d0 0x7fdf9c19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.208+0000 7fdfa0ad6640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf9c101910 0x7fdf9c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.209+0000 7fdfa12d7640 1 -- 172.21.15.116:0/3539696451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf9c101910 msgr2=0x7fdf9c1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.209+0000 7fdfa12d7640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf9c101910 0x7fdf9c1980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.209+0000 7fdfa12d7640 1 -- 172.21.15.116:0/3539696451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf9c1022d0 msgr2=0x7fdf9c19f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.209+0000 7fdfa12d7640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf9c1022d0 0x7fdf9c19f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.209+0000 7fdfa12d7640 1 -- 172.21.15.116:0/3539696451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9c19f7a0 con 0x7fdf9c100720 2024-03-19T01:12:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.209+0000 7fdfa12d7640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9c100720 0x7fdf9c197b80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdf98000c00 tx=0x7fdf9802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.209+0000 7fdf927fc640 1 -- 172.21.15.116:0/3539696451 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98041070 con 0x7fdf9c100720 2024-03-19T01:12:27.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.209+0000 7fdf927fc640 1 -- 172.21.15.116:0/3539696451 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf98004450 con 0x7fdf9c100720 2024-03-19T01:12:27.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.209+0000 7fdfa353b640 1 -- 172.21.15.116:0/3539696451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9c19fa30 con 0x7fdf9c100720 2024-03-19T01:12:27.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.209+0000 7fdf927fc640 1 -- 172.21.15.116:0/3539696451 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf98034440 con 0x7fdf9c100720 2024-03-19T01:12:27.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.209+0000 7fdfa353b640 1 -- 172.21.15.116:0/3539696451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf9c19ff20 con 0x7fdf9c100720 2024-03-19T01:12:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.211+0000 7fdf927fc640 1 -- 172.21.15.116:0/3539696451 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fdf98044070 con 0x7fdf9c100720 2024-03-19T01:12:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.211+0000 7fdfa620a740 1 Processor -- start 2024-03-19T01:12:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa620a740 1 -- start start 2024-03-19T01:12:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa620a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac5ff0 0x55a399ac6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa620a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399ac69e0 0x55a399acb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa620a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399acbb50 0x55a399acdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa620a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a399a2ade0 con 0x55a399ac69e0 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa620a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a3999d3c80 con 0x55a399acbb50 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa620a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a399a26e80 con 0x55a399ac5ff0 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa12d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399ac69e0 0x55a399acb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa0ad6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399acbb50 0x55a399acdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa1ad8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac5ff0 0x55a399ac6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa1ad8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac5ff0 0x55a399ac6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54480/0 (socket says 172.21.15.116:54480) 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa0ad6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399acbb50 0x55a399acdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:36706/0 (socket says 172.21.15.116:36706) 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa12d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399ac69e0 0x55a399acb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52688/0 (socket says 172.21.15.116:52688) 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa1ad8640 1 -- 172.21.15.116:0/3572779316 learned_addr learned my addr 172.21.15.116:0/3572779316 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdf927fc640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdf880a2800 0x7fdf880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa1ad8640 1 -- 172.21.15.116:0/3572779316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399acbb50 msgr2=0x55a399acdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa1ad8640 1 --2- 172.21.15.116:0/3572779316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399acbb50 0x55a399acdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa1ad8640 1 -- 172.21.15.116:0/3572779316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399ac69e0 msgr2=0x55a399acb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa1ad8640 1 --2- 172.21.15.116:0/3572779316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399ac69e0 0x55a399acb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa1ad8640 1 -- 172.21.15.116:0/3572779316 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3999e5fb0 con 0x55a399ac5ff0 2024-03-19T01:12:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa0ad6640 1 --2- 172.21.15.116:0/3572779316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399acbb50 0x55a399acdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdf927fc640 1 -- 172.21.15.116:0/3539696451 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fdf980f05a0 con 0x7fdf9c100720 2024-03-19T01:12:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa1ad8640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdf880a2800 0x7fdf880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.212+0000 7fdfa1ad8640 1 --2- 172.21.15.116:0/3572779316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac5ff0 0x55a399ac6410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009fd0 tx=0x7fdf940118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7296bbc5c1aa5705 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.213+0000 7fdf877fe640 1 -- 172.21.15.116:0/3572779316 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94004050 con 0x55a399ac5ff0 2024-03-19T01:12:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.213+0000 7fdf877fe640 1 -- 172.21.15.116:0/3572779316 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf94004750 con 0x55a399ac5ff0 2024-03-19T01:12:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.213+0000 7fdf877fe640 1 -- 172.21.15.116:0/3572779316 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94019400 con 0x55a399ac5ff0 2024-03-19T01:12:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.213+0000 7fdfa620a740 1 -- 172.21.15.116:0/3572779316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac5ff0 msgr2=0x55a399ac6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.213+0000 7fdfa1ad8640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdf880a2800 0x7fdf880a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdf9401d000 tx=0x7fdf94019710 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.213+0000 7fdfa620a740 1 --2- 172.21.15.116:0/3572779316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac5ff0 0x55a399ac6410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009fd0 tx=0x7fdf940118d0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.213+0000 7fdfa620a740 1 -- 172.21.15.116:0/3572779316 shutdown_connections 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.213+0000 7fdfa620a740 1 --2- 172.21.15.116:0/3572779316 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399acbb50 0x55a399acdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.213+0000 7fdfa620a740 1 --2- 172.21.15.116:0/3572779316 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399ac69e0 0x55a399acb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.213+0000 7fdfa620a740 1 --2- 172.21.15.116:0/3572779316 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac5ff0 0x55a399ac6410 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.213+0000 7fdfa620a740 1 -- 172.21.15.116:0/3572779316 >> 172.21.15.116:0/3572779316 conn(0x55a399ac1560 msgr2=0x55a399ac1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.213+0000 7fdfa620a740 1 -- 172.21.15.116:0/3572779316 shutdown_connections 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.213+0000 7fdfa620a740 1 -- 172.21.15.116:0/3572779316 wait complete. 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa620a740 1 -- start start 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa620a740 1 Processor -- start 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa620a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac5ff0 0x55a399ad04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa620a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399ac69e0 0x55a399ad09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa620a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399acbb50 0x55a399ad7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa620a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a399aceb00 con 0x55a399ac69e0 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa620a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a3999e6fc0 con 0x55a399acbb50 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa620a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a399acec80 con 0x55a399ac5ff0 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa620a740 10 client.0 _ll_register_callbacks cb 0x55a399a22a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa0ad6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399ac69e0 0x55a399ad09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa0ad6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399ac69e0 0x55a399ad09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52698/0 (socket says 172.21.15.116:52698) 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa0ad6640 1 -- 172.21.15.116:0/232454963 learned_addr learned my addr 172.21.15.116:0/232454963 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa12d7640 1 --2- 172.21.15.116:0/232454963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac5ff0 0x55a399ad04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa1ad8640 1 --2- 172.21.15.116:0/232454963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399acbb50 0x55a399ad7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa0ad6640 1 -- 172.21.15.116:0/232454963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac5ff0 msgr2=0x55a399ad04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa0ad6640 1 --2- 172.21.15.116:0/232454963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac5ff0 0x55a399ad04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa0ad6640 1 -- 172.21.15.116:0/232454963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399acbb50 msgr2=0x55a399ad7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa0ad6640 1 --2- 172.21.15.116:0/232454963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399acbb50 0x55a399ad7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.214+0000 7fdfa0ad6640 1 -- 172.21.15.116:0/232454963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a39973f2d0 con 0x55a399ac69e0 2024-03-19T01:12:27.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdfa0ad6640 1 --2- 172.21.15.116:0/232454963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399ac69e0 0x55a399ad09e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c012720 tx=0x7fdf8c012b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c366546fdc41c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdf84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdf84ff9640 1 -- 172.21.15.116:0/232454963 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c004050 con 0x55a399ac69e0 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdf84ff9640 1 -- 172.21.15.116:0/232454963 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fdf8c0041f0 con 0x55a399ac69e0 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdfa620a740 1 -- 172.21.15.116:0/232454963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf94004750 con 0x55a399ac69e0 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdf84ff9640 1 -- 172.21.15.116:0/232454963 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdf8c0043f0 con 0x55a399ac69e0 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdfa620a740 10 client.15267 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdfa620a740 1 -- 172.21.15.116:0/232454963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a3998096b0 con 0x55a399ac69e0 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdf84ff9640 1 -- 172.21.15.116:0/232454963 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf94004750 con 0x55a399ac69e0 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdf84ff9640 1 -- 172.21.15.116:0/232454963 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fdf8c003c70 con 0x55a399ac69e0 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdfa620a740 10 client.15267 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdfa620a740 10 client.15267 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdfa620a740 20 client.15267 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdfa620a740 1 --2- 172.21.15.116:0/232454963 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x55a399add360 0x55a399ade7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdfa620a740 4 client.15267 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:12:27.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.215+0000 7fdfa620a740 1 -- 172.21.15.116:0/232454963 --> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a399a38320 con 0x55a399add360 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.216+0000 7fdfa12d7640 1 --2- 172.21.15.116:0/232454963 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x55a399add360 0x55a399ade7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.216+0000 7fdfa12d7640 1 --2- 172.21.15.116:0/232454963 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x55a399add360 0x55a399ade7d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=143a81944938c8d4 server_cookie=9055a7278d4de783 in_seq=0 out_seq=0 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.216+0000 7fdf84ff9640 10 client.15267 ms_handle_connect on v2:172.21.15.162:6838/441486392 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdf84ff9640 1 -- 172.21.15.116:0/232454963 <== mds.3 v2:172.21.15.162:6838/441486392 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a399a38320 con 0x55a399add360 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdf84ff9640 10 client.15267 handle_command_reply: tid=0 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdfa620a740 1 client.15267 shutdown 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdfa620a740 1 -- 172.21.15.116:0/232454963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399ac69e0 msgr2=0x55a399ad09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdfa620a740 1 --2- 172.21.15.116:0/232454963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399ac69e0 0x55a399ad09e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c012720 tx=0x7fdf8c012b20 comp rx=0 tx=0).stop 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdfa620a740 1 -- 172.21.15.116:0/232454963 shutdown_connections 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdfa620a740 1 --2- 172.21.15.116:0/232454963 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x55a399add360 0x55a399ade7d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdfa620a740 1 --2- 172.21.15.116:0/232454963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399acbb50 0x55a399ad7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdf84ff9640 0 client.15267 ms_handle_reset on v2:172.21.15.162:6838/441486392 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdfa620a740 1 --2- 172.21.15.116:0/232454963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399ac69e0 0x55a399ad09e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdfa620a740 1 --2- 172.21.15.116:0/232454963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac5ff0 0x55a399ad04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdfa620a740 1 -- 172.21.15.116:0/232454963 >> 172.21.15.116:0/232454963 conn(0x55a399ac1560 msgr2=0x55a399ac07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdfa620a740 1 -- 172.21.15.116:0/232454963 shutdown_connections 2024-03-19T01:12:27.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdfa620a740 1 -- 172.21.15.116:0/232454963 wait complete. 2024-03-19T01:12:27.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.217+0000 7fdfa620a740 20 client.15267 trim_cache size 0 max 16384 2024-03-19T01:12:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.235+0000 7fdfa620a740 1 Processor -- start 2024-03-19T01:12:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.235+0000 7fdfa620a740 1 -- start start 2024-03-19T01:12:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.235+0000 7fdfa620a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399ac5ff0 0x55a399ac7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.235+0000 7fdfa620a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac69e0 0x55a399acdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.235+0000 7fdfa620a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 0x55a399ac7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.235+0000 7fdfa620a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a399adcb10 con 0x55a399acbb50 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.235+0000 7fdfa620a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a399add1e0 con 0x55a399ac5ff0 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.235+0000 7fdfa620a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a399add8b0 con 0x55a399ac69e0 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.235+0000 7fdfa1ad8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 0x55a399ac7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.235+0000 7fdfa1ad8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 0x55a399ac7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52708/0 (socket says 172.21.15.116:52708) 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.235+0000 7fdfa1ad8640 1 -- 172.21.15.116:0/983212958 learned_addr learned my addr 172.21.15.116:0/983212958 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.235+0000 7fdfa12d7640 1 --2- 172.21.15.116:0/983212958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399ac5ff0 0x55a399ac7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.235+0000 7fdfa0ad6640 1 --2- 172.21.15.116:0/983212958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac69e0 0x55a399acdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa1ad8640 1 -- 172.21.15.116:0/983212958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac69e0 msgr2=0x55a399acdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa1ad8640 1 --2- 172.21.15.116:0/983212958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac69e0 0x55a399acdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa1ad8640 1 -- 172.21.15.116:0/983212958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399ac5ff0 msgr2=0x55a399ac7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa1ad8640 1 --2- 172.21.15.116:0/983212958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399ac5ff0 0x55a399ac7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa1ad8640 1 -- 172.21.15.116:0/983212958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a399adeb00 con 0x55a399acbb50 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa12d7640 1 --2- 172.21.15.116:0/983212958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399ac5ff0 0x55a399ac7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa1ad8640 1 --2- 172.21.15.116:0/983212958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 0x55a399ac7e00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009fd0 tx=0x7fdf94028000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92156f98324f6b53 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdf86ffd640 1 -- 172.21.15.116:0/983212958 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94034070 con 0x55a399acbb50 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdf86ffd640 1 -- 172.21.15.116:0/983212958 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf94019400 con 0x55a399acbb50 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdf86ffd640 1 -- 172.21.15.116:0/983212958 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9400f740 con 0x55a399acbb50 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa620a740 1 -- 172.21.15.116:0/983212958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 msgr2=0x55a399ac7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa620a740 1 --2- 172.21.15.116:0/983212958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 0x55a399ac7e00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fdf94009fd0 tx=0x7fdf94028000 comp rx=0 tx=0).stop 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa620a740 1 -- 172.21.15.116:0/983212958 shutdown_connections 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa620a740 1 --2- 172.21.15.116:0/983212958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 0x55a399ac7e00 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa620a740 1 --2- 172.21.15.116:0/983212958 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac69e0 0x55a399acdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa620a740 1 --2- 172.21.15.116:0/983212958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399ac5ff0 0x55a399ac7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.236+0000 7fdfa620a740 1 -- 172.21.15.116:0/983212958 >> 172.21.15.116:0/983212958 conn(0x55a399ac1560 msgr2=0x55a399ac1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa620a740 1 -- 172.21.15.116:0/983212958 shutdown_connections 2024-03-19T01:12:27.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa620a740 1 -- 172.21.15.116:0/983212958 wait complete. 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa620a740 1 -- start start 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa620a740 1 Processor -- start 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa620a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399ac5ff0 0x55a399b3f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa620a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac69e0 0x55a399b3f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa620a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 0x55a399b464a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa620a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a399ade650 con 0x55a399acbb50 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa620a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a399addf80 con 0x55a399ac5ff0 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa620a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a399aded20 con 0x55a399ac69e0 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa620a740 10 client.0 _ll_register_callbacks cb 0x7fdf8c009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa1ad8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 0x55a399b464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa1ad8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 0x55a399b464a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52712/0 (socket says 172.21.15.116:52712) 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa12d7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399ac5ff0 0x55a399b3f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa0ad6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac69e0 0x55a399b3f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.237+0000 7fdfa1ad8640 1 -- 172.21.15.116:0/4215007560 learned_addr learned my addr 172.21.15.116:0/4215007560 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa1ad8640 1 -- 172.21.15.116:0/4215007560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac69e0 msgr2=0x55a399b3f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa1ad8640 1 --2- 172.21.15.116:0/4215007560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac69e0 0x55a399b3f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa1ad8640 1 -- 172.21.15.116:0/4215007560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399ac5ff0 msgr2=0x55a399b3f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa1ad8640 1 --2- 172.21.15.116:0/4215007560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399ac5ff0 0x55a399b3f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa1ad8640 1 -- 172.21.15.116:0/4215007560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a399b46ab0 con 0x55a399acbb50 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa0ad6640 1 --2- 172.21.15.116:0/4215007560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac69e0 0x55a399b3f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa1ad8640 1 --2- 172.21.15.116:0/4215007560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 0x55a399b464a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdf94011910 tx=0x7fdf94004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=397957cbb5018339 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdf84ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdf84ff9640 1 -- 172.21.15.116:0/4215007560 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf94034040 con 0x55a399acbb50 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdf84ff9640 1 -- 172.21.15.116:0/4215007560 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fdf9400f4e0 con 0x55a399acbb50 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa620a740 1 -- 172.21.15.116:0/4215007560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf94019400 con 0x55a399acbb50 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdf84ff9640 1 -- 172.21.15.116:0/4215007560 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdf94004950 con 0x55a399acbb50 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa620a740 10 client.15273 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa620a740 1 -- 172.21.15.116:0/4215007560 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a399b47db0 con 0x55a399acbb50 2024-03-19T01:12:27.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdf84ff9640 1 -- 172.21.15.116:0/4215007560 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdf94019400 con 0x55a399acbb50 2024-03-19T01:12:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdf84ff9640 1 -- 172.21.15.116:0/4215007560 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fdf940398a0 con 0x55a399acbb50 2024-03-19T01:12:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa620a740 10 client.15273 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa620a740 10 client.15273 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-19T01:12:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa620a740 20 client.15273 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa620a740 1 --2- 172.21.15.116:0/4215007560 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x55a399add360 0x55a399b4b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa620a740 4 client.15273 mds_command: new command op to 24460 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T01:12:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.238+0000 7fdfa620a740 1 -- 172.21.15.116:0/4215007560 --> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] -- command(tid 0: {"prefix": "status"}) -- 0x55a399a32670 con 0x55a399add360 2024-03-19T01:12:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa12d7640 1 --2- 172.21.15.116:0/4215007560 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x55a399add360 0x55a399b4b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdf84ff9640 10 client.15273 ms_handle_connect on v2:172.21.15.162:6838/441486392 2024-03-19T01:12:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa12d7640 1 --2- 172.21.15.116:0/4215007560 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x55a399add360 0x55a399b4b040 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e54702921cdb3185 server_cookie=31b7f22f801b0687 in_seq=0 out_seq=0 2024-03-19T01:12:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdf84ff9640 1 -- 172.21.15.116:0/4215007560 <== mds.3 v2:172.21.15.162:6838/441486392 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55a399a32670 con 0x55a399add360 2024-03-19T01:12:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdf84ff9640 10 client.15273 handle_command_reply: tid=0 2024-03-19T01:12:27.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa620a740 1 client.15273 shutdown 2024-03-19T01:12:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa620a740 1 -- 172.21.15.116:0/4215007560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 msgr2=0x55a399b464a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa620a740 1 --2- 172.21.15.116:0/4215007560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 0x55a399b464a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdf94011910 tx=0x7fdf94004700 comp rx=0 tx=0).stop 2024-03-19T01:12:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa620a740 1 -- 172.21.15.116:0/4215007560 shutdown_connections 2024-03-19T01:12:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa620a740 1 --2- 172.21.15.116:0/4215007560 >> [v2:172.21.15.162:6838/441486392,v1:172.21.15.162:6839/441486392] conn(0x55a399add360 0x55a399b4b040 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa620a740 1 --2- 172.21.15.116:0/4215007560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a399acbb50 0x55a399b464a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa620a740 1 --2- 172.21.15.116:0/4215007560 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a399ac69e0 0x55a399b3f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdf84ff9640 0 client.15273 ms_handle_reset on v2:172.21.15.162:6838/441486392 2024-03-19T01:12:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa620a740 1 --2- 172.21.15.116:0/4215007560 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a399ac5ff0 0x55a399b3f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa620a740 1 -- 172.21.15.116:0/4215007560 >> 172.21.15.116:0/4215007560 conn(0x55a399ac1560 msgr2=0x55a399abc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa620a740 1 -- 172.21.15.116:0/4215007560 shutdown_connections 2024-03-19T01:12:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa620a740 1 -- 172.21.15.116:0/4215007560 wait complete. 2024-03-19T01:12:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.239+0000 7fdfa620a740 20 client.15273 trim_cache size 0 max 16384 2024-03-19T01:12:27.248 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:12:27.248 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "35912c14-e58c-11ee-95c9-87774f69a715", 2024-03-19T01:12:27.248 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 3, 2024-03-19T01:12:27.248 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24460, 2024-03-19T01:12:27.248 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T01:12:27.248 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T01:12:27.248 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T01:12:27.248 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 110.268954718, 2024-03-19T01:12:27.248 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 24, 2024-03-19T01:12:27.248 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 87, 2024-03-19T01:12:27.248 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 85, 2024-03-19T01:12:27.248 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 128.53409129400001 2024-03-19T01:12:27.248 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:12:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.240+0000 7fdfa353b640 1 -- 172.21.15.116:0/3539696451 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdf880a2800 msgr2=0x7fdf880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.240+0000 7fdfa353b640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdf880a2800 0x7fdf880a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdf9401d000 tx=0x7fdf94019710 comp rx=0 tx=0).stop 2024-03-19T01:12:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.240+0000 7fdfa353b640 1 -- 172.21.15.116:0/3539696451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9c100720 msgr2=0x7fdf9c197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.240+0000 7fdfa353b640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9c100720 0x7fdf9c197b80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdf98000c00 tx=0x7fdf9802c640 comp rx=0 tx=0).stop 2024-03-19T01:12:27.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.240+0000 7fdfa353b640 1 -- 172.21.15.116:0/3539696451 shutdown_connections 2024-03-19T01:12:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.240+0000 7fdfa353b640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fdf880a2800 0x7fdf880a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.240+0000 7fdfa353b640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf9c1022d0 0x7fdf9c19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.240+0000 7fdfa353b640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fdf9c101910 0x7fdf9c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.240+0000 7fdfa353b640 1 --2- 172.21.15.116:0/3539696451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf9c100720 0x7fdf9c197b80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.240+0000 7fdfa353b640 1 -- 172.21.15.116:0/3539696451 >> 172.21.15.116:0/3539696451 conn(0x7fdf9c0fbcb0 msgr2=0x7fdf9c104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.240+0000 7fdfa353b640 1 -- 172.21.15.116:0/3539696451 shutdown_connections 2024-03-19T01:12:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.240+0000 7fdfa353b640 1 -- 172.21.15.116:0/3539696451 wait complete. 2024-03-19T01:12:27.250 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:27 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/842327880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:12:27.250 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:27 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1175247167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:12:27.250 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:27 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1665202576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:12:27.251 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-19T01:12:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.353+0000 7f2dea8c0640 1 -- 172.21.15.116:0/1012262672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de41022d0 msgr2=0x7f2de4108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.353+0000 7f2dea8c0640 1 --2- 172.21.15.116:0/1012262672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de41022d0 0x7f2de4108b60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc009ca0 tx=0x7f2ddc02c0f0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.353+0000 7f2dea8c0640 1 -- 172.21.15.116:0/1012262672 shutdown_connections 2024-03-19T01:12:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.353+0000 7f2dea8c0640 1 --2- 172.21.15.116:0/1012262672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de41022d0 0x7f2de4108b60 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.353+0000 7f2dea8c0640 1 --2- 172.21.15.116:0/1012262672 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2de4101910 0x7f2de4101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.353+0000 7f2dea8c0640 1 --2- 172.21.15.116:0/1012262672 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de4100720 0x7f2de4100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.354+0000 7f2dea8c0640 1 -- 172.21.15.116:0/1012262672 >> 172.21.15.116:0/1012262672 conn(0x7f2de40fbcb0 msgr2=0x7f2de40fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.354+0000 7f2dea8c0640 1 -- 172.21.15.116:0/1012262672 shutdown_connections 2024-03-19T01:12:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.354+0000 7f2dea8c0640 1 -- 172.21.15.116:0/1012262672 wait complete. 2024-03-19T01:12:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.354+0000 7f2dea8c0640 1 Processor -- start 2024-03-19T01:12:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.354+0000 7f2dea8c0640 1 -- start start 2024-03-19T01:12:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2dea8c0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de4100720 0x7f2de4196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2dea8c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de4101910 0x7f2de4196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2dea8c0640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2de41022d0 0x7f2de419dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2dea8c0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2de410c3a0 con 0x7f2de4101910 2024-03-19T01:12:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2dea8c0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2de410c220 con 0x7f2de4100720 2024-03-19T01:12:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2dea8c0640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2de410c520 con 0x7f2de41022d0 2024-03-19T01:12:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de4101910 0x7f2de4196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de4101910 0x7f2de4196f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52718/0 (socket says 172.21.15.116:52718) 2024-03-19T01:12:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de37fe640 1 -- 172.21.15.116:0/1287837472 learned_addr learned my addr 172.21.15.116:0/1287837472 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de3fff640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de4100720 0x7f2de4196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de8e5d640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2de41022d0 0x7f2de419dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de37fe640 1 -- 172.21.15.116:0/1287837472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2de41022d0 msgr2=0x7f2de419dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de37fe640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2de41022d0 0x7f2de419dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de37fe640 1 -- 172.21.15.116:0/1287837472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de4100720 msgr2=0x7f2de4196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de37fe640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de4100720 0x7f2de4196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de37fe640 1 -- 172.21.15.116:0/1287837472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de419e630 con 0x7f2de4101910 2024-03-19T01:12:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de3fff640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de4100720 0x7f2de4196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de37fe640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de4101910 0x7f2de4196f50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00e9f0 tx=0x7f2dcc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de17fa640 1 -- 172.21.15.116:0/1287837472 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc00cd80 con 0x7f2de4101910 2024-03-19T01:12:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de17fa640 1 -- 172.21.15.116:0/1287837472 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dcc005500 con 0x7f2de4101910 2024-03-19T01:12:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2dea8c0640 1 -- 172.21.15.116:0/1287837472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de419e920 con 0x7f2de4101910 2024-03-19T01:12:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2de17fa640 1 -- 172.21.15.116:0/1287837472 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc010690 con 0x7f2de4101910 2024-03-19T01:12:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.355+0000 7f2dea8c0640 1 -- 172.21.15.116:0/1287837472 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de419ee20 con 0x7f2de4101910 2024-03-19T01:12:27.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.358+0000 7f2de17fa640 1 -- 172.21.15.116:0/1287837472 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f2dcc0046c0 con 0x7f2de4101910 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.358+0000 7f2de17fa640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2dd00a2800 0x7f2dd00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.358+0000 7f2de3fff640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2dd00a2800 0x7f2dd00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.358+0000 7f2de17fa640 1 -- 172.21.15.116:0/1287837472 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2dcc0cee70 con 0x7f2de4101910 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.358+0000 7f2ded587740 1 Processor -- start 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.358+0000 7f2ded587740 1 -- start start 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2de3fff640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2dd00a2800 0x7f2dd00a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2de4101770 tx=0x7f2dd4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2ded587740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82dff0 0x56177e82e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2ded587740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82e9e0 0x56177e8334e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2ded587740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e833b50 0x56177e835fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2ded587740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56177e792de0 con 0x56177e82dff0 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2ded587740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56177e73bc80 con 0x56177e82e9e0 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2ded587740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56177e78ee80 con 0x56177e833b50 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2de3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82dff0 0x56177e82e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2de3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82dff0 0x56177e82e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52732/0 (socket says 172.21.15.116:52732) 2024-03-19T01:12:27.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2de3fff640 1 -- 172.21.15.116:0/1476856685 learned_addr learned my addr 172.21.15.116:0/1476856685 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2de37fe640 1 --2- 172.21.15.116:0/1476856685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82e9e0 0x56177e8334e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2de8e5d640 1 --2- 172.21.15.116:0/1476856685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e833b50 0x56177e835fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2de3fff640 1 -- 172.21.15.116:0/1476856685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e833b50 msgr2=0x56177e835fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2de3fff640 1 --2- 172.21.15.116:0/1476856685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e833b50 0x56177e835fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2de3fff640 1 -- 172.21.15.116:0/1476856685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82e9e0 msgr2=0x56177e8334e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2de3fff640 1 --2- 172.21.15.116:0/1476856685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82e9e0 0x56177e8334e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2de3fff640 1 -- 172.21.15.116:0/1476856685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56177e74dfb0 con 0x56177e82dff0 2024-03-19T01:12:27.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2de3fff640 1 --2- 172.21.15.116:0/1476856685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82dff0 0x56177e82e410 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4004340 tx=0x7f2dd4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2c5cd6c45744d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2dda7fc640 1 -- 172.21.15.116:0/1476856685 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4026430 con 0x56177e82dff0 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2dda7fc640 1 -- 172.21.15.116:0/1476856685 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dd401c8d0 con 0x56177e82dff0 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.359+0000 7f2dda7fc640 1 -- 172.21.15.116:0/1476856685 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd401da30 con 0x56177e82dff0 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.360+0000 7f2ded587740 1 -- 172.21.15.116:0/1476856685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82dff0 msgr2=0x56177e82e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.360+0000 7f2ded587740 1 --2- 172.21.15.116:0/1476856685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82dff0 0x56177e82e410 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4004340 tx=0x7f2dd4014000 comp rx=0 tx=0).stop 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.360+0000 7f2ded587740 1 -- 172.21.15.116:0/1476856685 shutdown_connections 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.360+0000 7f2ded587740 1 --2- 172.21.15.116:0/1476856685 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e833b50 0x56177e835fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.360+0000 7f2ded587740 1 --2- 172.21.15.116:0/1476856685 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82e9e0 0x56177e8334e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.360+0000 7f2ded587740 1 --2- 172.21.15.116:0/1476856685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82dff0 0x56177e82e410 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.360+0000 7f2ded587740 1 -- 172.21.15.116:0/1476856685 >> 172.21.15.116:0/1476856685 conn(0x56177e829560 msgr2=0x56177e829960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.360+0000 7f2ded587740 1 -- 172.21.15.116:0/1476856685 shutdown_connections 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.360+0000 7f2ded587740 1 -- 172.21.15.116:0/1476856685 wait complete. 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.360+0000 7f2ded587740 1 -- start start 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2ded587740 1 Processor -- start 2024-03-19T01:12:27.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2ded587740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e82dff0 0x56177e83a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2ded587740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 0x56177e83abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2ded587740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e833b50 0x56177e841c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2ded587740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56177e836b00 con 0x56177e82e9e0 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2ded587740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56177e74efc0 con 0x56177e833b50 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2ded587740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56177e836c80 con 0x56177e82dff0 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2ded587740 10 client.0 _ll_register_callbacks cb 0x56177e78aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 0x56177e83abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2de3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e82dff0 0x56177e83a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 0x56177e83abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52736/0 (socket says 172.21.15.116:52736) 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2de37fe640 1 -- 172.21.15.116:0/3852806967 learned_addr learned my addr 172.21.15.116:0/3852806967 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2de8e5d640 1 --2- 172.21.15.116:0/3852806967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e833b50 0x56177e841c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2de3fff640 1 -- 172.21.15.116:0/3852806967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e833b50 msgr2=0x56177e841c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2de3fff640 1 --2- 172.21.15.116:0/3852806967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e833b50 0x56177e841c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2de3fff640 1 -- 172.21.15.116:0/3852806967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 msgr2=0x56177e83abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2de3fff640 1 --2- 172.21.15.116:0/3852806967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 0x56177e83abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2de3fff640 1 -- 172.21.15.116:0/3852806967 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56177e4a72d0 con 0x56177e82dff0 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2de37fe640 1 --2- 172.21.15.116:0/3852806967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 0x56177e83abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2de3fff640 1 --2- 172.21.15.116:0/3852806967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e82dff0 0x56177e83a6b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40095f0 tx=0x7f2dd401d480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae9b9197947b5c96 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.361+0000 7f2dc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2dc37fe640 1 -- 172.21.15.116:0/3852806967 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4034060 con 0x56177e82dff0 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2ded587740 1 -- 172.21.15.116:0/3852806967 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dd401c8d0 con 0x56177e82dff0 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2dc37fe640 1 -- 172.21.15.116:0/3852806967 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2dd401c4b0 con 0x56177e82dff0 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2dc37fe640 1 -- 172.21.15.116:0/3852806967 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2dd401c6b0 con 0x56177e82dff0 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2ded587740 10 client.25105 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2ded587740 1 -- 172.21.15.116:0/3852806967 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56177e5716b0 con 0x56177e82dff0 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2dc37fe640 1 -- 172.21.15.116:0/3852806967 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dd401c8d0 con 0x56177e82dff0 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2dc37fe640 1 -- 172.21.15.116:0/3852806967 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f2dd401a070 con 0x56177e82dff0 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2ded587740 10 client.25105 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:27.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2ded587740 10 client.25105 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T01:12:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2ded587740 20 client.25105 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2ded587740 1 --2- 172.21.15.116:0/3852806967 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x56177e847020 0x56177e848490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2ded587740 4 client.25105 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:12:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2ded587740 1 -- 172.21.15.116:0/3852806967 --> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56177e7a0320 con 0x56177e847020 2024-03-19T01:12:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.362+0000 7f2de37fe640 1 --2- 172.21.15.116:0/3852806967 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x56177e847020 0x56177e848490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.363+0000 7f2de37fe640 1 --2- 172.21.15.116:0/3852806967 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x56177e847020 0x56177e848490 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=98cac1546b8c8832 server_cookie=21a463e7d8a4fd39 in_seq=0 out_seq=0 2024-03-19T01:12:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.363+0000 7f2dc37fe640 10 client.25105 ms_handle_connect on v2:172.21.15.156:6836/4147060396 2024-03-19T01:12:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.363+0000 7f2dc37fe640 1 -- 172.21.15.116:0/3852806967 <== mds.2 v2:172.21.15.156:6836/4147060396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56177e7a0320 con 0x56177e847020 2024-03-19T01:12:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.363+0000 7f2dc37fe640 10 client.25105 handle_command_reply: tid=0 2024-03-19T01:12:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.363+0000 7f2ded587740 1 client.25105 shutdown 2024-03-19T01:12:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.364+0000 7f2ded587740 1 -- 172.21.15.116:0/3852806967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e82dff0 msgr2=0x56177e83a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.364+0000 7f2ded587740 1 --2- 172.21.15.116:0/3852806967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e82dff0 0x56177e83a6b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2dd40095f0 tx=0x7f2dd401d480 comp rx=0 tx=0).stop 2024-03-19T01:12:27.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.364+0000 7f2ded587740 1 -- 172.21.15.116:0/3852806967 shutdown_connections 2024-03-19T01:12:27.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.364+0000 7f2ded587740 1 --2- 172.21.15.116:0/3852806967 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x56177e847020 0x56177e848490 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.364+0000 7f2ded587740 1 --2- 172.21.15.116:0/3852806967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e833b50 0x56177e841c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.364+0000 7f2ded587740 1 --2- 172.21.15.116:0/3852806967 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 0x56177e83abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.364+0000 7f2dc37fe640 0 client.25105 ms_handle_reset on v2:172.21.15.156:6836/4147060396 2024-03-19T01:12:27.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.364+0000 7f2ded587740 1 --2- 172.21.15.116:0/3852806967 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e82dff0 0x56177e83a6b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.364+0000 7f2ded587740 1 -- 172.21.15.116:0/3852806967 >> 172.21.15.116:0/3852806967 conn(0x56177e829560 msgr2=0x56177e8287e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.364+0000 7f2ded587740 1 -- 172.21.15.116:0/3852806967 shutdown_connections 2024-03-19T01:12:27.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.364+0000 7f2ded587740 1 -- 172.21.15.116:0/3852806967 wait complete. 2024-03-19T01:12:27.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.364+0000 7f2ded587740 20 client.25105 trim_cache size 0 max 16384 2024-03-19T01:12:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2ded587740 1 Processor -- start 2024-03-19T01:12:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2ded587740 1 -- start start 2024-03-19T01:12:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2ded587740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82dff0 0x56177e82fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2ded587740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 0x56177e8303e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2ded587740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e833b50 0x56177e841bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2ded587740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56177e8467d0 con 0x56177e82e9e0 2024-03-19T01:12:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2ded587740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56177e846ea0 con 0x56177e82dff0 2024-03-19T01:12:27.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2ded587740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56177e847570 con 0x56177e833b50 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 0x56177e8303e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2de37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 0x56177e8303e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52740/0 (socket says 172.21.15.116:52740) 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2de37fe640 1 -- 172.21.15.116:0/3506588394 learned_addr learned my addr 172.21.15.116:0/3506588394 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2de3fff640 1 --2- 172.21.15.116:0/3506588394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82dff0 0x56177e82fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2de8e5d640 1 --2- 172.21.15.116:0/3506588394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e833b50 0x56177e841bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2de37fe640 1 -- 172.21.15.116:0/3506588394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e833b50 msgr2=0x56177e841bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2de37fe640 1 --2- 172.21.15.116:0/3506588394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e833b50 0x56177e841bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2de37fe640 1 -- 172.21.15.116:0/3506588394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82dff0 msgr2=0x56177e82fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2de37fe640 1 --2- 172.21.15.116:0/3506588394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82dff0 0x56177e82fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2de37fe640 1 -- 172.21.15.116:0/3506588394 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56177e823bf0 con 0x56177e82e9e0 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2de3fff640 1 --2- 172.21.15.116:0/3506588394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82dff0 0x56177e82fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2de37fe640 1 --2- 172.21.15.116:0/3506588394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 0x56177e8303e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc09b780 tx=0x7f2dcc086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269faa58de5331ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2dd9ffb640 1 -- 172.21.15.116:0/3506588394 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc09b7f0 con 0x56177e82e9e0 2024-03-19T01:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2dd9ffb640 1 -- 172.21.15.116:0/3506588394 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dcc09f940 con 0x56177e82e9e0 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.382+0000 7f2dd9ffb640 1 -- 172.21.15.116:0/3506588394 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc0afe60 con 0x56177e82e9e0 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.383+0000 7f2ded587740 1 -- 172.21.15.116:0/3506588394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 msgr2=0x56177e8303e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.383+0000 7f2ded587740 1 --2- 172.21.15.116:0/3506588394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 0x56177e8303e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc09b780 tx=0x7f2dcc086040 comp rx=0 tx=0).stop 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.383+0000 7f2ded587740 1 -- 172.21.15.116:0/3506588394 shutdown_connections 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.383+0000 7f2ded587740 1 --2- 172.21.15.116:0/3506588394 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e833b50 0x56177e841bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.383+0000 7f2ded587740 1 --2- 172.21.15.116:0/3506588394 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e82e9e0 0x56177e8303e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.383+0000 7f2ded587740 1 --2- 172.21.15.116:0/3506588394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82dff0 0x56177e82fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.383+0000 7f2ded587740 1 -- 172.21.15.116:0/3506588394 >> 172.21.15.116:0/3506588394 conn(0x56177e829560 msgr2=0x56177e82f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.383+0000 7f2ded587740 1 -- 172.21.15.116:0/3506588394 shutdown_connections 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.383+0000 7f2ded587740 1 -- 172.21.15.116:0/3506588394 wait complete. 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.383+0000 7f2ded587740 1 -- start start 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2ded587740 1 Processor -- start 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2ded587740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82dff0 0x56177e83fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2ded587740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e82e9e0 0x56177e840370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2ded587740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e833b50 0x56177e8408b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2ded587740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56177e848310 con 0x56177e833b50 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2ded587740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56177e847c40 con 0x56177e82dff0 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2ded587740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56177e8489e0 con 0x56177e82e9e0 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2ded587740 10 client.0 _ll_register_callbacks cb 0x7f2dcc0056b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2de8e5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e833b50 0x56177e8408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2de3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82dff0 0x56177e83fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2de8e5d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e833b50 0x56177e8408b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52750/0 (socket says 172.21.15.116:52750) 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2de8e5d640 1 -- 172.21.15.116:0/3645741972 learned_addr learned my addr 172.21.15.116:0/3645741972 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2de37fe640 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e82e9e0 0x56177e840370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2de3fff640 1 -- 172.21.15.116:0/3645741972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e82e9e0 msgr2=0x56177e840370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2de3fff640 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e82e9e0 0x56177e840370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2de3fff640 1 -- 172.21.15.116:0/3645741972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e833b50 msgr2=0x56177e8408b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2de3fff640 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e833b50 0x56177e8408b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2de3fff640 1 -- 172.21.15.116:0/3645741972 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56177e841020 con 0x56177e82dff0 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2de8e5d640 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e833b50 0x56177e8408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.384+0000 7f2de37fe640 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e82e9e0 0x56177e840370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2de3fff640 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82dff0 0x56177e83fe30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2dd401d980 tx=0x7f2dd4005c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eca969cf9a21895f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2dc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2dc37fe640 1 -- 172.21.15.116:0/3645741972 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd4034040 con 0x56177e82dff0 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2ded587740 1 -- 172.21.15.116:0/3645741972 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2dcc09f940 con 0x56177e82dff0 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2dc37fe640 1 -- 172.21.15.116:0/3645741972 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2dd401d4f0 con 0x56177e82dff0 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2dc37fe640 1 -- 172.21.15.116:0/3645741972 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2dd4049050 con 0x56177e82dff0 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2ded587740 10 client.25133 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2ded587740 1 -- 172.21.15.116:0/3645741972 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56177e8acee0 con 0x56177e82dff0 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2dc37fe640 1 -- 172.21.15.116:0/3645741972 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2dcc09f940 con 0x56177e82dff0 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2dc37fe640 1 -- 172.21.15.116:0/3645741972 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f2dd4041020 con 0x56177e82dff0 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2ded587740 10 client.25133 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:27.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2ded587740 10 client.25133 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2ded587740 20 client.25133 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2ded587740 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x56177e8b1a20 0x56177e8b0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2ded587740 4 client.25133 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.385+0000 7f2ded587740 1 -- 172.21.15.116:0/3645741972 --> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] -- command(tid 0: {"prefix": "status"}) -- 0x56177e79a670 con 0x56177e8b1a20 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2de37fe640 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x56177e8b1a20 0x56177e8b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2dc37fe640 10 client.25133 ms_handle_connect on v2:172.21.15.156:6836/4147060396 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2de37fe640 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x56177e8b1a20 0x56177e8b0730 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7cb00d6821c221da server_cookie=4c52d5cb735a4436 in_seq=0 out_seq=0 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2dc37fe640 1 -- 172.21.15.116:0/3645741972 <== mds.2 v2:172.21.15.156:6836/4147060396 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56177e79a670 con 0x56177e8b1a20 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2dc37fe640 10 client.25133 handle_command_reply: tid=0 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2ded587740 1 client.25133 shutdown 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2ded587740 1 -- 172.21.15.116:0/3645741972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82dff0 msgr2=0x56177e83fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2ded587740 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82dff0 0x56177e83fe30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2dd401d980 tx=0x7f2dd4005c70 comp rx=0 tx=0).stop 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2ded587740 1 -- 172.21.15.116:0/3645741972 shutdown_connections 2024-03-19T01:12:27.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2ded587740 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.156:6836/4147060396,v1:172.21.15.156:6837/4147060396] conn(0x56177e8b1a20 0x56177e8b0730 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2ded587740 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56177e833b50 0x56177e8408b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2ded587740 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56177e82e9e0 0x56177e840370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2ded587740 1 --2- 172.21.15.116:0/3645741972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56177e82dff0 0x56177e83fe30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2dc37fe640 0 client.25133 ms_handle_reset on v2:172.21.15.156:6836/4147060396 2024-03-19T01:12:27.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2ded587740 1 -- 172.21.15.116:0/3645741972 >> 172.21.15.116:0/3645741972 conn(0x56177e829560 msgr2=0x56177e82fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2ded587740 1 -- 172.21.15.116:0/3645741972 shutdown_connections 2024-03-19T01:12:27.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2ded587740 1 -- 172.21.15.116:0/3645741972 wait complete. 2024-03-19T01:12:27.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.386+0000 7f2ded587740 20 client.25133 trim_cache size 0 max 16384 2024-03-19T01:12:27.394 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:12:27.394 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "35912c14-e58c-11ee-95c9-87774f69a715", 2024-03-19T01:12:27.395 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 2, 2024-03-19T01:12:27.395 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24473, 2024-03-19T01:12:27.395 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T01:12:27.395 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T01:12:27.395 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T01:12:27.395 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 111.423603783, 2024-03-19T01:12:27.395 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 24, 2024-03-19T01:12:27.395 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 87, 2024-03-19T01:12:27.395 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 85, 2024-03-19T01:12:27.395 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 131.52440169499999 2024-03-19T01:12:27.395 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:12:27.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.387+0000 7f2dea8c0640 1 -- 172.21.15.116:0/1287837472 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2dd00a2800 msgr2=0x7f2dd00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.387+0000 7f2dea8c0640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2dd00a2800 0x7f2dd00a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f2de4101770 tx=0x7f2dd4005f90 comp rx=0 tx=0).stop 2024-03-19T01:12:27.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.387+0000 7f2dea8c0640 1 -- 172.21.15.116:0/1287837472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de4101910 msgr2=0x7f2de4196f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.387+0000 7f2dea8c0640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de4101910 0x7f2de4196f50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00e9f0 tx=0x7f2dcc00edf0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.387+0000 7f2dea8c0640 1 -- 172.21.15.116:0/1287837472 shutdown_connections 2024-03-19T01:12:27.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.387+0000 7f2dea8c0640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2de41022d0 0x7f2de419dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.387+0000 7f2dea8c0640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2de4101910 0x7f2de4196f50 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.387+0000 7f2dea8c0640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2dd00a2800 0x7f2dd00a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.387+0000 7f2dea8c0640 1 --2- 172.21.15.116:0/1287837472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de4100720 0x7f2de4196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.387+0000 7f2dea8c0640 1 -- 172.21.15.116:0/1287837472 >> 172.21.15.116:0/1287837472 conn(0x7f2de40fbcb0 msgr2=0x7f2de4104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.387+0000 7f2dea8c0640 1 -- 172.21.15.116:0/1287837472 shutdown_connections 2024-03-19T01:12:27.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.387+0000 7f2dea8c0640 1 -- 172.21.15.116:0/1287837472 wait complete. 2024-03-19T01:12:27.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-19T01:12:27.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:27 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/842327880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:12:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:27 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1175247167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:12:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:27 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1665202576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:12:27.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.501+0000 7f0a2071b640 1 -- 172.21.15.116:0/2714697208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a18100720 msgr2=0x7f0a18100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.501+0000 7f0a2071b640 1 --2- 172.21.15.116:0/2714697208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a18100720 0x7f0a18100b40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009900 tx=0x7f0a1002bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:27.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.501+0000 7f0a2071b640 1 -- 172.21.15.116:0/2714697208 shutdown_connections 2024-03-19T01:12:27.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.501+0000 7f0a2071b640 1 --2- 172.21.15.116:0/2714697208 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a181022d0 0x7f0a18108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.501+0000 7f0a2071b640 1 --2- 172.21.15.116:0/2714697208 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a18101910 0x7f0a18101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.501+0000 7f0a2071b640 1 --2- 172.21.15.116:0/2714697208 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a18100720 0x7f0a18100b40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.501+0000 7f0a2071b640 1 -- 172.21.15.116:0/2714697208 >> 172.21.15.116:0/2714697208 conn(0x7f0a180fbcb0 msgr2=0x7f0a180fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.501+0000 7f0a2071b640 1 -- 172.21.15.116:0/2714697208 shutdown_connections 2024-03-19T01:12:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.501+0000 7f0a2071b640 1 -- 172.21.15.116:0/2714697208 wait complete. 2024-03-19T01:12:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a2071b640 1 Processor -- start 2024-03-19T01:12:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a2071b640 1 -- start start 2024-03-19T01:12:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a2071b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a18100720 0x7f0a18197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a2071b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a18101910 0x7f0a18198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a2071b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a181022d0 0x7f0a1819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a2071b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1810c370 con 0x7f0a181022d0 2024-03-19T01:12:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a2071b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1810c1f0 con 0x7f0a18101910 2024-03-19T01:12:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a2071b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1810c4f0 con 0x7f0a18100720 2024-03-19T01:12:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a1e4b7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a18100720 0x7f0a18197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a1ecb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a181022d0 0x7f0a1819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a1e4b7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a18100720 0x7f0a18197c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54548/0 (socket says 172.21.15.116:54548) 2024-03-19T01:12:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a1e4b7640 1 -- 172.21.15.116:0/3152408410 learned_addr learned my addr 172.21.15.116:0/3152408410 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a1dcb6640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a18101910 0x7f0a18198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a1e4b7640 1 -- 172.21.15.116:0/3152408410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a18101910 msgr2=0x7f0a18198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a1e4b7640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a18101910 0x7f0a18198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a1e4b7640 1 -- 172.21.15.116:0/3152408410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a181022d0 msgr2=0x7f0a1819f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a1e4b7640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a181022d0 0x7f0a1819f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a1e4b7640 1 -- 172.21.15.116:0/3152408410 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1819f830 con 0x7f0a18100720 2024-03-19T01:12:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.502+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a181022d0 0x7f0a1819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.503+0000 7f0a1e4b7640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a18100720 0x7f0a18197c20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0a10000c00 tx=0x7f0a1002c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.503+0000 7f0a177fe640 1 -- 172.21.15.116:0/3152408410 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a10041070 con 0x7f0a18100720 2024-03-19T01:12:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.503+0000 7f0a177fe640 1 -- 172.21.15.116:0/3152408410 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a10004450 con 0x7f0a18100720 2024-03-19T01:12:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.503+0000 7f0a177fe640 1 -- 172.21.15.116:0/3152408410 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a10003df0 con 0x7f0a18100720 2024-03-19T01:12:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.503+0000 7f0a2071b640 1 -- 172.21.15.116:0/3152408410 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a1819fac0 con 0x7f0a18100720 2024-03-19T01:12:27.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.503+0000 7f0a2071b640 1 -- 172.21.15.116:0/3152408410 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a181a0000 con 0x7f0a18100720 2024-03-19T01:12:27.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.505+0000 7f0a233e4740 1 Processor -- start 2024-03-19T01:12:27.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.505+0000 7f0a233e4740 1 -- start start 2024-03-19T01:12:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.505+0000 7f0a233e4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x564484866410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.505+0000 7f0a233e4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644848669e0 0x56448486b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.505+0000 7f0a233e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56448486bb50 0x56448486dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.505+0000 7f0a233e4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5644847cade0 con 0x564484865ff0 2024-03-19T01:12:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.505+0000 7f0a233e4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564484773c80 con 0x5644848669e0 2024-03-19T01:12:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.505+0000 7f0a233e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5644847c6e80 con 0x56448486bb50 2024-03-19T01:12:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.505+0000 7f0a1ecb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x564484866410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.505+0000 7f0a1ecb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x564484866410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52790/0 (socket says 172.21.15.116:52790) 2024-03-19T01:12:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.505+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/3549797145 learned_addr learned my addr 172.21.15.116:0/3549797145 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.506+0000 7f0a1dcb6640 1 --2- 172.21.15.116:0/3549797145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56448486bb50 0x56448486dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.506+0000 7f0a177fe640 1 -- 172.21.15.116:0/3152408410 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f0a10044070 con 0x7f0a18100720 2024-03-19T01:12:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.506+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/3549797145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56448486bb50 msgr2=0x56448486dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.506+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/3549797145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56448486bb50 0x56448486dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.506+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/3549797145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644848669e0 msgr2=0x56448486b4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.506+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/3549797145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644848669e0 0x56448486b4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.506+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/3549797145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564484785fb0 con 0x564484865ff0 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.506+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/3549797145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x564484866410 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0a08011810 tx=0x7f0a08011cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00230ed7fae6ee2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.506+0000 7f0a14ff9640 1 -- 172.21.15.116:0/3549797145 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08011310 con 0x564484865ff0 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.506+0000 7f0a14ff9640 1 -- 172.21.15.116:0/3549797145 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a0800f9a0 con 0x564484865ff0 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.506+0000 7f0a14ff9640 1 -- 172.21.15.116:0/3549797145 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08005730 con 0x564484865ff0 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.506+0000 7f0a177fe640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0a000a2880 0x7f0a000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.506+0000 7f0a1e4b7640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0a000a2880 0x7f0a000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.507+0000 7f0a177fe640 1 -- 172.21.15.116:0/3152408410 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0a100f0560 con 0x7f0a18100720 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.507+0000 7f0a233e4740 1 -- 172.21.15.116:0/3549797145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 msgr2=0x564484866410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.507+0000 7f0a233e4740 1 --2- 172.21.15.116:0/3549797145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x564484866410 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0a08011810 tx=0x7f0a08011cd0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.507+0000 7f0a1e4b7640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0a000a2880 0x7f0a000a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0a1002cfd0 tx=0x7f0a1008fc10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.507+0000 7f0a233e4740 1 -- 172.21.15.116:0/3549797145 shutdown_connections 2024-03-19T01:12:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.507+0000 7f0a233e4740 1 --2- 172.21.15.116:0/3549797145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56448486bb50 0x56448486dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.507+0000 7f0a233e4740 1 --2- 172.21.15.116:0/3549797145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644848669e0 0x56448486b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.507+0000 7f0a233e4740 1 --2- 172.21.15.116:0/3549797145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x564484866410 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.507+0000 7f0a233e4740 1 -- 172.21.15.116:0/3549797145 >> 172.21.15.116:0/3549797145 conn(0x564484861560 msgr2=0x564484861960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.507+0000 7f0a233e4740 1 -- 172.21.15.116:0/3549797145 shutdown_connections 2024-03-19T01:12:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.507+0000 7f0a233e4740 1 -- 172.21.15.116:0/3549797145 wait complete. 2024-03-19T01:12:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.507+0000 7f0a233e4740 1 -- start start 2024-03-19T01:12:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.507+0000 7f0a233e4740 1 Processor -- start 2024-03-19T01:12:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a233e4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x564484870500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a233e4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644848669e0 0x564484870a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a233e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56448486bb50 0x564484877a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a233e4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56448486eb00 con 0x564484865ff0 2024-03-19T01:12:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a233e4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564484786fc0 con 0x5644848669e0 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a233e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56448486ec80 con 0x56448486bb50 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a233e4740 10 client.0 _ll_register_callbacks cb 0x5644847c2a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a1ecb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x564484870500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a1ecb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x564484870500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52806/0 (socket says 172.21.15.116:52806) 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/875035629 learned_addr learned my addr 172.21.15.116:0/875035629 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a1dcb6640 1 --2- 172.21.15.116:0/875035629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644848669e0 0x564484870a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/875035629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56448486bb50 0x564484877a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/875035629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56448486bb50 msgr2=0x564484877a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/875035629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56448486bb50 0x564484877a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/875035629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644848669e0 msgr2=0x564484870a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/875035629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644848669e0 0x564484870a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/875035629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644844df2d0 con 0x564484865ff0 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/875035629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x564484870500 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0a080098d0 tx=0x7f0a08009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5596cb4f67a1d1df server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f09fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f09fdffb640 1 -- 172.21.15.116:0/875035629 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08003970 con 0x564484865ff0 2024-03-19T01:12:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f09fdffb640 1 -- 172.21.15.116:0/875035629 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0a08003b10 con 0x564484865ff0 2024-03-19T01:12:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a233e4740 1 -- 172.21.15.116:0/875035629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a0800f9a0 con 0x564484865ff0 2024-03-19T01:12:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f09fdffb640 1 -- 172.21.15.116:0/875035629 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0a08003d10 con 0x564484865ff0 2024-03-19T01:12:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a233e4740 10 client.15303 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.508+0000 7f0a233e4740 1 -- 172.21.15.116:0/875035629 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5644845a96b0 con 0x564484865ff0 2024-03-19T01:12:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.509+0000 7f09fdffb640 1 -- 172.21.15.116:0/875035629 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a0800f9a0 con 0x564484865ff0 2024-03-19T01:12:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.509+0000 7f09fdffb640 1 -- 172.21.15.116:0/875035629 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f0a08018070 con 0x564484865ff0 2024-03-19T01:12:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.509+0000 7f0a233e4740 10 client.15303 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.509+0000 7f0a233e4740 10 client.15303 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-19T01:12:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.509+0000 7f0a233e4740 20 client.15303 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.509+0000 7f0a233e4740 1 --2- 172.21.15.116:0/875035629 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x56448487d040 0x56448487e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.509+0000 7f0a233e4740 4 client.15303 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:12:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.509+0000 7f0a233e4740 1 -- 172.21.15.116:0/875035629 --> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5644847d8320 con 0x56448487d040 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.509+0000 7f0a1dcb6640 1 --2- 172.21.15.116:0/875035629 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x56448487d040 0x56448487e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.509+0000 7f09fdffb640 10 client.15303 ms_handle_connect on v2:172.21.15.116:6840/3343188504 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.509+0000 7f0a1dcb6640 1 --2- 172.21.15.116:0/875035629 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x56448487d040 0x56448487e4b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fbcd20cfc119c08c server_cookie=bb9c610ec0e188f8 in_seq=0 out_seq=0 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f09fdffb640 1 -- 172.21.15.116:0/875035629 <== mds.1 v2:172.21.15.116:6840/3343188504 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5644847d8320 con 0x56448487d040 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f09fdffb640 10 client.15303 handle_command_reply: tid=0 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f0a233e4740 1 client.15303 shutdown 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f0a233e4740 1 -- 172.21.15.116:0/875035629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 msgr2=0x564484870500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f0a233e4740 1 --2- 172.21.15.116:0/875035629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x564484870500 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0a080098d0 tx=0x7f0a08009820 comp rx=0 tx=0).stop 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f0a233e4740 1 -- 172.21.15.116:0/875035629 shutdown_connections 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f0a233e4740 1 --2- 172.21.15.116:0/875035629 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56448486bb50 0x564484877a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f0a233e4740 1 --2- 172.21.15.116:0/875035629 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5644848669e0 0x564484870a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f0a233e4740 1 --2- 172.21.15.116:0/875035629 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x56448487d040 0x56448487e4b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f0a233e4740 1 --2- 172.21.15.116:0/875035629 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x564484870500 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f09fdffb640 0 client.15303 ms_handle_reset on v2:172.21.15.116:6840/3343188504 2024-03-19T01:12:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f0a233e4740 1 -- 172.21.15.116:0/875035629 >> 172.21.15.116:0/875035629 conn(0x564484861560 msgr2=0x5644848607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f0a233e4740 1 -- 172.21.15.116:0/875035629 shutdown_connections 2024-03-19T01:12:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f0a233e4740 1 -- 172.21.15.116:0/875035629 wait complete. 2024-03-19T01:12:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.510+0000 7f0a233e4740 20 client.15303 trim_cache size 0 max 16384 2024-03-19T01:12:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.540+0000 7f0a233e4740 1 Processor -- start 2024-03-19T01:12:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.540+0000 7f0a233e4740 1 -- start start 2024-03-19T01:12:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.540+0000 7f0a233e4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564484865ff0 0x564484861940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a233e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644848669e0 0x564484870110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a233e4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56448486bb50 0x564484870650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a233e4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56448487c7f0 con 0x56448486bb50 2024-03-19T01:12:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a233e4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56448487cec0 con 0x564484865ff0 2024-03-19T01:12:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a233e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56448487d590 con 0x5644848669e0 2024-03-19T01:12:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a1ecb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56448486bb50 0x564484870650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a1ecb8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564484865ff0 0x564484861940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a1dcb6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644848669e0 0x564484870110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a1ecb8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564484865ff0 0x564484861940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:36808/0 (socket says 172.21.15.116:36808) 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/3509021248 learned_addr learned my addr 172.21.15.116:0/3509021248 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/3509021248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644848669e0 msgr2=0x564484870110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/3509021248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644848669e0 0x564484870110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/3509021248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56448486bb50 msgr2=0x564484870650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/3509021248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56448486bb50 0x564484870650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/3509021248 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564484861100 con 0x564484865ff0 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a1dcb6640 1 --2- 172.21.15.116:0/3509021248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644848669e0 0x564484870110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/3509021248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564484865ff0 0x564484861940 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800f790 tx=0x7f0a08005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d94f22ca4fb15f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f09fffff640 1 -- 172.21.15.116:0/3509021248 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08038070 con 0x564484865ff0 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f09fffff640 1 -- 172.21.15.116:0/3509021248 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a08039040 con 0x564484865ff0 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.541+0000 7f09fffff640 1 -- 172.21.15.116:0/3509021248 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a080277b0 con 0x564484865ff0 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 -- 172.21.15.116:0/3509021248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564484865ff0 msgr2=0x564484861940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 --2- 172.21.15.116:0/3509021248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564484865ff0 0x564484861940 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800f790 tx=0x7f0a08005fd0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 -- 172.21.15.116:0/3509021248 shutdown_connections 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 --2- 172.21.15.116:0/3509021248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56448486bb50 0x564484870650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 --2- 172.21.15.116:0/3509021248 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644848669e0 0x564484870110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 --2- 172.21.15.116:0/3509021248 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564484865ff0 0x564484861940 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 -- 172.21.15.116:0/3509021248 >> 172.21.15.116:0/3509021248 conn(0x564484861560 msgr2=0x56448487c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 -- 172.21.15.116:0/3509021248 shutdown_connections 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 -- 172.21.15.116:0/3509021248 wait complete. 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 -- start start 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 Processor -- start 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x5644848e0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644848669e0 0x5644848e32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56448486bb50 0x5644848e5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56448487e330 con 0x564484865ff0 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56448487dc60 con 0x56448486bb50 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56448487ea00 con 0x5644848669e0 2024-03-19T01:12:27.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a233e4740 10 client.0 _ll_register_callbacks cb 0x7f0a0c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:27.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a1ecb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x5644848e0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a1ecb8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x5644848e0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52822/0 (socket says 172.21.15.116:52822) 2024-03-19T01:12:27.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.542+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/2519099152 learned_addr learned my addr 172.21.15.116:0/2519099152 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/2519099152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56448486bb50 0x5644848e5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f0a1dcb6640 1 --2- 172.21.15.116:0/2519099152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644848669e0 0x5644848e32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/2519099152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644848669e0 msgr2=0x5644848e32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/2519099152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644848669e0 0x5644848e32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/2519099152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56448486bb50 msgr2=0x5644848e5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/2519099152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56448486bb50 0x5644848e5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f0a1ecb8640 1 -- 172.21.15.116:0/2519099152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5644848e5d80 con 0x564484865ff0 2024-03-19T01:12:27.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f0a1ecb8640 1 --2- 172.21.15.116:0/2519099152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x5644848e0da0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0a08009690 tx=0x7f0a08009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324ded366aea2c27 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f09fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:27.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f09fdffb640 1 -- 172.21.15.116:0/2519099152 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08038070 con 0x564484865ff0 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f09fdffb640 1 -- 172.21.15.116:0/2519099152 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0a08034430 con 0x564484865ff0 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f0a233e4740 1 -- 172.21.15.116:0/2519099152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a08039040 con 0x564484865ff0 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f09fdffb640 1 -- 172.21.15.116:0/2519099152 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0a08034630 con 0x564484865ff0 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f0a233e4740 10 client.15309 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f0a233e4740 1 -- 172.21.15.116:0/2519099152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5644848e6fb0 con 0x564484865ff0 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f09fdffb640 1 -- 172.21.15.116:0/2519099152 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a08039040 con 0x564484865ff0 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.543+0000 7f09fdffb640 1 -- 172.21.15.116:0/2519099152 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7f0a08018040 con 0x564484865ff0 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 10 client.15309 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 10 client.15309 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 20 client.15309 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 1 --2- 172.21.15.116:0/2519099152 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x5644848eb860 0x5644848e8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 4 client.15309 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 1 -- 172.21.15.116:0/2519099152 --> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] -- command(tid 0: {"prefix": "status"}) -- 0x5644847d2670 con 0x5644848eb860 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a1dcb6640 1 --2- 172.21.15.116:0/2519099152 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x5644848eb860 0x5644848e8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f09fdffb640 10 client.15309 ms_handle_connect on v2:172.21.15.116:6840/3343188504 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a1dcb6640 1 --2- 172.21.15.116:0/2519099152 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x5644848eb860 0x5644848e8370 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5e6b80d3ce0120bb server_cookie=9364d30933253ffb in_seq=0 out_seq=0 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f09fdffb640 1 -- 172.21.15.116:0/2519099152 <== mds.1 v2:172.21.15.116:6840/3343188504 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5644847d2670 con 0x5644848eb860 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f09fdffb640 10 client.15309 handle_command_reply: tid=0 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 1 client.15309 shutdown 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 1 -- 172.21.15.116:0/2519099152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 msgr2=0x5644848e0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 1 --2- 172.21.15.116:0/2519099152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x5644848e0da0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0a08009690 tx=0x7f0a08009fa0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 1 -- 172.21.15.116:0/2519099152 shutdown_connections 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 1 --2- 172.21.15.116:0/2519099152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56448486bb50 0x5644848e5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 1 --2- 172.21.15.116:0/2519099152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5644848669e0 0x5644848e32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 1 --2- 172.21.15.116:0/2519099152 >> [v2:172.21.15.116:6840/3343188504,v1:172.21.15.116:6841/3343188504] conn(0x5644848eb860 0x5644848e8370 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 1 --2- 172.21.15.116:0/2519099152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564484865ff0 0x5644848e0da0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f09fdffb640 0 client.15309 ms_handle_reset on v2:172.21.15.116:6840/3343188504 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 1 -- 172.21.15.116:0/2519099152 >> 172.21.15.116:0/2519099152 conn(0x564484861560 msgr2=0x564484861940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 1 -- 172.21.15.116:0/2519099152 shutdown_connections 2024-03-19T01:12:27.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.544+0000 7f0a233e4740 1 -- 172.21.15.116:0/2519099152 wait complete. 2024-03-19T01:12:27.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.545+0000 7f0a233e4740 20 client.15309 trim_cache size 0 max 16384 2024-03-19T01:12:27.553 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:12:27.553 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "35912c14-e58c-11ee-95c9-87774f69a715", 2024-03-19T01:12:27.553 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 1, 2024-03-19T01:12:27.553 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24488, 2024-03-19T01:12:27.553 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T01:12:27.553 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T01:12:27.553 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T01:12:27.553 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 112.60420620399999, 2024-03-19T01:12:27.554 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 24, 2024-03-19T01:12:27.554 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 87, 2024-03-19T01:12:27.554 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 85, 2024-03-19T01:12:27.554 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 123.118854814 2024-03-19T01:12:27.554 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:12:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.545+0000 7f0a2071b640 1 -- 172.21.15.116:0/3152408410 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0a000a2880 msgr2=0x7f0a000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.545+0000 7f0a2071b640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0a000a2880 0x7f0a000a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0a1002cfd0 tx=0x7f0a1008fc10 comp rx=0 tx=0).stop 2024-03-19T01:12:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.545+0000 7f0a2071b640 1 -- 172.21.15.116:0/3152408410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a18100720 msgr2=0x7f0a18197c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.545+0000 7f0a2071b640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a18100720 0x7f0a18197c20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0a10000c00 tx=0x7f0a1002c640 comp rx=0 tx=0).stop 2024-03-19T01:12:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.545+0000 7f0a2071b640 1 -- 172.21.15.116:0/3152408410 shutdown_connections 2024-03-19T01:12:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.545+0000 7f0a2071b640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a181022d0 0x7f0a1819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.545+0000 7f0a2071b640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0a000a2880 0x7f0a000a4d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.545+0000 7f0a2071b640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a18101910 0x7f0a18198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.545+0000 7f0a2071b640 1 --2- 172.21.15.116:0/3152408410 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0a18100720 0x7f0a18197c20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.545+0000 7f0a2071b640 1 -- 172.21.15.116:0/3152408410 >> 172.21.15.116:0/3152408410 conn(0x7f0a180fbcb0 msgr2=0x7f0a18069e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.545+0000 7f0a2071b640 1 -- 172.21.15.116:0/3152408410 shutdown_connections 2024-03-19T01:12:27.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.545+0000 7f0a2071b640 1 -- 172.21.15.116:0/3152408410 wait complete. 2024-03-19T01:12:27.560 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-19T01:12:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:27 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/842327880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:12:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:27 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1175247167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:12:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:27 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1665202576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:12:27.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 -- 172.21.15.116:0/762203644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa148100720 msgr2=0x7fa148100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 --2- 172.21.15.116:0/762203644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa148100720 0x7fa148100b40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa13c009860 tx=0x7fa13c02bb70 comp rx=0 tx=0).stop 2024-03-19T01:12:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 -- 172.21.15.116:0/762203644 shutdown_connections 2024-03-19T01:12:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 --2- 172.21.15.116:0/762203644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa1481022d0 0x7fa148108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 --2- 172.21.15.116:0/762203644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa148101910 0x7fa148101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 --2- 172.21.15.116:0/762203644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa148100720 0x7fa148100b40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 -- 172.21.15.116:0/762203644 >> 172.21.15.116:0/762203644 conn(0x7fa1480fbcb0 msgr2=0x7fa1480fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 -- 172.21.15.116:0/762203644 shutdown_connections 2024-03-19T01:12:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 -- 172.21.15.116:0/762203644 wait complete. 2024-03-19T01:12:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 Processor -- start 2024-03-19T01:12:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 -- start start 2024-03-19T01:12:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa148100720 0x7fa148197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa148101910 0x7fa1481980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1481022d0 0x7fa14819f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa14810d4c0 con 0x7fa148100720 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa14810d340 con 0x7fa148101910 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.668+0000 7fa14ee1f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa14810d640 con 0x7fa1481022d0 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa14cbbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa148100720 0x7fa148197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa14cbbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa148100720 0x7fa148197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52844/0 (socket says 172.21.15.116:52844) 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa14cbbb640 1 -- 172.21.15.116:0/1484318933 learned_addr learned my addr 172.21.15.116:0/1484318933 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa147fff640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa148101910 0x7fa1481980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa14d3bc640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1481022d0 0x7fa14819f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa14cbbb640 1 -- 172.21.15.116:0/1484318933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1481022d0 msgr2=0x7fa14819f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1481022d0 0x7fa14819f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa14cbbb640 1 -- 172.21.15.116:0/1484318933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa148101910 msgr2=0x7fa1481980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa148101910 0x7fa1481980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa14cbbb640 1 -- 172.21.15.116:0/1484318933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa14819f7a0 con 0x7fa148100720 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa148100720 0x7fa148197b80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0349e0 tx=0x7fa13c032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa145ffb640 1 -- 172.21.15.116:0/1484318933 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c041020 con 0x7fa148100720 2024-03-19T01:12:27.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa145ffb640 1 -- 172.21.15.116:0/1484318933 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa13c02c9f0 con 0x7fa148100720 2024-03-19T01:12:27.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa14ee1f640 1 -- 172.21.15.116:0/1484318933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa14819fa30 con 0x7fa148100720 2024-03-19T01:12:27.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa145ffb640 1 -- 172.21.15.116:0/1484318933 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c039040 con 0x7fa148100720 2024-03-19T01:12:27.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.669+0000 7fa14ee1f640 1 -- 172.21.15.116:0/1484318933 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa14819ff20 con 0x7fa148100720 2024-03-19T01:12:27.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa145ffb640 1 -- 172.21.15.116:0/1484318933 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fa13c044020 con 0x7fa148100720 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa151a05740 1 Processor -- start 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa151a05740 1 -- start start 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa151a05740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8240ff0 0x55c4d8241410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa151a05740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d82419e0 0x55c4d82464e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa151a05740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d8246b50 0x55c4d8248fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa151a05740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c4d81a5de0 con 0x55c4d82419e0 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa151a05740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c4d814ec80 con 0x55c4d8246b50 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa151a05740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c4d81a1e80 con 0x55c4d8240ff0 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa14cbbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d82419e0 0x55c4d82464e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa14cbbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d82419e0 0x55c4d82464e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52850/0 (socket says 172.21.15.116:52850) 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa14cbbb640 1 -- 172.21.15.116:0/454899934 learned_addr learned my addr 172.21.15.116:0/454899934 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa14d3bc640 1 --2- 172.21.15.116:0/454899934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8240ff0 0x55c4d8241410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa147fff640 1 --2- 172.21.15.116:0/454899934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d8246b50 0x55c4d8248fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa145ffb640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa1380a2800 0x7fa1380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.672+0000 7fa14cbbb640 1 -- 172.21.15.116:0/454899934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8240ff0 msgr2=0x55c4d8241410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.673+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/454899934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8240ff0 0x55c4d8241410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.673+0000 7fa14cbbb640 1 -- 172.21.15.116:0/454899934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d8246b50 msgr2=0x55c4d8248fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.673+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/454899934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d8246b50 0x55c4d8248fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.673+0000 7fa14cbbb640 1 -- 172.21.15.116:0/454899934 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4d8160fb0 con 0x55c4d82419e0 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.673+0000 7fa14d3bc640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa1380a2800 0x7fa1380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.673+0000 7fa145ffb640 1 -- 172.21.15.116:0/1484318933 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa13c0f02c0 con 0x7fa148100720 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.673+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/454899934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d82419e0 0x55c4d82464e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0ad2e0 tx=0x7fa13c0b2120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceba6622a4289203 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.673+0000 7fa14d3bc640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa1380a2800 0x7fa1380a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa140009fd0 tx=0x7fa140004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.673+0000 7fa132ffd640 1 -- 172.21.15.116:0/454899934 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c0bc780 con 0x55c4d82419e0 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.673+0000 7fa132ffd640 1 -- 172.21.15.116:0/454899934 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa13c0bd340 con 0x55c4d82419e0 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.673+0000 7fa132ffd640 1 -- 172.21.15.116:0/454899934 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c0aa000 con 0x55c4d82419e0 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.674+0000 7fa151a05740 1 -- 172.21.15.116:0/454899934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d82419e0 msgr2=0x55c4d82464e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.674+0000 7fa151a05740 1 --2- 172.21.15.116:0/454899934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d82419e0 0x55c4d82464e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0ad2e0 tx=0x7fa13c0b2120 comp rx=0 tx=0).stop 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.674+0000 7fa151a05740 1 -- 172.21.15.116:0/454899934 shutdown_connections 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.674+0000 7fa151a05740 1 --2- 172.21.15.116:0/454899934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d8246b50 0x55c4d8248fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.674+0000 7fa151a05740 1 --2- 172.21.15.116:0/454899934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d82419e0 0x55c4d82464e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.674+0000 7fa151a05740 1 --2- 172.21.15.116:0/454899934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8240ff0 0x55c4d8241410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.674+0000 7fa151a05740 1 -- 172.21.15.116:0/454899934 >> 172.21.15.116:0/454899934 conn(0x55c4d823c560 msgr2=0x55c4d823c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.674+0000 7fa151a05740 1 -- 172.21.15.116:0/454899934 shutdown_connections 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.674+0000 7fa151a05740 1 -- 172.21.15.116:0/454899934 wait complete. 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.674+0000 7fa151a05740 1 -- start start 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.674+0000 7fa151a05740 1 Processor -- start 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa151a05740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa151a05740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 0x55c4d824b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa151a05740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 0x55c4d82526c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa151a05740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c4d82498c0 con 0x55c4d8240ff0 2024-03-19T01:12:27.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa151a05740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c4d8161fc0 con 0x55c4d82419e0 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa151a05740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c4d8249a40 con 0x55c4d8246b50 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa151a05740 10 client.0 _ll_register_callbacks cb 0x55c4d819da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa14cbbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa14cbbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52854/0 (socket says 172.21.15.116:52854) 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa147fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 0x55c4d824b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa14cbbb640 1 -- 172.21.15.116:0/898813698 learned_addr learned my addr 172.21.15.116:0/898813698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa14d3bc640 1 --2- 172.21.15.116:0/898813698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 0x55c4d82526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa14cbbb640 1 -- 172.21.15.116:0/898813698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 msgr2=0x55c4d82526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/898813698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 0x55c4d82526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa14cbbb640 1 -- 172.21.15.116:0/898813698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 msgr2=0x55c4d824b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/898813698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 0x55c4d824b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa14cbbb640 1 -- 172.21.15.116:0/898813698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4d7eba2d0 con 0x55c4d8240ff0 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa147fff640 1 --2- 172.21.15.116:0/898813698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 0x55c4d824b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:27.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/898813698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824b140 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa13c02bbb0 tx=0x7fa13c0bcb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3564f1c4176391 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa127fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa127fff640 1 -- 172.21.15.116:0/898813698 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c085fc0 con 0x55c4d8240ff0 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa127fff640 1 -- 172.21.15.116:0/898813698 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa13c0f3a30 con 0x55c4d8240ff0 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa151a05740 1 -- 172.21.15.116:0/898813698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa13c0bd340 con 0x55c4d8240ff0 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa127fff640 1 -- 172.21.15.116:0/898813698 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa13c0bc5a0 con 0x55c4d8240ff0 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa151a05740 10 client.15321 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.675+0000 7fa151a05740 1 -- 172.21.15.116:0/898813698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c4d7f846b0 con 0x55c4d8240ff0 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.676+0000 7fa127fff640 1 -- 172.21.15.116:0/898813698 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa13c0bd340 con 0x55c4d8240ff0 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.676+0000 7fa127fff640 1 -- 172.21.15.116:0/898813698 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fa13c03c030 con 0x55c4d8240ff0 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.676+0000 7fa151a05740 10 client.15321 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.676+0000 7fa151a05740 10 client.15321 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.676+0000 7fa151a05740 20 client.15321 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.676+0000 7fa151a05740 1 --2- 172.21.15.116:0/898813698 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c4d8257dc0 0x55c4d8259230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.676+0000 7fa151a05740 4 client.15321 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:12:27.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.676+0000 7fa151a05740 1 -- 172.21.15.116:0/898813698 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4d81b3320 con 0x55c4d8257dc0 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.676+0000 7fa147fff640 1 --2- 172.21.15.116:0/898813698 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c4d8257dc0 0x55c4d8259230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.676+0000 7fa127fff640 10 client.15321 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.676+0000 7fa147fff640 1 --2- 172.21.15.116:0/898813698 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c4d8257dc0 0x55c4d8259230 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea9cbac304e38828 server_cookie=faccbc8ec3af81bb in_seq=0 out_seq=0 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.677+0000 7fa127fff640 1 -- 172.21.15.116:0/898813698 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c4d81b3320 con 0x55c4d8257dc0 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.677+0000 7fa127fff640 10 client.15321 handle_command_reply: tid=0 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.677+0000 7fa151a05740 1 client.15321 shutdown 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.677+0000 7fa151a05740 1 -- 172.21.15.116:0/898813698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 msgr2=0x55c4d824b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.677+0000 7fa151a05740 1 --2- 172.21.15.116:0/898813698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824b140 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa13c02bbb0 tx=0x7fa13c0bcb00 comp rx=0 tx=0).stop 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.678+0000 7fa151a05740 1 -- 172.21.15.116:0/898813698 shutdown_connections 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.678+0000 7fa151a05740 1 --2- 172.21.15.116:0/898813698 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c4d8257dc0 0x55c4d8259230 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.678+0000 7fa151a05740 1 --2- 172.21.15.116:0/898813698 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 0x55c4d82526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.678+0000 7fa151a05740 1 --2- 172.21.15.116:0/898813698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 0x55c4d824b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.678+0000 7fa127fff640 0 client.15321 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:27.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.678+0000 7fa151a05740 1 --2- 172.21.15.116:0/898813698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824b140 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.678+0000 7fa151a05740 1 -- 172.21.15.116:0/898813698 >> 172.21.15.116:0/898813698 conn(0x55c4d823c560 msgr2=0x55c4d823b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.678+0000 7fa151a05740 1 -- 172.21.15.116:0/898813698 shutdown_connections 2024-03-19T01:12:27.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.678+0000 7fa151a05740 1 -- 172.21.15.116:0/898813698 wait complete. 2024-03-19T01:12:27.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.678+0000 7fa151a05740 20 client.15321 trim_cache size 0 max 16384 2024-03-19T01:12:27.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.695+0000 7fa151a05740 1 Processor -- start 2024-03-19T01:12:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.695+0000 7fa151a05740 1 -- start start 2024-03-19T01:12:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.695+0000 7fa151a05740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.695+0000 7fa151a05740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 0x55c4d8252680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa151a05740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 0x55c4d8242eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa151a05740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c4d8257570 con 0x55c4d8240ff0 2024-03-19T01:12:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa151a05740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c4d8257c40 con 0x55c4d82419e0 2024-03-19T01:12:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa151a05740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c4d8258310 con 0x55c4d8246b50 2024-03-19T01:12:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa14cbbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa14cbbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52868/0 (socket says 172.21.15.116:52868) 2024-03-19T01:12:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa14cbbb640 1 -- 172.21.15.116:0/675482557 learned_addr learned my addr 172.21.15.116:0/675482557 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa147fff640 1 --2- 172.21.15.116:0/675482557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 0x55c4d8252680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa14d3bc640 1 --2- 172.21.15.116:0/675482557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 0x55c4d8242eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa14cbbb640 1 -- 172.21.15.116:0/675482557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 msgr2=0x55c4d8242eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/675482557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 0x55c4d8242eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa14cbbb640 1 -- 172.21.15.116:0/675482557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 msgr2=0x55c4d8252680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/675482557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 0x55c4d8252680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa14cbbb640 1 -- 172.21.15.116:0/675482557 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4d823d2d0 con 0x55c4d8240ff0 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/675482557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824a760 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0bcbd0 tx=0x7fa13c0097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c19cf509497635 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa1327fc640 1 -- 172.21.15.116:0/675482557 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c0bc780 con 0x55c4d8240ff0 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa1327fc640 1 -- 172.21.15.116:0/675482557 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa13c0bd340 con 0x55c4d8240ff0 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.696+0000 7fa1327fc640 1 -- 172.21.15.116:0/675482557 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c0f3b50 con 0x55c4d8240ff0 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 -- 172.21.15.116:0/675482557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 msgr2=0x55c4d824a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 --2- 172.21.15.116:0/675482557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824a760 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0bcbd0 tx=0x7fa13c0097b0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 -- 172.21.15.116:0/675482557 shutdown_connections 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 --2- 172.21.15.116:0/675482557 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 0x55c4d8242eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 --2- 172.21.15.116:0/675482557 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 0x55c4d8252680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 --2- 172.21.15.116:0/675482557 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824a760 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 -- 172.21.15.116:0/675482557 >> 172.21.15.116:0/675482557 conn(0x55c4d823c560 msgr2=0x55c4d823c940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 -- 172.21.15.116:0/675482557 shutdown_connections 2024-03-19T01:12:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 -- 172.21.15.116:0/675482557 wait complete. 2024-03-19T01:12:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 -- start start 2024-03-19T01:12:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 Processor -- start 2024-03-19T01:12:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 0x55c4d824e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 0x55c4d8251030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c4d82590b0 con 0x55c4d8240ff0 2024-03-19T01:12:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c4d82589e0 con 0x55c4d82419e0 2024-03-19T01:12:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.697+0000 7fa151a05740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c4d8259780 con 0x55c4d8246b50 2024-03-19T01:12:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa151a05740 10 client.0 _ll_register_callbacks cb 0x7fa134009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:27.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa14cbbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa14cbbb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52874/0 (socket says 172.21.15.116:52874) 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa14cbbb640 1 -- 172.21.15.116:0/1685839238 learned_addr learned my addr 172.21.15.116:0/1685839238 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa14d3bc640 1 --2- 172.21.15.116:0/1685839238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 0x55c4d8251030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa147fff640 1 --2- 172.21.15.116:0/1685839238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 0x55c4d824e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa14cbbb640 1 -- 172.21.15.116:0/1685839238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 msgr2=0x55c4d8251030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/1685839238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 0x55c4d8251030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa14cbbb640 1 -- 172.21.15.116:0/1685839238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 msgr2=0x55c4d824e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/1685839238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 0x55c4d824e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa14cbbb640 1 -- 172.21.15.116:0/1685839238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4d8251620 con 0x55c4d8240ff0 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa14cbbb640 1 --2- 172.21.15.116:0/1685839238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824dad0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0ee110 tx=0x7fa13c032870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7cf13efd913013b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa127fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa127fff640 1 -- 172.21.15.116:0/1685839238 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13c0bc780 con 0x55c4d8240ff0 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa127fff640 1 -- 172.21.15.116:0/1685839238 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa13c0bd0b0 con 0x55c4d8240ff0 2024-03-19T01:12:27.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa151a05740 1 -- 172.21.15.116:0/1685839238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa13c0bd340 con 0x55c4d8240ff0 2024-03-19T01:12:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa127fff640 1 -- 172.21.15.116:0/1685839238 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa13c0d0d30 con 0x55c4d8240ff0 2024-03-19T01:12:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa151a05740 10 client.15327 fetch_fsmap learned FSMap version 24 2024-03-19T01:12:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa151a05740 1 -- 172.21.15.116:0/1685839238 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c4d8251620 con 0x55c4d8240ff0 2024-03-19T01:12:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.698+0000 7fa127fff640 1 -- 172.21.15.116:0/1685839238 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa13c0bd340 con 0x55c4d8240ff0 2024-03-19T01:12:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa127fff640 1 -- 172.21.15.116:0/1685839238 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 24) ==== 8354+0+0 (secure 0 0 0) 0x7fa13c0bc090 con 0x55c4d8240ff0 2024-03-19T01:12:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa151a05740 10 client.15327 fetch_fsmap finished waiting for FSMap version 24 2024-03-19T01:12:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa151a05740 10 client.15327 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-19T01:12:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa151a05740 20 client.15327 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa151a05740 1 --2- 172.21.15.116:0/1685839238 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c4d82c1bf0 0x55c4d82c0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa151a05740 4 client.15327 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-19T01:12:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa151a05740 1 -- 172.21.15.116:0/1685839238 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "status"}) -- 0x55c4d81ad670 con 0x55c4d82c1bf0 2024-03-19T01:12:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa147fff640 1 --2- 172.21.15.116:0/1685839238 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c4d82c1bf0 0x55c4d82c0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:27.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa147fff640 1 --2- 172.21.15.116:0/1685839238 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c4d82c1bf0 0x55c4d82c0900 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4003b9f26965ec9 server_cookie=21e890b24f458f88 in_seq=0 out_seq=0 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa127fff640 10 client.15327 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa127fff640 1 -- 172.21.15.116:0/1685839238 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55c4d81ad670 con 0x55c4d82c1bf0 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa127fff640 10 client.15327 handle_command_reply: tid=0 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa151a05740 1 client.15327 shutdown 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa151a05740 1 -- 172.21.15.116:0/1685839238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 msgr2=0x55c4d824dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.699+0000 7fa151a05740 1 --2- 172.21.15.116:0/1685839238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824dad0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0ee110 tx=0x7fa13c032870 comp rx=0 tx=0).stop 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa151a05740 1 -- 172.21.15.116:0/1685839238 shutdown_connections 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa151a05740 1 --2- 172.21.15.116:0/1685839238 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c4d82c1bf0 0x55c4d82c0900 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa151a05740 1 --2- 172.21.15.116:0/1685839238 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c4d8246b50 0x55c4d8251030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa127fff640 0 client.15327 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa151a05740 1 --2- 172.21.15.116:0/1685839238 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c4d82419e0 0x55c4d824e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa151a05740 1 --2- 172.21.15.116:0/1685839238 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c4d8240ff0 0x55c4d824dad0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa151a05740 1 -- 172.21.15.116:0/1685839238 >> 172.21.15.116:0/1685839238 conn(0x55c4d823c560 msgr2=0x55c4d8242eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa151a05740 1 -- 172.21.15.116:0/1685839238 shutdown_connections 2024-03-19T01:12:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa151a05740 1 -- 172.21.15.116:0/1685839238 wait complete. 2024-03-19T01:12:27.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa151a05740 20 client.15327 trim_cache size 0 max 16384 2024-03-19T01:12:27.708 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:12:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "35912c14-e58c-11ee-95c9-87774f69a715", 2024-03-19T01:12:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-03-19T01:12:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 24500, 2024-03-19T01:12:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-03-19T01:12:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-03-19T01:12:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "cephfs", 2024-03-19T01:12:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 114.550851389, 2024-03-19T01:12:27.708 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 24, 2024-03-19T01:12:27.709 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 87, 2024-03-19T01:12:27.709 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 85, 2024-03-19T01:12:27.709 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 120.322856264 2024-03-19T01:12:27.709 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:12:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa14ee1f640 1 -- 172.21.15.116:0/1484318933 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa1380a2800 msgr2=0x7fa1380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa14ee1f640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa1380a2800 0x7fa1380a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa140009fd0 tx=0x7fa140004a40 comp rx=0 tx=0).stop 2024-03-19T01:12:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa14ee1f640 1 -- 172.21.15.116:0/1484318933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa148100720 msgr2=0x7fa148197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa14ee1f640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa148100720 0x7fa148197b80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa13c0349e0 tx=0x7fa13c032430 comp rx=0 tx=0).stop 2024-03-19T01:12:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa14ee1f640 1 -- 172.21.15.116:0/1484318933 shutdown_connections 2024-03-19T01:12:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa14ee1f640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa1481022d0 0x7fa14819f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa14ee1f640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa1380a2800 0x7fa1380a4cc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa14ee1f640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa148101910 0x7fa1481980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa14ee1f640 1 --2- 172.21.15.116:0/1484318933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa148100720 0x7fa148197b80 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa14ee1f640 1 -- 172.21.15.116:0/1484318933 >> 172.21.15.116:0/1484318933 conn(0x7fa1480fbcb0 msgr2=0x7fa148104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa14ee1f640 1 -- 172.21.15.116:0/1484318933 shutdown_connections 2024-03-19T01:12:27.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:27.700+0000 7fa14ee1f640 1 -- 172.21.15.116:0/1484318933 wait complete. 2024-03-19T01:12:27.712 INFO:teuthology.run_tasks:Running task kclient... 2024-03-19T01:12:27.735 INFO:tasks.kclient:Mounting kernel clients... 2024-03-19T01:12:27.736 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-19T01:12:27.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:12:27.736 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-19T01:12:27.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:12:27.751 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-03-19T01:12:27.821 INFO:teuthology.orchestra.run.smithi116.stderr:Cannot find device "ceph-brx" 2024-03-19T01:12:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:12:27.823 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync', 'context=system_u:object_r:postgresql_db_t:s0'], 'syntax': 'v2', 'mount_subvol_num': 1, 'mountpoint': '/tmp/cephfs'} 2024-03-19T01:12:27.823 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-19T01:12:27.823 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2024-03-19T01:12:27.823 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2024-03-19T01:12:27.823 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-19T01:12:28.109 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:12:28.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:28 smithi156 ceph-mon[66799]: pgmap v347: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:12:28.479 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:28 smithi116 ceph-mon[62701]: pgmap v347: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:12:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:28 smithi162 ceph-mon[67178]: pgmap v347: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-19T01:12:29.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:29 smithi156 ceph-mon[66799]: pgmap v348: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:12:29.518 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:29 smithi116 ceph-mon[62701]: pgmap v348: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:12:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:29 smithi162 ceph-mon[67178]: pgmap v348: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-19T01:12:29.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.848+0000 7f2021f34700 1 -- 172.21.15.116:0/2181446040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f201c107380 msgr2=0x7f201c107760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:29.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.848+0000 7f2021f34700 1 --2- 172.21.15.116:0/2181446040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f201c107380 0x7f201c107760 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f200c009990 tx=0x7f200c009ca0 comp rx=0 tx=0).stop 2024-03-19T01:12:29.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.848+0000 7f2021f34700 1 -- 172.21.15.116:0/2181446040 shutdown_connections 2024-03-19T01:12:29.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.848+0000 7f2021f34700 1 --2- 172.21.15.116:0/2181446040 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f201c1012c0 0x7f201c10f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:29.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.848+0000 7f2021f34700 1 --2- 172.21.15.116:0/2181446040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f201c107380 0x7f201c107760 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:29.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.848+0000 7f2021f34700 1 --2- 172.21.15.116:0/2181446040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f201c100900 0x7f201c100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:29.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.848+0000 7f2021f34700 1 -- 172.21.15.116:0/2181446040 >> 172.21.15.116:0/2181446040 conn(0x7f201c077810 msgr2=0x7f201c0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:29.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.848+0000 7f2021f34700 1 -- 172.21.15.116:0/2181446040 shutdown_connections 2024-03-19T01:12:29.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.848+0000 7f2021f34700 1 -- 172.21.15.116:0/2181446040 wait complete. 2024-03-19T01:12:29.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.849+0000 7f2021f34700 1 Processor -- start 2024-03-19T01:12:29.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.849+0000 7f2021f34700 1 -- start start 2024-03-19T01:12:29.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.849+0000 7f2021f34700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f201c100900 0x7f201c19e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:29.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.849+0000 7f2021f34700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f201c1012c0 0x7f201c19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:29.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.849+0000 7f2021f34700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f201c107380 0x7f201c198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:29.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.849+0000 7f2021f34700 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f201c112740 con 0x7f201c107380 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.849+0000 7f2021f34700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f201c1125c0 con 0x7f201c1012c0 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.849+0000 7f2021f34700 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f201c1128c0 con 0x7f201c100900 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f201bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f201c107380 0x7f201c198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f201b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f201c100900 0x7f201c19e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f201affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f201c1012c0 0x7f201c19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f201bfff700 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f201c107380 0x7f201c198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52894/0 (socket says 172.21.15.116:52894) 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f201b7fe700 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f201c100900 0x7f201c19e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54676/0 (socket says 172.21.15.116:54676) 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f201bfff700 1 -- 172.21.15.116:0/749957390 learned_addr learned my addr 172.21.15.116:0/749957390 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f201bfff700 1 -- 172.21.15.116:0/749957390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f201c100900 msgr2=0x7f201c19e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f201bfff700 1 --2- 172.21.15.116:0/749957390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f201c100900 0x7f201c19e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f201bfff700 1 -- 172.21.15.116:0/749957390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f201c1012c0 msgr2=0x7f201c19ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f201bfff700 1 --2- 172.21.15.116:0/749957390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f201c1012c0 0x7f201c19ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f201bfff700 1 -- 172.21.15.116:0/749957390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c199160 con 0x7f201c107380 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f201bfff700 1 --2- 172.21.15.116:0/749957390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f201c107380 0x7f201c198a00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2014007ec0 tx=0x7f201400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:29.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f2018ff9700 1 -- 172.21.15.116:0/749957390 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201400dd20 con 0x7f201c107380 2024-03-19T01:12:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f2021f34700 1 -- 172.21.15.116:0/749957390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f201c199450 con 0x7f201c107380 2024-03-19T01:12:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f2018ff9700 1 -- 172.21.15.116:0/749957390 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2014018070 con 0x7f201c107380 2024-03-19T01:12:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f2018ff9700 1 -- 172.21.15.116:0/749957390 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2014012690 con 0x7f201c107380 2024-03-19T01:12:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.850+0000 7f2021f34700 1 -- 172.21.15.116:0/749957390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f201c101a10 con 0x7f201c107380 2024-03-19T01:12:29.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.852+0000 7f2021f34700 1 -- 172.21.15.116:0/749957390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f201c10a950 con 0x7f201c107380 2024-03-19T01:12:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.852+0000 7f2018ff9700 1 -- 172.21.15.116:0/749957390 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f20140133f0 con 0x7f201c107380 2024-03-19T01:12:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.853+0000 7f2018ff9700 1 --2- 172.21.15.116:0/749957390 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20040a2890 0x7f20040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.853+0000 7f201b7fe700 1 --2- 172.21.15.116:0/749957390 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20040a2890 0x7f20040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.853+0000 7f2018ff9700 1 -- 172.21.15.116:0/749957390 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f20140ce720 con 0x7f201c107380 2024-03-19T01:12:29.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.853+0000 7f201b7fe700 1 --2- 172.21.15.116:0/749957390 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20040a2890 0x7f20040a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f201c106da0 tx=0x7f200c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:29.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:29.860+0000 7f2018ff9700 1 -- 172.21.15.116:0/749957390 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2014010070 con 0x7f201c107380 2024-03-19T01:12:30.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.106+0000 7f2021f34700 1 -- 172.21.15.116:0/749957390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f201c10a330 con 0x7f201c107380 2024-03-19T01:12:30.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.107+0000 7f2018ff9700 1 -- 172.21.15.116:0/749957390 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+73 (secure 0 0 0) 0x7f2014010250 con 0x7f201c107380 2024-03-19T01:12:30.108 INFO:teuthology.orchestra.run.smithi116.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-19T01:12:30.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.109+0000 7f2021f34700 1 -- 172.21.15.116:0/749957390 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20040a2890 msgr2=0x7f20040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:30.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.109+0000 7f2021f34700 1 --2- 172.21.15.116:0/749957390 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20040a2890 0x7f20040a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f201c106da0 tx=0x7f200c005960 comp rx=0 tx=0).stop 2024-03-19T01:12:30.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.109+0000 7f2021f34700 1 -- 172.21.15.116:0/749957390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f201c107380 msgr2=0x7f201c198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:30.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.109+0000 7f2021f34700 1 --2- 172.21.15.116:0/749957390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f201c107380 0x7f201c198a00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2014007ec0 tx=0x7f201400d3b0 comp rx=0 tx=0).stop 2024-03-19T01:12:30.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.109+0000 7f2021f34700 1 -- 172.21.15.116:0/749957390 shutdown_connections 2024-03-19T01:12:30.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.109+0000 7f2021f34700 1 --2- 172.21.15.116:0/749957390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f201c107380 0x7f201c198a00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:30.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.109+0000 7f2021f34700 1 --2- 172.21.15.116:0/749957390 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20040a2890 0x7f20040a4d50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:30.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.109+0000 7f2021f34700 1 --2- 172.21.15.116:0/749957390 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f201c100900 0x7f201c19e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:30.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.109+0000 7f2021f34700 1 --2- 172.21.15.116:0/749957390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f201c1012c0 0x7f201c19ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:30.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.109+0000 7f2021f34700 1 -- 172.21.15.116:0/749957390 >> 172.21.15.116:0/749957390 conn(0x7f201c077810 msgr2=0x7f201c105420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:30.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.109+0000 7f2021f34700 1 -- 172.21.15.116:0/749957390 shutdown_connections 2024-03-19T01:12:30.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30.109+0000 7f2021f34700 1 -- 172.21.15.116:0/749957390 wait complete. 2024-03-19T01:12:30.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:30 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/749957390' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T01:12:30.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:30 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/749957390' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T01:12:30.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:30 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/749957390' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-19T01:12:30.852 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-19T01:12:30.852 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-19T01:12:30.852 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2024-03-19T01:12:30.852 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-19T01:12:30.853 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2024-03-19T01:12:30.853 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-19T01:12:30.853 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-19T01:12:30.853 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-19T01:12:30.853 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2024-03-19T01:12:30.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:12:30.853 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2024-03-19T01:12:30.869 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-19T01:12:30.869 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-19T01:12:30.869 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-03-19T01:12:30.869 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:12:30.869 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-03-19T01:12:30.869 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:12:30.869 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T01:12:30.869 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2024-03-19T01:12:30.870 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2024-03-19T01:12:30.870 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T01:12:30.870 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2024-03-19T01:12:30.870 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2024-03-19T01:12:30.870 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-19T01:12:30.870 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2024-03-19T01:12:30.870 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-19T01:12:30.870 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2024-03-19T01:12:30.870 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-19T01:12:30.870 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 40443sec preferred_lft 40443sec 2024-03-19T01:12:30.870 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-03-19T01:12:30.870 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-03-19T01:12:30.871 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-19T01:12:30.871 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:12:30.871 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T01:12:30.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add name ceph-brx type bridge 2024-03-19T01:12:30.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr flush dev ceph-brx 2024-03-19T01:12:30.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set ceph-brx up 2024-03-19T01:12:30.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-19T01:12:30.871 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T01:12:30.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:12:31.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:12:31.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:12:31.090 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-19T01:12:31.117 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-03-19T01:12:31.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:12:31.118 DEBUG:teuthology.orchestra.run.smithi116:> route 2024-03-19T01:12:31.188 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2024-03-19T01:12:31.188 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-19T01:12:31.188 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-19T01:12:31.188 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-19T01:12:31.188 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-19T01:12:31.189 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:12:31.189 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T01:12:31.189 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-19T01:12:31.189 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-19T01:12:31.189 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-19T01:12:31.189 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T01:12:31.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:12:31.383 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:31 smithi116 ceph-mon[62701]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T01:12:31.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:31 smithi156 ceph-mon[66799]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T01:12:31.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:12:31.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:12:31.491 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-19T01:12:31.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:12:31.506 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2024-03-19T01:12:31.561 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:12:31.561 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T01:12:31.561 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--tmp-cephfs 2024-03-19T01:12:31.561 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--tmp-cephfs 0 2024-03-19T01:12:31.561 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T01:12:31.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:12:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:31 smithi162 ceph-mon[67178]: pgmap v349: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-19T01:12:31.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:12:31.686 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2024-03-19T01:12:31.686 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:12:31.686 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T01:12:31.686 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2024-03-19T01:12:31.686 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-19T01:12:31.686 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2024-03-19T01:12:31.686 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2024-03-19T01:12:31.687 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2024-03-19T01:12:31.687 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T01:12:31.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:12:31.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:12:31.870 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-19T01:12:31.870 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-03-19T01:12:31.870 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2024-03-19T01:12:31.870 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.0 master ceph-brx 2024-03-19T01:12:31.870 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-03-19T01:12:31.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:12:31.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:12:31.961 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T01:12:31.961 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /tmp/cephfs 2024-03-19T01:12:31.976 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/tmp/cephfs' 2024-03-19T01:12:31.977 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-19T01:12:31.977 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /tmp/cephfs 2024-03-19T01:12:32.032 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-19T01:12:32.032 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-19T01:12:32.032 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/26db1d39-573c-4af5-8f4a-46f4ed020a58 2024-03-19T01:12:32.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:12:32.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/26db1d39-573c-4af5-8f4a-46f4ed020a58 /tmp/cephfs -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context=system_u:object_r:postgresql_db_t:s0,nofallback 2024-03-19T01:12:32.118 INFO:teuthology.orchestra.run.smithi116.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-03-19T01:12:32.118 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-03-19T01:12:32.242 INFO:teuthology.orchestra.run.smithi116.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",nofallback 2024-03-19T01:12:32.242 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0"". 2024-03-19T01:12:32.242 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: resolved to: "172.21.15.116:3300,172.21.15.156:3300,172.21.15.162:3300" 2024-03-19T01:12:32.243 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: trying mount with new device syntax: 0@35912c14-e58c-11ee-95c9-87774f69a715.cephfs=/volumes/qa/sv_1/26db1d39-573c-4af5-8f4a-46f4ed020a58 2024-03-19T01:12:32.243 INFO:teuthology.orchestra.run.smithi116.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,context="system_u:object_r:postgresql_db_t:s0",name=0,key=0,mon_addr=172.21.15.116:3300/172.21.15.156:3300/172.21.15.162:3300" will pass to kernel 2024-03-19T01:12:32.245 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-19T01:12:32.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:12:32.245 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /tmp/cephfs 2024-03-19T01:12:32.317 DEBUG:teuthology.orchestra.run.smithi116:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T01:12:32.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T01:12:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:12:32.415 INFO:teuthology.orchestra.run.smithi116.stdout:client15336 2024-03-19T01:12:32.415 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:12:32.415 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T01:12:32.432 DEBUG:teuthology.orchestra.run.smithi116:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T01:12:32.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T01:12:32.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:12:32.530 INFO:teuthology.orchestra.run.smithi116.stdout:client15336 2024-03-19T01:12:32.530 DEBUG:teuthology.orchestra.run.smithi116:> (cd /tmp/cephfs && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-19T01:12:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-19T01:12:32.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:12:32.589 INFO:teuthology.orchestra.run.smithi116.stdout:client15336 2024-03-19T01:12:32.589 DEBUG:teuthology.orchestra.run.smithi116:> (cd /tmp/cephfs && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/35912c14-e58c-11ee-95c9-87774f69a715.client15336/status') 2024-03-19T01:12:32.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-19T01:12:32.657 INFO:teuthology.orchestra.run.smithi116.stdout:instance: client.15336 (3)192.168.0.1:0/3808966976 2024-03-19T01:12:32.657 INFO:teuthology.orchestra.run.smithi116.stdout:blocklisted: false 2024-03-19T01:12:32.657 INFO:teuthology.orchestra.run.smithi116.stderr:0+1 records in 2024-03-19T01:12:32.657 INFO:teuthology.orchestra.run.smithi116.stderr:0+1 records out 2024-03-19T01:12:32.657 INFO:teuthology.orchestra.run.smithi116.stderr:70 bytes copied, 2.1588e-05 s, 3.2 MB/s 2024-03-19T01:12:32.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-19T01:12:32.665 INFO:teuthology.run_tasks:Running task exec... 2024-03-19T01:12:32.688 INFO:teuthology.task.exec:Executing custom commands... 2024-03-19T01:12:32.688 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T01:12:32.688 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-19T01:12:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.805+0000 7ff584bc4640 1 -- 172.21.15.116:0/1967804298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff580100750 msgr2=0x7ff580100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.805+0000 7ff584bc4640 1 --2- 172.21.15.116:0/1967804298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff580100750 0x7ff580100b70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff578009900 tx=0x7ff57802bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:32.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.805+0000 7ff584bc4640 1 -- 172.21.15.116:0/1967804298 shutdown_connections 2024-03-19T01:12:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.805+0000 7ff584bc4640 1 --2- 172.21.15.116:0/1967804298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff580102300 0x7ff580108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.805+0000 7ff584bc4640 1 --2- 172.21.15.116:0/1967804298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff580101940 0x7ff580101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.805+0000 7ff584bc4640 1 --2- 172.21.15.116:0/1967804298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff580100750 0x7ff580100b70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.805+0000 7ff584bc4640 1 -- 172.21.15.116:0/1967804298 >> 172.21.15.116:0/1967804298 conn(0x7ff5800fbcc0 msgr2=0x7ff5800fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.805+0000 7ff584bc4640 1 -- 172.21.15.116:0/1967804298 shutdown_connections 2024-03-19T01:12:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.805+0000 7ff584bc4640 1 -- 172.21.15.116:0/1967804298 wait complete. 2024-03-19T01:12:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.805+0000 7ff584bc4640 1 Processor -- start 2024-03-19T01:12:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff584bc4640 1 -- start start 2024-03-19T01:12:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff584bc4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff580100750 0x7ff5801969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff584bc4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff580101940 0x7ff580196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff584bc4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff580102300 0x7ff58019df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff584bc4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff58010c3c0 con 0x7ff580101940 2024-03-19T01:12:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff584bc4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff58010c240 con 0x7ff580102300 2024-03-19T01:12:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff584bc4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff58010c540 con 0x7ff580100750 2024-03-19T01:12:32.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff57dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff580101940 0x7ff580196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff57dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff580101940 0x7ff580196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52912/0 (socket says 172.21.15.116:52912) 2024-03-19T01:12:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff57dd9b640 1 -- 172.21.15.116:0/109538767 learned_addr learned my addr 172.21.15.116:0/109538767 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff57e59c640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff580100750 0x7ff5801969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff57ed9d640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff580102300 0x7ff58019df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff57dd9b640 1 -- 172.21.15.116:0/109538767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff580100750 msgr2=0x7ff5801969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff57dd9b640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff580100750 0x7ff5801969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff57dd9b640 1 -- 172.21.15.116:0/109538767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff580102300 msgr2=0x7ff58019df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff57dd9b640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff580102300 0x7ff58019df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff57dd9b640 1 -- 172.21.15.116:0/109538767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff58019e5c0 con 0x7ff580101940 2024-03-19T01:12:32.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff57dd9b640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff580101940 0x7ff580196ee0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff57400eb90 tx=0x7ff57400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:32.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff56f7fe640 1 -- 172.21.15.116:0/109538767 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff574004d20 con 0x7ff580101940 2024-03-19T01:12:32.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff584bc4640 1 -- 172.21.15.116:0/109538767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff58019e8b0 con 0x7ff580101940 2024-03-19T01:12:32.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff56f7fe640 1 -- 172.21.15.116:0/109538767 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff57400ee90 con 0x7ff580101940 2024-03-19T01:12:32.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff56f7fe640 1 -- 172.21.15.116:0/109538767 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff574010830 con 0x7ff580101940 2024-03-19T01:12:32.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.806+0000 7ff584bc4640 1 -- 172.21.15.116:0/109538767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff58019ee00 con 0x7ff580101940 2024-03-19T01:12:32.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.808+0000 7ff584bc4640 1 -- 172.21.15.116:0/109538767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5801079a0 con 0x7ff580101940 2024-03-19T01:12:32.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.808+0000 7ff56f7fe640 1 -- 172.21.15.116:0/109538767 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7ff574005020 con 0x7ff580101940 2024-03-19T01:12:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.809+0000 7ff56f7fe640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff5680a28a0 0x7ff5680a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.809+0000 7ff57e59c640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff5680a28a0 0x7ff5680a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.809+0000 7ff56f7fe640 1 -- 172.21.15.116:0/109538767 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7ff5740cf000 con 0x7ff580101940 2024-03-19T01:12:32.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.810+0000 7ff57e59c640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff5680a28a0 0x7ff5680a4d60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff578000c00 tx=0x7ff578005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:32.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:32.814+0000 7ff56f7fe640 1 -- 172.21.15.116:0/109538767 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff57409c120 con 0x7ff580101940 2024-03-19T01:12:33.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.037+0000 7ff584bc4640 1 -- 172.21.15.116:0/109538767 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7ff58002d3a0 con 0x7ff580101940 2024-03-19T01:12:33.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.089+0000 7ff56f7fe640 1 -- 172.21.15.116:0/109538767 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v25) ==== 117+0+0 (secure 0 0 0) 0x7ff57409ba20 con 0x7ff580101940 2024-03-19T01:12:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.092+0000 7ff584bc4640 1 -- 172.21.15.116:0/109538767 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff5680a28a0 msgr2=0x7ff5680a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.092+0000 7ff584bc4640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff5680a28a0 0x7ff5680a4d60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff578000c00 tx=0x7ff578005fd0 comp rx=0 tx=0).stop 2024-03-19T01:12:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.092+0000 7ff584bc4640 1 -- 172.21.15.116:0/109538767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff580101940 msgr2=0x7ff580196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.092+0000 7ff584bc4640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff580101940 0x7ff580196ee0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff57400eb90 tx=0x7ff57400c6a0 comp rx=0 tx=0).stop 2024-03-19T01:12:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.092+0000 7ff584bc4640 1 -- 172.21.15.116:0/109538767 shutdown_connections 2024-03-19T01:12:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.092+0000 7ff584bc4640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff5680a28a0 0x7ff5680a4d60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.092+0000 7ff584bc4640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff580102300 0x7ff58019df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.092+0000 7ff584bc4640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff580101940 0x7ff580196ee0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.092+0000 7ff584bc4640 1 --2- 172.21.15.116:0/109538767 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff580100750 0x7ff5801969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.092+0000 7ff584bc4640 1 -- 172.21.15.116:0/109538767 >> 172.21.15.116:0/109538767 conn(0x7ff5800fbcc0 msgr2=0x7ff580104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.092+0000 7ff584bc4640 1 -- 172.21.15.116:0/109538767 shutdown_connections 2024-03-19T01:12:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:12:33.092+0000 7ff584bc4640 1 -- 172.21.15.116:0/109538767 wait complete. 2024-03-19T01:12:33.104 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-19T01:12:33.120 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-19T01:12:33.135 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T01:12:33.135 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-19T01:12:33.138 INFO:teuthology.run_tasks:Running task exec... 2024-03-19T01:12:33.149 INFO:teuthology.task.exec:Executing custom commands... 2024-03-19T01:12:33.150 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T01:12:33.150 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2024-03-19T01:12:33.192 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:33.209 INFO:teuthology.orchestra.run.smithi116.stdout:Filesystem Size Used Avail Use% Mounted on 2024-03-19T01:12:33.209 INFO:teuthology.orchestra.run.smithi116.stdout:0@35912c14-e58c-11ee-95c9-87774f69a715.cephfs=/volumes/qa/sv_1/26db1d39-573c-4af5-8f4a-46f4ed020a58 340G 0 340G 0% /tmp/cephfs 2024-03-19T01:12:33.211 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2024-03-19T01:12:33.308 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2024-03-19T01:12:33.345 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:33 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/109538767' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-19T01:12:33.352 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2024-03-19T01:12:33.398 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:33 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/109538767' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-19T01:12:33.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:33 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/109538767' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-19T01:12:34.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:34 smithi162 ceph-mon[67178]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:12:34.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:34 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/109538767' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-19T01:12:34.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:34 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:34.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:34 smithi116 ceph-mon[62701]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:12:34.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:34 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/109538767' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-19T01:12:34.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:34 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:34.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:34 smithi156 ceph-mon[66799]: pgmap v350: 97 pgs: 97 active+clean; 643 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-19T01:12:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:34 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/109538767' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-19T01:12:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:34 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:35.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2024-03-19T01:12:35.106 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2024-03-19T01:12:35.109 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2024-03-19T01:12:35.191 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-03-19T01:12:35.271 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/: 2024-03-19T01:12:35.271 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:35.272 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 . 2024-03-19T01:12:35.272 INFO:teuthology.orchestra.run.smithi116.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 .. 2024-03-19T01:12:35.272 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 data 2024-03-19T01:12:35.272 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:35.272 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data: 2024-03-19T01:12:35.272 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:35.272 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:35.272 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 .. 2024-03-19T01:12:35.274 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2024-03-19T01:12:35.364 INFO:teuthology.orchestra.run.smithi116.stdout:[Service] 2024-03-19T01:12:35.364 INFO:teuthology.orchestra.run.smithi116.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2024-03-19T01:12:35.365 INFO:teuthology.orchestra.run.smithi116.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2024-03-19T01:12:35.367 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2024-03-19T01:12:35.450 INFO:teuthology.orchestra.run.smithi116.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2024-03-19T01:12:36.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:36 smithi162 ceph-mon[67178]: pgmap v351: 97 pgs: 97 active+clean; 649 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-19T01:12:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:36 smithi116 ceph-mon[62701]: pgmap v351: 97 pgs: 97 active+clean; 649 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-19T01:12:36.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:36 smithi156 ceph-mon[66799]: pgmap v351: 97 pgs: 97 active+clean; 649 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-19T01:12:38.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:38 smithi162 ceph-mon[67178]: pgmap v352: 97 pgs: 97 active+clean; 649 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-19T01:12:38.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:38 smithi116 ceph-mon[62701]: pgmap v352: 97 pgs: 97 active+clean; 649 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-19T01:12:38.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:38 smithi156 ceph-mon[66799]: pgmap v352: 97 pgs: 97 active+clean; 649 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-03-19T01:12:39.712 INFO:teuthology.orchestra.run.smithi116.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2024-03-19T01:12:39.718 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-03-19T01:12:39.764 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/: 2024-03-19T01:12:39.764 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.764 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 . 2024-03-19T01:12:39.764 INFO:teuthology.orchestra.run.smithi116.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 .. 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 data 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data: 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout:total 35 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 . 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 .. 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 base 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Mar 19 01:12 global 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 log 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_commit_ts 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_dynshmem 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Mar 19 01:12 pg_hba.conf 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Mar 19 01:12 pg_ident.conf 2024-03-19T01:12:39.765 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 pg_logical 2024-03-19T01:12:39.766 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 19 01:12 pg_multixact 2024-03-19T01:12:39.766 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_notify 2024-03-19T01:12:39.766 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_replslot 2024-03-19T01:12:39.766 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_serial 2024-03-19T01:12:39.766 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_snapshots 2024-03-19T01:12:39.766 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_stat 2024-03-19T01:12:39.766 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_stat_tmp 2024-03-19T01:12:39.766 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 pg_subtrans 2024-03-19T01:12:39.766 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_tblspc 2024-03-19T01:12:39.766 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_twophase 2024-03-19T01:12:39.766 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 PG_VERSION 2024-03-19T01:12:39.766 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 19 01:12 pg_wal 2024-03-19T01:12:39.766 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 pg_xact 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Mar 19 01:12 postgresql.auto.conf 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Mar 19 01:12 postgresql.conf 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/base: 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 . 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Mar 19 01:12 1 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Mar 19 01:12 13493 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Mar 19 01:12 13494 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/base/1: 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout:total 7777 2024-03-19T01:12:39.767 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Mar 19 01:12 . 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 .. 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 112 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 113 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 19 01:12 1247 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1247_fsm 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1247_vm 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Mar 19 01:12 1249 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1249_fsm 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1249_vm 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Mar 19 01:12 1255 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1255_fsm 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1255_vm 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 19 01:12 1259 2024-03-19T01:12:39.768 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1259_fsm 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1259_vm 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Mar 19 01:12 13344 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13344_fsm 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13344_vm 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13346 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13348 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13349 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13349_fsm 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13349_vm 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13351 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13353 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13354 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13354_fsm 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13354_vm 2024-03-19T01:12:39.769 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13356 2024-03-19T01:12:39.770 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13358 2024-03-19T01:12:39.770 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13359 2024-03-19T01:12:39.770 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13359_fsm 2024-03-19T01:12:39.770 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13359_vm 2024-03-19T01:12:39.770 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13361 2024-03-19T01:12:39.770 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13363 2024-03-19T01:12:39.770 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 1417 2024-03-19T01:12:39.770 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 1418 2024-03-19T01:12:39.770 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 174 2024-03-19T01:12:39.770 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 175 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2187 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2224 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2328 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2336 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2337 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2579 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2600 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2600_fsm 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2600_vm 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2601 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2601_fsm 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2601_vm 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2602 2024-03-19T01:12:39.771 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2602_fsm 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2602_vm 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2603 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2603_fsm 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2603_vm 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2604 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2605 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2605_fsm 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2605_vm 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2606 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2606_fsm 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2606_vm 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2607 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2607_fsm 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2607_vm 2024-03-19T01:12:39.772 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Mar 19 01:12 2608 2024-03-19T01:12:39.773 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2608_fsm 2024-03-19T01:12:39.773 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2608_vm 2024-03-19T01:12:39.773 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Mar 19 01:12 2609 2024-03-19T01:12:39.773 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2609_fsm 2024-03-19T01:12:39.773 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2609_vm 2024-03-19T01:12:39.773 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2610 2024-03-19T01:12:39.773 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2610_fsm 2024-03-19T01:12:39.773 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2610_vm 2024-03-19T01:12:39.773 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2611 2024-03-19T01:12:39.773 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2612 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2612_fsm 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2612_vm 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2613 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2615 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2615_fsm 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2615_vm 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2616 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2616_fsm 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2616_vm 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Mar 19 01:12 2617 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2617_fsm 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2617_vm 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 19 01:12 2618 2024-03-19T01:12:39.774 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2618_fsm 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2618_vm 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Mar 19 01:12 2619 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2619_fsm 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2619_vm 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2620 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2650 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2651 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2652 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2653 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2654 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2655 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2656 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2657 2024-03-19T01:12:39.775 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 19 01:12 2658 2024-03-19T01:12:39.776 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 19 01:12 2659 2024-03-19T01:12:39.776 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2660 2024-03-19T01:12:39.776 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2661 2024-03-19T01:12:39.776 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2662 2024-03-19T01:12:39.776 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2663 2024-03-19T01:12:39.776 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2664 2024-03-19T01:12:39.776 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2665 2024-03-19T01:12:39.776 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2666 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2667 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2668 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2669 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2670 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Mar 19 01:12 2673 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Mar 19 01:12 2674 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Mar 19 01:12 2675 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2678 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2679 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2680 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2681 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2682 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2683 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2684 2024-03-19T01:12:39.777 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2685 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2686 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2687 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2688 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2689 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 19 01:12 2690 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Mar 19 01:12 2691 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2692 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2693 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2696 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2699 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2701 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2702 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2703 2024-03-19T01:12:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2704 2024-03-19T01:12:39.782 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2753 2024-03-19T01:12:39.782 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2753_fsm 2024-03-19T01:12:39.782 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2753_vm 2024-03-19T01:12:39.782 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2754 2024-03-19T01:12:39.782 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2755 2024-03-19T01:12:39.782 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2756 2024-03-19T01:12:39.783 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2757 2024-03-19T01:12:39.783 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2830 2024-03-19T01:12:39.783 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2831 2024-03-19T01:12:39.783 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2832 2024-03-19T01:12:39.783 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2833 2024-03-19T01:12:39.783 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2834 2024-03-19T01:12:39.783 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2835 2024-03-19T01:12:39.784 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2836 2024-03-19T01:12:39.784 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2837 2024-03-19T01:12:39.784 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Mar 19 01:12 2838 2024-03-19T01:12:39.784 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2838_fsm 2024-03-19T01:12:39.784 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2838_vm 2024-03-19T01:12:39.784 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2839 2024-03-19T01:12:39.784 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2840 2024-03-19T01:12:39.785 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2840_fsm 2024-03-19T01:12:39.785 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2840_vm 2024-03-19T01:12:39.785 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2841 2024-03-19T01:12:39.785 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2995 2024-03-19T01:12:39.785 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2996 2024-03-19T01:12:39.785 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3079 2024-03-19T01:12:39.785 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3079_fsm 2024-03-19T01:12:39.786 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3079_vm 2024-03-19T01:12:39.786 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3080 2024-03-19T01:12:39.786 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3081 2024-03-19T01:12:39.786 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 3085 2024-03-19T01:12:39.786 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3118 2024-03-19T01:12:39.786 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3119 2024-03-19T01:12:39.786 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 19 01:12 3164 2024-03-19T01:12:39.787 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3256 2024-03-19T01:12:39.787 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3257 2024-03-19T01:12:39.787 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3258 2024-03-19T01:12:39.787 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3350 2024-03-19T01:12:39.787 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3351 2024-03-19T01:12:39.787 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3379 2024-03-19T01:12:39.787 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3380 2024-03-19T01:12:39.788 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3381 2024-03-19T01:12:39.788 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3394 2024-03-19T01:12:39.788 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3394_fsm 2024-03-19T01:12:39.788 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3394_vm 2024-03-19T01:12:39.788 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3395 2024-03-19T01:12:39.788 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3429 2024-03-19T01:12:39.788 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3430 2024-03-19T01:12:39.789 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3431 2024-03-19T01:12:39.789 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3433 2024-03-19T01:12:39.789 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3439 2024-03-19T01:12:39.789 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3440 2024-03-19T01:12:39.789 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 3455 2024-03-19T01:12:39.789 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Mar 19 01:12 3456 2024-03-19T01:12:39.789 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3456_fsm 2024-03-19T01:12:39.790 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3456_vm 2024-03-19T01:12:39.790 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3466 2024-03-19T01:12:39.790 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3467 2024-03-19T01:12:39.790 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3468 2024-03-19T01:12:39.790 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3501 2024-03-19T01:12:39.790 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3502 2024-03-19T01:12:39.790 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3503 2024-03-19T01:12:39.791 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3534 2024-03-19T01:12:39.791 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3541 2024-03-19T01:12:39.791 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3541_fsm 2024-03-19T01:12:39.791 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3541_vm 2024-03-19T01:12:39.791 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3542 2024-03-19T01:12:39.791 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3574 2024-03-19T01:12:39.791 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3575 2024-03-19T01:12:39.791 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3576 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3596 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3597 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3598 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3599 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3600 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3600_fsm 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3600_vm 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3601 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3601_fsm 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3601_vm 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3602 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3602_fsm 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3602_vm 2024-03-19T01:12:39.792 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3603 2024-03-19T01:12:39.793 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3603_fsm 2024-03-19T01:12:39.793 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3603_vm 2024-03-19T01:12:39.793 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3604 2024-03-19T01:12:39.793 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3605 2024-03-19T01:12:39.793 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3606 2024-03-19T01:12:39.793 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3607 2024-03-19T01:12:39.793 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3608 2024-03-19T01:12:39.793 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 3609 2024-03-19T01:12:39.793 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3712 2024-03-19T01:12:39.793 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3764 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3764_fsm 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3764_vm 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3766 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3767 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3997 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4143 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4144 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4145 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4146 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4147 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4148 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4149 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4150 2024-03-19T01:12:39.794 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4151 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4152 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4153 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4154 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4155 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4156 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4157 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4158 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4159 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4160 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4161 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4162 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4163 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4164 2024-03-19T01:12:39.795 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4165 2024-03-19T01:12:39.799 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4166 2024-03-19T01:12:39.799 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4167 2024-03-19T01:12:39.799 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4168 2024-03-19T01:12:39.799 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4169 2024-03-19T01:12:39.799 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4170 2024-03-19T01:12:39.799 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4171 2024-03-19T01:12:39.799 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4172 2024-03-19T01:12:39.799 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4173 2024-03-19T01:12:39.799 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4174 2024-03-19T01:12:39.799 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 5002 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 548 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 549 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6102 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6104 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6106 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6110 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6111 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6112 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6113 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6117 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 826 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 827 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 828 2024-03-19T01:12:39.800 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 19 01:12 pg_filenode.map 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 PG_VERSION 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:total 7777 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Mar 19 01:12 . 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 .. 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 112 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 113 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 19 01:12 1247 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1247_fsm 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1247_vm 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Mar 19 01:12 1249 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1249_fsm 2024-03-19T01:12:39.801 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1249_vm 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Mar 19 01:12 1255 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1255_fsm 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1255_vm 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 19 01:12 1259 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1259_fsm 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1259_vm 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Mar 19 01:12 13344 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13344_fsm 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13344_vm 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13346 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13348 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13349 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13349_fsm 2024-03-19T01:12:39.802 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13349_vm 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13351 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13353 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13354 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13354_fsm 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13354_vm 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13356 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13358 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13359 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13359_fsm 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13359_vm 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13361 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13363 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 1417 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 1418 2024-03-19T01:12:39.803 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 174 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 175 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2187 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2224 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2328 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2336 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2337 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2579 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2600 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2600_fsm 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2600_vm 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2601 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2601_fsm 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2601_vm 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2602 2024-03-19T01:12:39.804 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2602_fsm 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2602_vm 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2603 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2603_fsm 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2603_vm 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2604 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2605 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2605_fsm 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2605_vm 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2606 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2606_fsm 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2606_vm 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2607 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2607_fsm 2024-03-19T01:12:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2607_vm 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Mar 19 01:12 2608 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2608_fsm 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2608_vm 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Mar 19 01:12 2609 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2609_fsm 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2609_vm 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2610 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2610_fsm 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2610_vm 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2611 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2612 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2612_fsm 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2612_vm 2024-03-19T01:12:39.806 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2613 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2615 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2615_fsm 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2615_vm 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2616 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2616_fsm 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2616_vm 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Mar 19 01:12 2617 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2617_fsm 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2617_vm 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 19 01:12 2618 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2618_fsm 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2618_vm 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Mar 19 01:12 2619 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2619_fsm 2024-03-19T01:12:39.807 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2619_vm 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2620 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2650 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2651 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2652 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2653 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2654 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2655 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2656 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2657 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 19 01:12 2658 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 19 01:12 2659 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2660 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2661 2024-03-19T01:12:39.808 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2662 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2663 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2664 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2665 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2666 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2667 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2668 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2669 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2670 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Mar 19 01:12 2673 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Mar 19 01:12 2674 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Mar 19 01:12 2675 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2678 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2679 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2680 2024-03-19T01:12:39.809 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2681 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2682 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2683 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2684 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2685 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2686 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2687 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2688 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2689 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 19 01:12 2690 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Mar 19 01:12 2691 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2692 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2693 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2696 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2699 2024-03-19T01:12:39.810 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2701 2024-03-19T01:12:39.811 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2702 2024-03-19T01:12:39.811 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2703 2024-03-19T01:12:39.811 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2704 2024-03-19T01:12:39.811 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2753 2024-03-19T01:12:39.811 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2753_fsm 2024-03-19T01:12:39.811 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2753_vm 2024-03-19T01:12:39.811 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2754 2024-03-19T01:12:39.811 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2755 2024-03-19T01:12:39.811 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2756 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2757 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2830 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2831 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2832 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2833 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2834 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2835 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2836 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2837 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Mar 19 01:12 2838 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2838_fsm 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2838_vm 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2839 2024-03-19T01:12:39.812 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2840 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2840_fsm 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2840_vm 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2841 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2995 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2996 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3079 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3079_fsm 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3079_vm 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3080 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3081 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 3085 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3118 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3119 2024-03-19T01:12:39.813 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 19 01:12 3164 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3256 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3257 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3258 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3350 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3351 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3379 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3380 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3381 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3394 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3394_fsm 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3394_vm 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3395 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3429 2024-03-19T01:12:39.814 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3430 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3431 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3433 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3439 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3440 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 3455 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Mar 19 01:12 3456 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3456_fsm 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3456_vm 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3466 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3467 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3468 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3501 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3502 2024-03-19T01:12:39.815 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3503 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3534 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3541 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3541_fsm 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3541_vm 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3542 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3574 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3575 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3576 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3596 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3597 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3598 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3599 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3600 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3600_fsm 2024-03-19T01:12:39.816 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3600_vm 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3601 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3601_fsm 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3601_vm 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3602 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3602_fsm 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3602_vm 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3603 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3603_fsm 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3603_vm 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3604 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3605 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3606 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3607 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3608 2024-03-19T01:12:39.817 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 3609 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3712 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3764 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3764_fsm 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3764_vm 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3766 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3767 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3997 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4143 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4144 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4145 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4146 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4147 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4148 2024-03-19T01:12:39.818 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4149 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4150 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4151 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4152 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4153 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4154 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4155 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4156 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4157 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4158 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4159 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4160 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4161 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4162 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4163 2024-03-19T01:12:39.819 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4164 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4165 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4166 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4167 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4168 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4169 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4170 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4171 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4172 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4173 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4174 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 5002 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 548 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 549 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6102 2024-03-19T01:12:39.820 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6104 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6106 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6110 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6111 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6112 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6113 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6117 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 826 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 827 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 828 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 19 01:12 pg_filenode.map 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 PG_VERSION 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-03-19T01:12:39.821 INFO:teuthology.orchestra.run.smithi116.stdout:total 7777 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Mar 19 01:12 . 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 .. 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 112 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 113 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 19 01:12 1247 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1247_fsm 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1247_vm 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Mar 19 01:12 1249 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1249_fsm 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1249_vm 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Mar 19 01:12 1255 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1255_fsm 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1255_vm 2024-03-19T01:12:39.822 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 19 01:12 1259 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1259_fsm 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1259_vm 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Mar 19 01:12 13344 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13344_fsm 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13344_vm 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13346 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13348 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13349 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13349_fsm 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13349_vm 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13351 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13353 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13354 2024-03-19T01:12:39.823 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13354_fsm 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13354_vm 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13356 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13358 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13359 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13359_fsm 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13359_vm 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13361 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13363 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 1417 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 1418 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 174 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 175 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2187 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2224 2024-03-19T01:12:39.824 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2328 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2336 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2337 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2579 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2600 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2600_fsm 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2600_vm 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2601 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2601_fsm 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2601_vm 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2602 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2602_fsm 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2602_vm 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2603 2024-03-19T01:12:39.825 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2603_fsm 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2603_vm 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2604 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2605 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2605_fsm 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2605_vm 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2606 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2606_fsm 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2606_vm 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2607 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2607_fsm 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2607_vm 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Mar 19 01:12 2608 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2608_fsm 2024-03-19T01:12:39.826 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2608_vm 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Mar 19 01:12 2609 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2609_fsm 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2609_vm 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2610 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2610_fsm 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2610_vm 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2611 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2612 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2612_fsm 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2612_vm 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2613 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2615 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2615_fsm 2024-03-19T01:12:39.827 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2615_vm 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2616 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2616_fsm 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2616_vm 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Mar 19 01:12 2617 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2617_fsm 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2617_vm 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 19 01:12 2618 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2618_fsm 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2618_vm 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Mar 19 01:12 2619 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2619_fsm 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2619_vm 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2620 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2650 2024-03-19T01:12:39.828 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2651 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2652 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2653 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2654 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2655 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2656 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2657 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 19 01:12 2658 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 19 01:12 2659 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2660 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2661 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2662 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2663 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2664 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2665 2024-03-19T01:12:39.829 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2666 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2667 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2668 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2669 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2670 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Mar 19 01:12 2673 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Mar 19 01:12 2674 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Mar 19 01:12 2675 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2678 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2679 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2680 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2681 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2682 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2683 2024-03-19T01:12:39.830 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2684 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2685 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2686 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2687 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2688 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2689 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 19 01:12 2690 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Mar 19 01:12 2691 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2692 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2693 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2696 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2699 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2701 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2702 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2703 2024-03-19T01:12:39.831 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2704 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2753 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2753_fsm 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2753_vm 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2754 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2755 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2756 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2757 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2830 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2831 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2832 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2833 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2834 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2835 2024-03-19T01:12:39.832 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2836 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2837 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Mar 19 01:12 2838 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2838_fsm 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2838_vm 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2839 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2840 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2840_fsm 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2840_vm 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2841 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2995 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2996 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3079 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3079_fsm 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3079_vm 2024-03-19T01:12:39.833 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3080 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3081 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 3085 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3118 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3119 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 19 01:12 3164 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3256 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3257 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3258 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3350 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3351 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3379 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3380 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3381 2024-03-19T01:12:39.834 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3394 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3394_fsm 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3394_vm 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3395 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3429 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3430 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3431 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3433 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3439 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3440 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 3455 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Mar 19 01:12 3456 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3456_fsm 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3456_vm 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3466 2024-03-19T01:12:39.835 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3467 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3468 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3501 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3502 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3503 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3534 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3541 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3541_fsm 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3541_vm 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3542 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3574 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3575 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3576 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3596 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3597 2024-03-19T01:12:39.836 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3598 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3599 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3600 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3600_fsm 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3600_vm 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3601 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3601_fsm 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3601_vm 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3602 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3602_fsm 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3602_vm 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3603 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3603_fsm 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3603_vm 2024-03-19T01:12:39.837 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3604 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3605 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3606 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3607 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3608 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 3609 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3712 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3764 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3764_fsm 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3764_vm 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3766 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3767 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3997 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4143 2024-03-19T01:12:39.838 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4144 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4145 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4146 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4147 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4148 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4149 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4150 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4151 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4152 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4153 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4154 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4155 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4156 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4157 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4158 2024-03-19T01:12:39.839 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4159 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4160 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4161 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4162 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4163 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4164 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4165 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4166 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4167 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4168 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4169 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4170 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4171 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4172 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4173 2024-03-19T01:12:39.840 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4174 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 5002 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 548 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 549 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6102 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6104 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6106 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6110 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6111 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6112 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6113 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6117 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 826 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 827 2024-03-19T01:12:39.841 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 828 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 19 01:12 pg_filenode.map 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 PG_VERSION 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/global: 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout:total 537 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 57 Mar 19 01:12 . 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1213 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1213_fsm 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1213_vm 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1214 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1214_fsm 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1214_vm 2024-03-19T01:12:39.842 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 1232 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 1233 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1260 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1260_fsm 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1260_vm 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1261 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1261_fsm 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1261_vm 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1262 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1262_fsm 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1262_vm 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2396 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2396_fsm 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2396_vm 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2397 2024-03-19T01:12:39.843 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2671 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2672 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2676 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2677 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2694 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2695 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2697 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2698 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2846 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2847 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2964 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2965 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2966 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2967 2024-03-19T01:12:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3592 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3593 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4060 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4061 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4175 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4176 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4177 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4178 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4181 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4182 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4183 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4184 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4185 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4186 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6000 2024-03-19T01:12:39.845 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6001 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6002 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6100 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6114 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6115 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 pg_control 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 19 01:12 pg_filenode.map 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/log: 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-03-19T01:12:39.846 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout:total 1 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 . 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 mappings 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Mar 19 01:12 replorigin_checkpoint 2024-03-19T01:12:39.847 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 snapshots 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 .. 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 .. 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.848 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 19 01:12 . 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 members 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 offsets 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout:total 8 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 . 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 19 01:12 .. 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 0000 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout:total 8 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 . 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 19 01:12 .. 2024-03-19T01:12:39.849 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 0000 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.850 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.851 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout:total 8 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 . 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 0000 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-03-19T01:12:39.852 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout:total 16384 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 19 01:12 . 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:12 000000010000000000000001 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 archive_status 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:12:39.853 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 19 01:12 .. 2024-03-19T01:12:39.854 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:12:39.854 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-03-19T01:12:39.854 INFO:teuthology.orchestra.run.smithi116.stdout:total 8 2024-03-19T01:12:39.854 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 . 2024-03-19T01:12:39.854 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 19 01:12 .. 2024-03-19T01:12:39.854 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 0000 2024-03-19T01:12:39.854 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2024-03-19T01:12:39.985 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 32 -i' 2024-03-19T01:12:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:dropping old tables... 2024-03-19T01:12:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2024-03-19T01:12:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2024-03-19T01:12:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2024-03-19T01:12:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2024-03-19T01:12:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:creating tables... 2024-03-19T01:12:40.048 INFO:teuthology.orchestra.run.smithi116.stderr:generating data (client-side)... 2024-03-19T01:12:40.126 INFO:teuthology.orchestra.run.smithi116.stderr:100000 of 3200000 tuples (3%) done (elapsed 0.06 s, remaining 1.90 s) 2024-03-19T01:12:40.292 INFO:teuthology.orchestra.run.smithi116.stderr:200000 of 3200000 tuples (6%) done (elapsed 0.23 s, remaining 3.42 s) 2024-03-19T01:12:40.381 INFO:teuthology.orchestra.run.smithi116.stderr:300000 of 3200000 tuples (9%) done (elapsed 0.32 s, remaining 3.06 s) 2024-03-19T01:12:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:40 smithi162 ceph-mon[67178]: pgmap v353: 97 pgs: 97 active+clean; 4.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 314 KiB/s wr, 21 op/s 2024-03-19T01:12:40.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:40 smithi116 ceph-mon[62701]: pgmap v353: 97 pgs: 97 active+clean; 4.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 314 KiB/s wr, 21 op/s 2024-03-19T01:12:40.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:40 smithi156 ceph-mon[66799]: pgmap v353: 97 pgs: 97 active+clean; 4.2 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 314 KiB/s wr, 21 op/s 2024-03-19T01:12:40.521 INFO:teuthology.orchestra.run.smithi116.stderr:400000 of 3200000 tuples (12%) done (elapsed 0.46 s, remaining 3.19 s) 2024-03-19T01:12:40.663 INFO:teuthology.orchestra.run.smithi116.stderr:500000 of 3200000 tuples (15%) done (elapsed 0.60 s, remaining 3.23 s) 2024-03-19T01:12:40.742 INFO:teuthology.orchestra.run.smithi116.stderr:600000 of 3200000 tuples (18%) done (elapsed 0.68 s, remaining 2.94 s) 2024-03-19T01:12:40.887 INFO:teuthology.orchestra.run.smithi116.stderr:700000 of 3200000 tuples (21%) done (elapsed 0.82 s, remaining 2.94 s) 2024-03-19T01:12:41.060 INFO:teuthology.orchestra.run.smithi116.stderr:800000 of 3200000 tuples (25%) done (elapsed 1.00 s, remaining 2.99 s) 2024-03-19T01:12:41.165 INFO:teuthology.orchestra.run.smithi116.stderr:900000 of 3200000 tuples (28%) done (elapsed 1.10 s, remaining 2.81 s) 2024-03-19T01:12:41.300 INFO:teuthology.orchestra.run.smithi116.stderr:1000000 of 3200000 tuples (31%) done (elapsed 1.24 s, remaining 2.72 s) 2024-03-19T01:12:41.470 INFO:teuthology.orchestra.run.smithi116.stderr:1100000 of 3200000 tuples (34%) done (elapsed 1.41 s, remaining 2.68 s) 2024-03-19T01:12:41.577 INFO:teuthology.orchestra.run.smithi116.stderr:1200000 of 3200000 tuples (37%) done (elapsed 1.51 s, remaining 2.52 s) 2024-03-19T01:12:41.739 INFO:teuthology.orchestra.run.smithi116.stderr:1300000 of 3200000 tuples (40%) done (elapsed 1.67 s, remaining 2.45 s) 2024-03-19T01:12:41.911 INFO:teuthology.orchestra.run.smithi116.stderr:1400000 of 3200000 tuples (43%) done (elapsed 1.85 s, remaining 2.37 s) 2024-03-19T01:12:42.000 INFO:teuthology.orchestra.run.smithi116.stderr:1500000 of 3200000 tuples (46%) done (elapsed 1.94 s, remaining 2.19 s) 2024-03-19T01:12:42.141 INFO:teuthology.orchestra.run.smithi116.stderr:1600000 of 3200000 tuples (50%) done (elapsed 2.08 s, remaining 2.08 s) 2024-03-19T01:12:42.240 INFO:teuthology.orchestra.run.smithi116.stderr:1700000 of 3200000 tuples (53%) done (elapsed 2.18 s, remaining 1.92 s) 2024-03-19T01:12:42.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:42 smithi162 ceph-mon[67178]: pgmap v354: 97 pgs: 97 active+clean; 17 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.4 MiB/s wr, 42 op/s 2024-03-19T01:12:42.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:42 smithi116 ceph-mon[62701]: pgmap v354: 97 pgs: 97 active+clean; 17 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.4 MiB/s wr, 42 op/s 2024-03-19T01:12:42.427 INFO:teuthology.orchestra.run.smithi116.stderr:1800000 of 3200000 tuples (56%) done (elapsed 2.36 s, remaining 1.84 s) 2024-03-19T01:12:42.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:42 smithi156 ceph-mon[66799]: pgmap v354: 97 pgs: 97 active+clean; 17 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 1.4 MiB/s wr, 42 op/s 2024-03-19T01:12:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:1900000 of 3200000 tuples (59%) done (elapsed 2.51 s, remaining 1.72 s) 2024-03-19T01:12:42.674 INFO:teuthology.orchestra.run.smithi116.stderr:2000000 of 3200000 tuples (62%) done (elapsed 2.61 s, remaining 1.57 s) 2024-03-19T01:12:42.831 INFO:teuthology.orchestra.run.smithi116.stderr:2100000 of 3200000 tuples (65%) done (elapsed 2.77 s, remaining 1.45 s) 2024-03-19T01:12:43.034 INFO:teuthology.orchestra.run.smithi116.stderr:2200000 of 3200000 tuples (68%) done (elapsed 2.97 s, remaining 1.35 s) 2024-03-19T01:12:43.123 INFO:teuthology.orchestra.run.smithi116.stderr:2300000 of 3200000 tuples (71%) done (elapsed 3.06 s, remaining 1.20 s) 2024-03-19T01:12:43.322 INFO:teuthology.orchestra.run.smithi116.stderr:2400000 of 3200000 tuples (75%) done (elapsed 3.26 s, remaining 1.09 s) 2024-03-19T01:12:43.458 INFO:teuthology.orchestra.run.smithi116.stderr:2500000 of 3200000 tuples (78%) done (elapsed 3.39 s, remaining 0.95 s) 2024-03-19T01:12:43.556 INFO:teuthology.orchestra.run.smithi116.stderr:2600000 of 3200000 tuples (81%) done (elapsed 3.49 s, remaining 0.81 s) 2024-03-19T01:12:43.708 INFO:teuthology.orchestra.run.smithi116.stderr:2700000 of 3200000 tuples (84%) done (elapsed 3.64 s, remaining 0.67 s) 2024-03-19T01:12:43.818 INFO:teuthology.orchestra.run.smithi116.stderr:2800000 of 3200000 tuples (87%) done (elapsed 3.75 s, remaining 0.54 s) 2024-03-19T01:12:43.977 INFO:teuthology.orchestra.run.smithi116.stderr:2900000 of 3200000 tuples (90%) done (elapsed 3.91 s, remaining 0.40 s) 2024-03-19T01:12:44.124 INFO:teuthology.orchestra.run.smithi116.stderr:3000000 of 3200000 tuples (93%) done (elapsed 4.06 s, remaining 0.27 s) 2024-03-19T01:12:44.218 INFO:teuthology.orchestra.run.smithi116.stderr:3100000 of 3200000 tuples (96%) done (elapsed 4.15 s, remaining 0.13 s) 2024-03-19T01:12:44.352 INFO:teuthology.orchestra.run.smithi116.stderr:3200000 of 3200000 tuples (100%) done (elapsed 4.29 s, remaining 0.00 s) 2024-03-19T01:12:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:44 smithi162 ceph-mon[67178]: pgmap v355: 97 pgs: 97 active+clean; 45 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 5.4 MiB/s wr, 57 op/s 2024-03-19T01:12:44.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:44 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:44.399 INFO:teuthology.orchestra.run.smithi116.stderr:vacuuming... 2024-03-19T01:12:44.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:44 smithi116 ceph-mon[62701]: pgmap v355: 97 pgs: 97 active+clean; 45 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 5.4 MiB/s wr, 57 op/s 2024-03-19T01:12:44.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:44 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:44.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:44 smithi156 ceph-mon[66799]: pgmap v355: 97 pgs: 97 active+clean; 45 MiB data, 398 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 5.4 MiB/s wr, 57 op/s 2024-03-19T01:12:44.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:44 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:44.990 INFO:teuthology.orchestra.run.smithi116.stderr:creating primary keys... 2024-03-19T01:12:46.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.312+0000 7feb1ea85640 1 -- 172.21.15.116:0/1630012639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb18100130 msgr2=0x7feb18100550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.312+0000 7feb1ea85640 1 --2- 172.21.15.116:0/1630012639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb18100130 0x7feb18100550 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009900 tx=0x7feb0c02bcd0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.312+0000 7feb1ea85640 1 -- 172.21.15.116:0/1630012639 shutdown_connections 2024-03-19T01:12:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.312+0000 7feb1ea85640 1 --2- 172.21.15.116:0/1630012639 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb18101430 0x7feb181018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.312+0000 7feb1ea85640 1 --2- 172.21.15.116:0/1630012639 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb18100a90 0x7feb18100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.312+0000 7feb1ea85640 1 --2- 172.21.15.116:0/1630012639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb18100130 0x7feb18100550 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.312+0000 7feb1ea85640 1 -- 172.21.15.116:0/1630012639 >> 172.21.15.116:0/1630012639 conn(0x7feb180fbcb0 msgr2=0x7feb180fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.313+0000 7feb1ea85640 1 -- 172.21.15.116:0/1630012639 shutdown_connections 2024-03-19T01:12:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.313+0000 7feb1ea85640 1 -- 172.21.15.116:0/1630012639 wait complete. 2024-03-19T01:12:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.313+0000 7feb1ea85640 1 Processor -- start 2024-03-19T01:12:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.313+0000 7feb1ea85640 1 -- start start 2024-03-19T01:12:46.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.313+0000 7feb1ea85640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb18100130 0x7feb18199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.313+0000 7feb1ea85640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb18100a90 0x7feb1819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.313+0000 7feb1ea85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb18101430 0x7feb181a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.313+0000 7feb1ea85640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feb1810f6c0 con 0x7feb18101430 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1ea85640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7feb1810f540 con 0x7feb18100a90 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1ea85640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb1810f840 con 0x7feb18100130 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1d022640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb18101430 0x7feb181a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1d022640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb18101430 0x7feb181a1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42132/0 (socket says 172.21.15.116:42132) 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1d022640 1 -- 172.21.15.116:0/1540858529 learned_addr learned my addr 172.21.15.116:0/1540858529 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb17fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb18100a90 0x7feb1819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1c821640 1 --2- 172.21.15.116:0/1540858529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb18100130 0x7feb18199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1d022640 1 -- 172.21.15.116:0/1540858529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb18100130 msgr2=0x7feb18199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1d022640 1 --2- 172.21.15.116:0/1540858529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb18100130 0x7feb18199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1d022640 1 -- 172.21.15.116:0/1540858529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb18100a90 msgr2=0x7feb1819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:46.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1d022640 1 --2- 172.21.15.116:0/1540858529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb18100a90 0x7feb1819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1d022640 1 -- 172.21.15.116:0/1540858529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb181a1910 con 0x7feb18101430 2024-03-19T01:12:46.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1d022640 1 --2- 172.21.15.116:0/1540858529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb18101430 0x7feb181a1270 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7feb1000eb10 tx=0x7feb1000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:46.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb15ffb640 1 -- 172.21.15.116:0/1540858529 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb10004b20 con 0x7feb18101430 2024-03-19T01:12:46.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb15ffb640 1 -- 172.21.15.116:0/1540858529 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb10004cc0 con 0x7feb18101430 2024-03-19T01:12:46.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb15ffb640 1 -- 172.21.15.116:0/1540858529 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb10010760 con 0x7feb18101430 2024-03-19T01:12:46.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1ea85640 1 -- 172.21.15.116:0/1540858529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb181a1c00 con 0x7feb18101430 2024-03-19T01:12:46.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.314+0000 7feb1ea85640 1 -- 172.21.15.116:0/1540858529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb181a2150 con 0x7feb18101430 2024-03-19T01:12:46.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb15ffb640 1 -- 172.21.15.116:0/1540858529 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7feb10005020 con 0x7feb18101430 2024-03-19T01:12:46.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb21674740 1 Processor -- start 2024-03-19T01:12:46.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb21674740 1 -- start start 2024-03-19T01:12:46.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb21674740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff923910 0x5629ff923d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb21674740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff924300 0x5629ff928e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb21674740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff92b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb21674740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5629ff87fbe0 con 0x5629ff929470 2024-03-19T01:12:46.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb21674740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5629ff828960 con 0x5629ff923910 2024-03-19T01:12:46.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb21674740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5629ff87bc80 con 0x5629ff924300 2024-03-19T01:12:46.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb17fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff923910 0x5629ff923d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb1c821640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff924300 0x5629ff928e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb17fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff923910 0x5629ff923d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:35280/0 (socket says 172.21.15.116:35280) 2024-03-19T01:12:46.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb1c821640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff924300 0x5629ff928e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52352/0 (socket says 172.21.15.116:52352) 2024-03-19T01:12:46.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb17fff640 1 -- 172.21.15.116:0/3214385638 learned_addr learned my addr 172.21.15.116:0/3214385638 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:46.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb17fff640 1 --2- 172.21.15.116:0/3214385638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff92b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb1c821640 1 -- 172.21.15.116:0/3214385638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff923910 msgr2=0x5629ff923d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:46.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb1c821640 1 --2- 172.21.15.116:0/3214385638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff923910 0x5629ff923d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb1c821640 1 -- 172.21.15.116:0/3214385638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 msgr2=0x5629ff92b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:46.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb1c821640 1 --2- 172.21.15.116:0/3214385638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff92b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb15ffb640 1 --2- 172.21.15.116:0/1540858529 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7feb080a2920 0x7feb080a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.318+0000 7feb1c821640 1 -- 172.21.15.116:0/3214385638 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629ff81fd20 con 0x5629ff924300 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.319+0000 7feb15ffb640 1 -- 172.21.15.116:0/1540858529 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7feb100ceff0 con 0x7feb18101430 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.319+0000 7feb1c821640 1 --2- 172.21.15.116:0/3214385638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff924300 0x5629ff928e00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009230 tx=0x7feb0c02c6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8542627a7d977b25 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.319+0000 7feb17fff640 1 --2- 172.21.15.116:0/1540858529 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7feb080a2920 0x7feb080a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.319+0000 7feb02ffd640 1 -- 172.21.15.116:0/3214385638 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c02cc20 con 0x5629ff924300 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.319+0000 7feb02ffd640 1 -- 172.21.15.116:0/3214385638 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0c005040 con 0x5629ff924300 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.319+0000 7feb02ffd640 1 -- 172.21.15.116:0/3214385638 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c0048c0 con 0x5629ff924300 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.320+0000 7feb21674740 1 -- 172.21.15.116:0/3214385638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff924300 msgr2=0x5629ff928e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.320+0000 7feb17fff640 1 --2- 172.21.15.116:0/1540858529 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7feb080a2920 0x7feb080a4de0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x5629ff924b60 tx=0x7feb04013040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.320+0000 7feb21674740 1 --2- 172.21.15.116:0/3214385638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff924300 0x5629ff928e00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009230 tx=0x7feb0c02c6f0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.320+0000 7feb21674740 1 -- 172.21.15.116:0/3214385638 shutdown_connections 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.320+0000 7feb21674740 1 --2- 172.21.15.116:0/3214385638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff92b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.320+0000 7feb21674740 1 --2- 172.21.15.116:0/3214385638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff924300 0x5629ff928e00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.320+0000 7feb21674740 1 --2- 172.21.15.116:0/3214385638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff923910 0x5629ff923d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.320+0000 7feb21674740 1 -- 172.21.15.116:0/3214385638 >> 172.21.15.116:0/3214385638 conn(0x5629ff916670 msgr2=0x5629ff916a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.320+0000 7feb21674740 1 -- 172.21.15.116:0/3214385638 shutdown_connections 2024-03-19T01:12:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.320+0000 7feb21674740 1 -- 172.21.15.116:0/3214385638 wait complete. 2024-03-19T01:12:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb21674740 1 -- start start 2024-03-19T01:12:46.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb21674740 1 Processor -- start 2024-03-19T01:12:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb21674740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff923910 0x5629ff92d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb21674740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff924300 0x5629ff92de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb21674740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff934df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb1c821640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff923910 0x5629ff92d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb1c821640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff923910 0x5629ff92d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52368/0 (socket says 172.21.15.116:52368) 2024-03-19T01:12:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb21674740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5629ff92c3c0 con 0x5629ff929470 2024-03-19T01:12:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb1c821640 1 -- 172.21.15.116:0/319689015 learned_addr learned my addr 172.21.15.116:0/319689015 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb21674740 1 -- 172.21.15.116:0/319689015 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5629ff83bdb0 con 0x5629ff924300 2024-03-19T01:12:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb21674740 1 -- 172.21.15.116:0/319689015 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5629ff92c540 con 0x5629ff923910 2024-03-19T01:12:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb17fff640 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff924300 0x5629ff92de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb21674740 10 client.0 _ll_register_callbacks cb 0x5629ff840e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.321+0000 7feb1d022640 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff934df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feb1c821640 1 -- 172.21.15.116:0/319689015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff924300 msgr2=0x5629ff92de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feb1c821640 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff924300 0x5629ff92de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feb1c821640 1 -- 172.21.15.116:0/319689015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 msgr2=0x5629ff934df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feb1c821640 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff934df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feb1c821640 1 -- 172.21.15.116:0/319689015 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629ff83ac60 con 0x5629ff923910 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feb17fff640 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff924300 0x5629ff92de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feb1d022640 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff934df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feb1c821640 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff923910 0x5629ff92d8e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0096c0 tx=0x7feb0c005420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d68134561f248f60 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feaf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feaf7fff640 1 -- 172.21.15.116:0/319689015 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c04c070 con 0x5629ff923910 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feb21674740 1 -- 172.21.15.116:0/319689015 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb0c005040 con 0x5629ff923910 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feaf7fff640 1 -- 172.21.15.116:0/319689015 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7feb0c004ad0 con 0x5629ff923910 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feaf7fff640 1 -- 172.21.15.116:0/319689015 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7feb0c005610 con 0x5629ff923910 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feb21674740 10 client.25171 fetch_fsmap learned FSMap version 26 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.322+0000 7feb21674740 1 -- 172.21.15.116:0/319689015 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5629ff594210 con 0x5629ff923910 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.323+0000 7feaf7fff640 1 -- 172.21.15.116:0/319689015 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb0c005040 con 0x5629ff923910 2024-03-19T01:12:46.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.323+0000 7feaf7fff640 1 -- 172.21.15.116:0/319689015 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7feb0c034440 con 0x5629ff923910 2024-03-19T01:12:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.323+0000 7feb21674740 10 client.25171 fetch_fsmap finished waiting for FSMap version 26 2024-03-19T01:12:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.323+0000 7feb21674740 10 client.25171 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:12:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.323+0000 7feb21674740 20 client.25171 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.323+0000 7feb21674740 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5629ff9392e0 0x5629ff93b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.323+0000 7feb21674740 4 client.25171 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:12:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.323+0000 7feb21674740 1 -- 172.21.15.116:0/319689015 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629ff88d120 con 0x5629ff9392e0 2024-03-19T01:12:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.323+0000 7feb17fff640 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5629ff9392e0 0x5629ff93b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.324+0000 7feb17fff640 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5629ff9392e0 0x5629ff93b6d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb413e5483c40f31 server_cookie=ad10acb38f348519 in_seq=0 out_seq=0 2024-03-19T01:12:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.324+0000 7feaf7fff640 10 client.25171 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feaf7fff640 1 -- 172.21.15.116:0/319689015 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5629ff88d120 con 0x5629ff9392e0 2024-03-19T01:12:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feaf7fff640 10 client.25171 handle_command_reply: tid=0 2024-03-19T01:12:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feb21674740 1 client.25171 shutdown 2024-03-19T01:12:46.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feb21674740 1 -- 172.21.15.116:0/319689015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff923910 msgr2=0x5629ff92d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feb21674740 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff923910 0x5629ff92d8e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feb0c0096c0 tx=0x7feb0c005420 comp rx=0 tx=0).stop 2024-03-19T01:12:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feb21674740 1 -- 172.21.15.116:0/319689015 shutdown_connections 2024-03-19T01:12:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feb21674740 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff934df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feb21674740 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff924300 0x5629ff92de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feb21674740 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5629ff9392e0 0x5629ff93b6d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feb21674740 1 --2- 172.21.15.116:0/319689015 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff923910 0x5629ff92d8e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feb21674740 1 -- 172.21.15.116:0/319689015 >> 172.21.15.116:0/319689015 conn(0x5629ff916670 msgr2=0x5629ff9178c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feaf7fff640 0 client.25171 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feb21674740 1 -- 172.21.15.116:0/319689015 shutdown_connections 2024-03-19T01:12:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feb21674740 1 -- 172.21.15.116:0/319689015 wait complete. 2024-03-19T01:12:46.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.325+0000 7feb21674740 20 client.25171 trim_cache size 0 max 16384 2024-03-19T01:12:46.347 INFO:teuthology.orchestra.run.smithi116.stderr:done in 6.31 s (drop tables 0.00 s, create tables 0.01 s, client-side generate 4.35 s, vacuum 0.59 s, primary keys 1.36 s). 2024-03-19T01:12:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.349+0000 7feb21674740 1 Processor -- start 2024-03-19T01:12:46.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.349+0000 7feb21674740 1 -- start start 2024-03-19T01:12:46.351 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2024-03-19T01:12:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.349+0000 7feb21674740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff923910 0x5629ff924d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.349+0000 7feb21674740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff924300 0x5629ff934db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.349+0000 7feb21674740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff9252b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.349+0000 7feb21674740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feb181a6a50 con 0x5629ff929470 2024-03-19T01:12:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.349+0000 7feb21674740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7feb181a85c0 con 0x5629ff924300 2024-03-19T01:12:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.349+0000 7feb21674740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb181a7120 con 0x5629ff923910 2024-03-19T01:12:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.349+0000 7feb1d022640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff9252b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.349+0000 7feb1d022640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff9252b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42158/0 (socket says 172.21.15.116:42158) 2024-03-19T01:12:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.349+0000 7feb1d022640 1 -- 172.21.15.116:0/865996832 learned_addr learned my addr 172.21.15.116:0/865996832 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:46.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.350+0000 7feb17fff640 1 --2- 172.21.15.116:0/865996832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff924300 0x5629ff934db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.350+0000 7feb1c821640 1 --2- 172.21.15.116:0/865996832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff923910 0x5629ff924d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.350+0000 7feb1d022640 1 -- 172.21.15.116:0/865996832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff923910 msgr2=0x5629ff924d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.350+0000 7feb1d022640 1 --2- 172.21.15.116:0/865996832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff923910 0x5629ff924d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.350+0000 7feb1d022640 1 -- 172.21.15.116:0/865996832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff924300 msgr2=0x5629ff934db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.350+0000 7feb1d022640 1 --2- 172.21.15.116:0/865996832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff924300 0x5629ff934db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.350+0000 7feb1d022640 1 -- 172.21.15.116:0/865996832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629ff660620 con 0x5629ff929470 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.350+0000 7feb1d022640 1 --2- 172.21.15.116:0/865996832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff9252b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feb100049d0 tx=0x7feb100058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f044581decdc9b43 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.350+0000 7feb027fc640 1 -- 172.21.15.116:0/865996832 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb100b3070 con 0x5629ff929470 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.350+0000 7feb027fc640 1 -- 172.21.15.116:0/865996832 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb1009c070 con 0x5629ff929470 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.350+0000 7feb027fc640 1 -- 172.21.15.116:0/865996832 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb100a3ee0 con 0x5629ff929470 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.351+0000 7feb21674740 1 -- 172.21.15.116:0/865996832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 msgr2=0x5629ff9252b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.351+0000 7feb21674740 1 --2- 172.21.15.116:0/865996832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff9252b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7feb100049d0 tx=0x7feb100058d0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.351+0000 7feb21674740 1 -- 172.21.15.116:0/865996832 shutdown_connections 2024-03-19T01:12:46.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.351+0000 7feb21674740 1 --2- 172.21.15.116:0/865996832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff929470 0x5629ff9252b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.351+0000 7feb21674740 1 --2- 172.21.15.116:0/865996832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff924300 0x5629ff934db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.351+0000 7feb21674740 1 --2- 172.21.15.116:0/865996832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff923910 0x5629ff924d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.351+0000 7feb21674740 1 -- 172.21.15.116:0/865996832 >> 172.21.15.116:0/865996832 conn(0x5629ff916670 msgr2=0x5629ff915220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:46.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.351+0000 7feb21674740 1 -- 172.21.15.116:0/865996832 shutdown_connections 2024-03-19T01:12:46.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.351+0000 7feb21674740 1 -- 172.21.15.116:0/865996832 wait complete. 2024-03-19T01:12:46.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.351+0000 7feb21674740 1 -- start start 2024-03-19T01:12:46.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.351+0000 7feb21674740 1 Processor -- start 2024-03-19T01:12:46.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb21674740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff923910 0x5629ff934cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb21674740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff924300 0x5629ff931760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb21674740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff929470 0x5629ff931ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff924300 0x5629ff931760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb1c821640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff923910 0x5629ff934cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff924300 0x5629ff931760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42168/0 (socket says 172.21.15.116:42168) 2024-03-19T01:12:46.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb17fff640 1 -- 172.21.15.116:0/3237616663 learned_addr learned my addr 172.21.15.116:0/3237616663 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:46.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb21674740 1 -- 172.21.15.116:0/3237616663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feb181a7ec0 con 0x5629ff924300 2024-03-19T01:12:46.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb21674740 1 -- 172.21.15.116:0/3237616663 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7feb181a77f0 con 0x5629ff923910 2024-03-19T01:12:46.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb21674740 1 -- 172.21.15.116:0/3237616663 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7feb181a4a30 con 0x5629ff929470 2024-03-19T01:12:46.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb1d022640 1 --2- 172.21.15.116:0/3237616663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff929470 0x5629ff931ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb21674740 10 client.0 _ll_register_callbacks cb 0x7feb040098b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb17fff640 1 -- 172.21.15.116:0/3237616663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff929470 msgr2=0x5629ff931ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb17fff640 1 --2- 172.21.15.116:0/3237616663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff929470 0x5629ff931ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb17fff640 1 -- 172.21.15.116:0/3237616663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff923910 msgr2=0x5629ff934cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb17fff640 1 --2- 172.21.15.116:0/3237616663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff923910 0x5629ff934cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb17fff640 1 -- 172.21.15.116:0/3237616663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629ff99d5e0 con 0x5629ff924300 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb17fff640 1 --2- 172.21.15.116:0/3237616663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff924300 0x5629ff931760 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7feb04010fd0 tx=0x7feb040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64baa98b22b8d2bf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feaf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feaf7fff640 1 -- 172.21.15.116:0/3237616663 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0402be60 con 0x5629ff924300 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feaf7fff640 1 -- 172.21.15.116:0/3237616663 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7feb0401a070 con 0x5629ff924300 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb21674740 1 -- 172.21.15.116:0/3237616663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feb1009c070 con 0x5629ff924300 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feaf7fff640 1 -- 172.21.15.116:0/3237616663 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7feb04039050 con 0x5629ff924300 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.352+0000 7feb21674740 10 client.15360 fetch_fsmap learned FSMap version 26 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.353+0000 7feb21674740 1 -- 172.21.15.116:0/3237616663 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5629ff99e960 con 0x5629ff924300 2024-03-19T01:12:46.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.353+0000 7feaf7fff640 1 -- 172.21.15.116:0/3237616663 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb1009c070 con 0x5629ff924300 2024-03-19T01:12:46.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.353+0000 7feaf7fff640 1 -- 172.21.15.116:0/3237616663 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 26) ==== 8358+0+0 (secure 0 0 0) 0x7feb0402b4d0 con 0x5629ff924300 2024-03-19T01:12:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.353+0000 7feb21674740 10 client.15360 fetch_fsmap finished waiting for FSMap version 26 2024-03-19T01:12:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.353+0000 7feb21674740 10 client.15360 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:12:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.353+0000 7feb21674740 20 client.15360 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.353+0000 7feb21674740 1 --2- 172.21.15.116:0/3237616663 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5629ff99f340 0x5629ff9a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.353+0000 7feb21674740 4 client.15360 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:12:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.353+0000 7feb21674740 1 -- 172.21.15.116:0/3237616663 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5629ff887470 con 0x5629ff99f340 2024-03-19T01:12:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.354+0000 7feb1c821640 1 --2- 172.21.15.116:0/3237616663 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5629ff99f340 0x5629ff9a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.354+0000 7feb1c821640 1 --2- 172.21.15.116:0/3237616663 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5629ff99f340 0x5629ff9a1760 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7284c07908df4bb0 server_cookie=b11f0a9b95696652 in_seq=0 out_seq=0 2024-03-19T01:12:46.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.354+0000 7feaf7fff640 10 client.15360 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:46.375 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:46 smithi116 ceph-mon[62701]: pgmap v356: 97 pgs: 97 active+clean; 228 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 35 MiB/s wr, 122 op/s 2024-03-19T01:12:46.398 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:46 smithi162 ceph-mon[67178]: pgmap v356: 97 pgs: 97 active+clean; 228 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 35 MiB/s wr, 122 op/s 2024-03-19T01:12:46.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:46 smithi156 ceph-mon[66799]: pgmap v356: 97 pgs: 97 active+clean; 228 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 35 MiB/s wr, 122 op/s 2024-03-19T01:12:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:starting vacuum...end. 2024-03-19T01:12:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feaf7fff640 1 -- 172.21.15.116:0/3237616663 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5629ff887470 con 0x5629ff99f340 2024-03-19T01:12:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feaf7fff640 10 client.15360 handle_command_reply: tid=0 2024-03-19T01:12:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feb21674740 1 client.15360 shutdown 2024-03-19T01:12:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feb21674740 1 -- 172.21.15.116:0/3237616663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff924300 msgr2=0x5629ff931760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feb21674740 1 --2- 172.21.15.116:0/3237616663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff924300 0x5629ff931760 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7feb04010fd0 tx=0x7feb040046c0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feb21674740 1 -- 172.21.15.116:0/3237616663 shutdown_connections 2024-03-19T01:12:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feb21674740 1 --2- 172.21.15.116:0/3237616663 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5629ff99f340 0x5629ff9a1760 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feb21674740 1 --2- 172.21.15.116:0/3237616663 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5629ff929470 0x5629ff931ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feb21674740 1 --2- 172.21.15.116:0/3237616663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5629ff924300 0x5629ff931760 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feaf7fff640 0 client.15360 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feb21674740 1 --2- 172.21.15.116:0/3237616663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5629ff923910 0x5629ff934cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feb21674740 1 -- 172.21.15.116:0/3237616663 >> 172.21.15.116:0/3237616663 conn(0x5629ff916670 msgr2=0x5629ff9198d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feb21674740 1 -- 172.21.15.116:0/3237616663 shutdown_connections 2024-03-19T01:12:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feb21674740 1 -- 172.21.15.116:0/3237616663 wait complete. 2024-03-19T01:12:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.576+0000 7feb21674740 20 client.15360 trim_cache size 0 max 16384 2024-03-19T01:12:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:12:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:12:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:12:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:12:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.577+0000 7feb1ea85640 1 -- 172.21.15.116:0/1540858529 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7feb080a2920 msgr2=0x7feb080a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.577+0000 7feb1ea85640 1 --2- 172.21.15.116:0/1540858529 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7feb080a2920 0x7feb080a4de0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x5629ff924b60 tx=0x7feb04013040 comp rx=0 tx=0).stop 2024-03-19T01:12:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.577+0000 7feb1ea85640 1 -- 172.21.15.116:0/1540858529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb18101430 msgr2=0x7feb181a1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.577+0000 7feb1ea85640 1 --2- 172.21.15.116:0/1540858529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb18101430 0x7feb181a1270 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7feb1000eb10 tx=0x7feb1000c2d0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.577+0000 7feb1ea85640 1 -- 172.21.15.116:0/1540858529 shutdown_connections 2024-03-19T01:12:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.577+0000 7feb1ea85640 1 --2- 172.21.15.116:0/1540858529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb18101430 0x7feb181a1270 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.577+0000 7feb1ea85640 1 --2- 172.21.15.116:0/1540858529 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7feb080a2920 0x7feb080a4de0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.577+0000 7feb1ea85640 1 --2- 172.21.15.116:0/1540858529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb18100a90 0x7feb1819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.577+0000 7feb1ea85640 1 --2- 172.21.15.116:0/1540858529 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7feb18100130 0x7feb18199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.577+0000 7feb1ea85640 1 -- 172.21.15.116:0/1540858529 >> 172.21.15.116:0/1540858529 conn(0x7feb180fbcb0 msgr2=0x7feb180fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.577+0000 7feb1ea85640 1 -- 172.21.15.116:0/1540858529 shutdown_connections 2024-03-19T01:12:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:46.577+0000 7feb1ea85640 1 -- 172.21.15.116:0/1540858529 wait complete. 2024-03-19T01:12:48.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:48 smithi162 ceph-mon[67178]: pgmap v357: 97 pgs: 97 active+clean; 328 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 52 MiB/s wr, 124 op/s 2024-03-19T01:12:48.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:48 smithi116 ceph-mon[62701]: pgmap v357: 97 pgs: 97 active+clean; 328 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 52 MiB/s wr, 124 op/s 2024-03-19T01:12:48.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:48 smithi156 ceph-mon[66799]: pgmap v357: 97 pgs: 97 active+clean; 328 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 52 MiB/s wr, 124 op/s 2024-03-19T01:12:49.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:49 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:49.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:49 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:49.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:49 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:50.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:50 smithi162 ceph-mon[67178]: pgmap v358: 97 pgs: 97 active+clean; 441 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 68 MiB/s wr, 236 op/s 2024-03-19T01:12:50.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:50 smithi116 ceph-mon[62701]: pgmap v358: 97 pgs: 97 active+clean; 441 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 68 MiB/s wr, 236 op/s 2024-03-19T01:12:50.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:50 smithi156 ceph-mon[66799]: pgmap v358: 97 pgs: 97 active+clean; 441 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 68 MiB/s wr, 236 op/s 2024-03-19T01:12:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 5.0 s, 3726.2 tps, lat 26.350 ms stddev 27.035 2024-03-19T01:12:52.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:52 smithi162 ceph-mon[67178]: pgmap v359: 97 pgs: 97 active+clean; 562 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 75 MiB/s wr, 343 op/s 2024-03-19T01:12:52.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:52 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:12:52.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:52 smithi116 ceph-mon[62701]: pgmap v359: 97 pgs: 97 active+clean; 562 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 75 MiB/s wr, 343 op/s 2024-03-19T01:12:52.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:52 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:12:52.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:52 smithi156 ceph-mon[66799]: pgmap v359: 97 pgs: 97 active+clean; 562 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 75 MiB/s wr, 343 op/s 2024-03-19T01:12:52.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:52 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:12:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:54 smithi162 ceph-mon[67178]: pgmap v360: 97 pgs: 97 active+clean; 624 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 75 MiB/s wr, 358 op/s 2024-03-19T01:12:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:54 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:54 smithi116 ceph-mon[62701]: pgmap v360: 97 pgs: 97 active+clean; 624 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 75 MiB/s wr, 358 op/s 2024-03-19T01:12:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:54 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:54 smithi156 ceph-mon[66799]: pgmap v360: 97 pgs: 97 active+clean; 624 MiB data, 735 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 75 MiB/s wr, 358 op/s 2024-03-19T01:12:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:54.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:54 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:55.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:55 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:12:55.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:55 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:12:55.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:55 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:55.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:55 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:12:55.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:55 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:12:55.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:55 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:55.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:55 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:12:55.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:55 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:12:55.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:55 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:12:56.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:56 smithi162 ceph-mon[67178]: pgmap v361: 97 pgs: 97 active+clean; 866 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 73 MiB/s wr, 500 op/s 2024-03-19T01:12:56.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:56 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:56.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:56 smithi116 ceph-mon[62701]: pgmap v361: 97 pgs: 97 active+clean; 866 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 73 MiB/s wr, 500 op/s 2024-03-19T01:12:56.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:56 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:56.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:56 smithi156 ceph-mon[66799]: pgmap v361: 97 pgs: 97 active+clean; 866 MiB data, 771 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 73 MiB/s wr, 500 op/s 2024-03-19T01:12:56.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:56 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 10.0 s, 3221.4 tps, lat 31.103 ms stddev 32.955 2024-03-19T01:12:58.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:57 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:58.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:12:57 smithi162 ceph-mon[67178]: pgmap v362: 97 pgs: 97 active+clean; 907 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 46 MiB/s wr, 607 op/s 2024-03-19T01:12:58.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:57 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:58.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:12:57 smithi116 ceph-mon[62701]: pgmap v362: 97 pgs: 97 active+clean; 907 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 46 MiB/s wr, 607 op/s 2024-03-19T01:12:58.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:57 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:12:58.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:12:57 smithi156 ceph-mon[66799]: pgmap v362: 97 pgs: 97 active+clean; 907 MiB data, 802 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 46 MiB/s wr, 607 op/s 2024-03-19T01:12:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.698+0000 7f42afe0b640 1 -- 172.21.15.116:0/3507165885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a81022e0 msgr2=0x7f42a81046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.698+0000 7f42afe0b640 1 --2- 172.21.15.116:0/3507165885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a81022e0 0x7f42a81046d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f42a4009900 tx=0x7f42a402bc30 comp rx=0 tx=0).stop 2024-03-19T01:12:59.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.698+0000 7f42afe0b640 1 -- 172.21.15.116:0/3507165885 shutdown_connections 2024-03-19T01:12:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.698+0000 7f42afe0b640 1 --2- 172.21.15.116:0/3507165885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42a8107540 0x7f42a8109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.698+0000 7f42afe0b640 1 --2- 172.21.15.116:0/3507165885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42a8104c10 0x7f42a8107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.698+0000 7f42afe0b640 1 --2- 172.21.15.116:0/3507165885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a81022e0 0x7f42a81046d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.698+0000 7f42afe0b640 1 -- 172.21.15.116:0/3507165885 >> 172.21.15.116:0/3507165885 conn(0x7f42a80fbcb0 msgr2=0x7f42a80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.698+0000 7f42afe0b640 1 -- 172.21.15.116:0/3507165885 shutdown_connections 2024-03-19T01:12:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.698+0000 7f42afe0b640 1 -- 172.21.15.116:0/3507165885 wait complete. 2024-03-19T01:12:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.698+0000 7f42afe0b640 1 Processor -- start 2024-03-19T01:12:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42afe0b640 1 -- start start 2024-03-19T01:12:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42afe0b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42a81022e0 0x7f42a8197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42afe0b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42a8104c10 0x7f42a8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42afe0b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a8107540 0x7f42a819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42afe0b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f42a810d510 con 0x7f42a8107540 2024-03-19T01:12:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42afe0b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f42a810d390 con 0x7f42a81022e0 2024-03-19T01:12:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42afe0b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f42a810d690 con 0x7f42a8104c10 2024-03-19T01:12:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42adba7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42a81022e0 0x7f42a8197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42ae3a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a8107540 0x7f42a819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42ad3a6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42a8104c10 0x7f42a8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42ae3a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a8107540 0x7f42a819f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33586/0 (socket says 172.21.15.116:33586) 2024-03-19T01:12:59.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42ad3a6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42a8104c10 0x7f42a8198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53792/0 (socket says 172.21.15.116:53792) 2024-03-19T01:12:59.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42ae3a8640 1 -- 172.21.15.116:0/510277759 learned_addr learned my addr 172.21.15.116:0/510277759 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:59.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42ad3a6640 1 -- 172.21.15.116:0/510277759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42a81022e0 msgr2=0x7f42a8197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:59.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42ad3a6640 1 --2- 172.21.15.116:0/510277759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42a81022e0 0x7f42a8197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42ad3a6640 1 -- 172.21.15.116:0/510277759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a8107540 msgr2=0x7f42a819f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:59.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42ad3a6640 1 --2- 172.21.15.116:0/510277759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a8107540 0x7f42a819f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42ad3a6640 1 -- 172.21.15.116:0/510277759 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a819f710 con 0x7f42a8104c10 2024-03-19T01:12:59.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.699+0000 7f42ad3a6640 1 --2- 172.21.15.116:0/510277759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42a8104c10 0x7f42a8198030 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f429800e9c0 tx=0x7f429800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:59.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.700+0000 7f429effd640 1 -- 172.21.15.116:0/510277759 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f429800ccc0 con 0x7f42a8104c10 2024-03-19T01:12:59.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.700+0000 7f429effd640 1 -- 172.21.15.116:0/510277759 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4298010910 con 0x7f42a8104c10 2024-03-19T01:12:59.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.700+0000 7f42afe0b640 1 -- 172.21.15.116:0/510277759 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a819fa00 con 0x7f42a8104c10 2024-03-19T01:12:59.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.700+0000 7f429effd640 1 -- 172.21.15.116:0/510277759 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42980188c0 con 0x7f42a8104c10 2024-03-19T01:12:59.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.700+0000 7f42afe0b640 1 -- 172.21.15.116:0/510277759 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a819fee0 con 0x7f42a8104c10 2024-03-19T01:12:59.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.703+0000 7f42b29f5740 1 Processor -- start 2024-03-19T01:12:59.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.703+0000 7f429effd640 1 -- 172.21.15.116:0/510277759 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f4298018a60 con 0x7f42a8104c10 2024-03-19T01:12:59.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42b29f5740 1 -- start start 2024-03-19T01:12:59.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42b29f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796100 0x5588e8796520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42b29f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796af0 0x5588e879b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42b29f5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e879bc60 0x5588e879e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42b29f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5588e86fabe0 con 0x5588e8796100 2024-03-19T01:12:59.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42b29f5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5588e86a3960 con 0x5588e879bc60 2024-03-19T01:12:59.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42b29f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5588e86f6c80 con 0x5588e8796af0 2024-03-19T01:12:59.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42ae3a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796100 0x5588e8796520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42ad3a6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e879bc60 0x5588e879e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42ae3a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796100 0x5588e8796520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33598/0 (socket says 172.21.15.116:33598) 2024-03-19T01:12:59.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42ae3a8640 1 -- 172.21.15.116:0/3633673259 learned_addr learned my addr 172.21.15.116:0/3633673259 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:59.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42adba7640 1 --2- 172.21.15.116:0/3633673259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796af0 0x5588e879b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f429effd640 1 --2- 172.21.15.116:0/510277759 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f42940a2970 0x7f42940a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42ae3a8640 1 --2- 172.21.15.116:0/510277759 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f42940a2970 0x7f42940a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42ad3a6640 1 -- 172.21.15.116:0/3633673259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796af0 msgr2=0x5588e879b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:59.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42ad3a6640 1 --2- 172.21.15.116:0/3633673259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796af0 0x5588e879b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42ad3a6640 1 -- 172.21.15.116:0/3633673259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796100 msgr2=0x5588e8796520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:59.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42ad3a6640 1 --2- 172.21.15.116:0/3633673259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796100 0x5588e8796520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.704+0000 7f42ad3a6640 1 -- 172.21.15.116:0/3633673259 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588e869ad20 con 0x5588e879bc60 2024-03-19T01:12:59.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.705+0000 7f429effd640 1 -- 172.21.15.116:0/510277759 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f42980cedc0 con 0x7f42a8104c10 2024-03-19T01:12:59.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.705+0000 7f42ad3a6640 1 --2- 172.21.15.116:0/3633673259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e879bc60 0x5588e879e0f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42980cc8f0 tx=0x7f4298024000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69a831ce9c1cc5d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:59.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.705+0000 7f42ae3a8640 1 --2- 172.21.15.116:0/510277759 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f42940a2970 0x7f42940a4e30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x5588e8797350 tx=0x7f42a0004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:59.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.705+0000 7f4293fff640 1 -- 172.21.15.116:0/3633673259 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42980071c0 con 0x5588e879bc60 2024-03-19T01:12:59.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.705+0000 7f4293fff640 1 -- 172.21.15.116:0/3633673259 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42980a3ee0 con 0x5588e879bc60 2024-03-19T01:12:59.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.705+0000 7f4293fff640 1 -- 172.21.15.116:0/3633673259 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4298010b30 con 0x5588e879bc60 2024-03-19T01:12:59.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.706+0000 7f42b29f5740 1 -- 172.21.15.116:0/3633673259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e879bc60 msgr2=0x5588e879e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:59.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.706+0000 7f42b29f5740 1 --2- 172.21.15.116:0/3633673259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e879bc60 0x5588e879e0f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f42980cc8f0 tx=0x7f4298024000 comp rx=0 tx=0).stop 2024-03-19T01:12:59.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.706+0000 7f42b29f5740 1 -- 172.21.15.116:0/3633673259 shutdown_connections 2024-03-19T01:12:59.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.707+0000 7f42b29f5740 1 --2- 172.21.15.116:0/3633673259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e879bc60 0x5588e879e0f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.707+0000 7f42b29f5740 1 --2- 172.21.15.116:0/3633673259 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796af0 0x5588e879b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.707+0000 7f42b29f5740 1 --2- 172.21.15.116:0/3633673259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796100 0x5588e8796520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.707+0000 7f42b29f5740 1 -- 172.21.15.116:0/3633673259 >> 172.21.15.116:0/3633673259 conn(0x5588e8791670 msgr2=0x5588e8791a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:59.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.707+0000 7f42b29f5740 1 -- 172.21.15.116:0/3633673259 shutdown_connections 2024-03-19T01:12:59.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.707+0000 7f42b29f5740 1 -- 172.21.15.116:0/3633673259 wait complete. 2024-03-19T01:12:59.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.707+0000 7f42b29f5740 1 -- start start 2024-03-19T01:12:59.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.707+0000 7f42b29f5740 1 Processor -- start 2024-03-19T01:12:59.714 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42b29f5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796100 0x5588e87a20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42b29f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796af0 0x5588e87a2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42b29f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e879bc60 0x5588e879eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42b29f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5588e87a2fc0 con 0x5588e879bc60 2024-03-19T01:12:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42b29f5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5588e86b6db0 con 0x5588e8796100 2024-03-19T01:12:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42b29f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5588e87a3140 con 0x5588e8796af0 2024-03-19T01:12:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42adba7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796100 0x5588e87a20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42adba7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796100 0x5588e87a20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:33550/0 (socket says 172.21.15.116:33550) 2024-03-19T01:12:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42adba7640 1 -- 172.21.15.116:0/2280881469 learned_addr learned my addr 172.21.15.116:0/2280881469 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42b29f5740 10 client.0 _ll_register_callbacks cb 0x5588e86bbe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42adba7640 1 -- 172.21.15.116:0/2280881469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796af0 msgr2=0x5588e87a2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42adba7640 1 --2- 172.21.15.116:0/2280881469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796af0 0x5588e87a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42adba7640 1 -- 172.21.15.116:0/2280881469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e879bc60 msgr2=0x5588e879eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:59.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42adba7640 1 --2- 172.21.15.116:0/2280881469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e879bc60 0x5588e879eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.708+0000 7f42adba7640 1 -- 172.21.15.116:0/2280881469 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588e86b5c60 con 0x5588e8796100 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.709+0000 7f42adba7640 1 --2- 172.21.15.116:0/2280881469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796100 0x5588e87a20c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f42a4006010 tx=0x7f42a4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=309389a20a1463f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.709+0000 7f42917fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.709+0000 7f42917fa640 1 -- 172.21.15.116:0/2280881469 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a4005030 con 0x5588e8796100 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.709+0000 7f42917fa640 1 -- 172.21.15.116:0/2280881469 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f42a40051d0 con 0x5588e8796100 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.710+0000 7f42b29f5740 1 -- 172.21.15.116:0/2280881469 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42980a3ee0 con 0x5588e8796100 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.710+0000 7f42917fa640 1 -- 172.21.15.116:0/2280881469 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f42a40053d0 con 0x5588e8796100 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.710+0000 7f42b29f5740 10 client.25208 fetch_fsmap learned FSMap version 29 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.710+0000 7f42b29f5740 1 -- 172.21.15.116:0/2280881469 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5588e840f210 con 0x5588e8796100 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.710+0000 7f42917fa640 1 -- 172.21.15.116:0/2280881469 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42980a3ee0 con 0x5588e8796100 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.710+0000 7f42917fa640 1 -- 172.21.15.116:0/2280881469 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 29) ==== 8378+0+0 (secure 0 0 0) 0x7f42a40036f0 con 0x5588e8796100 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.710+0000 7f42b29f5740 10 client.25208 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.710+0000 7f42b29f5740 10 client.25208 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.710+0000 7f42b29f5740 20 client.25208 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:59.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.710+0000 7f42b29f5740 1 --2- 172.21.15.116:0/2280881469 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5588e87b47d0 0x5588e87b6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.710+0000 7f42b29f5740 4 client.25208 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.710+0000 7f42b29f5740 1 -- 172.21.15.116:0/2280881469 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588e8708120 con 0x5588e87b47d0 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.711+0000 7f42ad3a6640 1 --2- 172.21.15.116:0/2280881469 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5588e87b47d0 0x5588e87b6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.711+0000 7f42917fa640 10 client.25208 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.711+0000 7f42ad3a6640 1 --2- 172.21.15.116:0/2280881469 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5588e87b47d0 0x5588e87b6c90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d4768a801339ed7 server_cookie=e48814ec320e8cc5 in_seq=0 out_seq=0 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.712+0000 7f42917fa640 1 -- 172.21.15.116:0/2280881469 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5588e8708120 con 0x5588e87b47d0 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.712+0000 7f42917fa640 10 client.25208 handle_command_reply: tid=0 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.712+0000 7f42b29f5740 1 client.25208 shutdown 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.712+0000 7f42b29f5740 1 -- 172.21.15.116:0/2280881469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796100 msgr2=0x5588e87a20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.712+0000 7f42b29f5740 1 --2- 172.21.15.116:0/2280881469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796100 0x5588e87a20c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f42a4006010 tx=0x7f42a4004270 comp rx=0 tx=0).stop 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.712+0000 7f42b29f5740 1 -- 172.21.15.116:0/2280881469 shutdown_connections 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.712+0000 7f42b29f5740 1 --2- 172.21.15.116:0/2280881469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e879bc60 0x5588e879eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.712+0000 7f42b29f5740 1 --2- 172.21.15.116:0/2280881469 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5588e87b47d0 0x5588e87b6c90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.712+0000 7f42b29f5740 1 --2- 172.21.15.116:0/2280881469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796af0 0x5588e87a2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.712+0000 7f42b29f5740 1 --2- 172.21.15.116:0/2280881469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796100 0x5588e87a20c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.713+0000 7f42917fa640 0 client.25208 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.713+0000 7f42b29f5740 1 -- 172.21.15.116:0/2280881469 >> 172.21.15.116:0/2280881469 conn(0x5588e8791670 msgr2=0x5588e87928c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.713+0000 7f42b29f5740 1 -- 172.21.15.116:0/2280881469 shutdown_connections 2024-03-19T01:12:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.713+0000 7f42b29f5740 1 -- 172.21.15.116:0/2280881469 wait complete. 2024-03-19T01:12:59.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.713+0000 7f42b29f5740 20 client.25208 trim_cache size 0 max 16384 2024-03-19T01:12:59.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.735+0000 7f42b29f5740 1 Processor -- start 2024-03-19T01:12:59.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.735+0000 7f42b29f5740 1 -- start start 2024-03-19T01:12:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42b29f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796100 0x5588e879e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42b29f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796af0 0x5588e87a1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42b29f5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e879bc60 0x5588e87a2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42b29f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f42a81a6bf0 con 0x5588e8796af0 2024-03-19T01:12:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42adba7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796100 0x5588e879e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42adba7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796100 0x5588e879e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53826/0 (socket says 172.21.15.116:53826) 2024-03-19T01:12:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42ad3a6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796af0 0x5588e87a1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42adba7640 1 -- 172.21.15.116:0/4258864740 learned_addr learned my addr 172.21.15.116:0/4258864740 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:59.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42ae3a8640 1 --2- 172.21.15.116:0/4258864740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e879bc60 0x5588e87a2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42b29f5740 1 -- 172.21.15.116:0/4258864740 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f42a81a8850 con 0x5588e879bc60 2024-03-19T01:12:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42b29f5740 1 -- 172.21.15.116:0/4258864740 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f42a81a72c0 con 0x5588e8796100 2024-03-19T01:12:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42adba7640 1 -- 172.21.15.116:0/4258864740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e879bc60 msgr2=0x5588e87a2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42adba7640 1 --2- 172.21.15.116:0/4258864740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e879bc60 0x5588e87a2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42adba7640 1 -- 172.21.15.116:0/4258864740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796af0 msgr2=0x5588e87a1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42adba7640 1 --2- 172.21.15.116:0/4258864740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796af0 0x5588e87a1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.736+0000 7f42adba7640 1 -- 172.21.15.116:0/4258864740 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588e84db620 con 0x5588e8796100 2024-03-19T01:12:59.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.737+0000 7f42adba7640 1 --2- 172.21.15.116:0/4258864740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796100 0x5588e879e090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f42a402c710 tx=0x7f42a402c2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeed355ff2e44e5f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.738+0000 7f42937fe640 1 -- 172.21.15.116:0/4258864740 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a4058030 con 0x5588e8796100 2024-03-19T01:12:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.738+0000 7f42937fe640 1 -- 172.21.15.116:0/4258864740 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42a4003da0 con 0x5588e8796100 2024-03-19T01:12:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.738+0000 7f42937fe640 1 -- 172.21.15.116:0/4258864740 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a4034a60 con 0x5588e8796100 2024-03-19T01:12:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.739+0000 7f42b29f5740 1 -- 172.21.15.116:0/4258864740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796100 msgr2=0x5588e879e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.739+0000 7f42b29f5740 1 --2- 172.21.15.116:0/4258864740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796100 0x5588e879e090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f42a402c710 tx=0x7f42a402c2b0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.739+0000 7f42b29f5740 1 -- 172.21.15.116:0/4258864740 shutdown_connections 2024-03-19T01:12:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.739+0000 7f42b29f5740 1 --2- 172.21.15.116:0/4258864740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e879bc60 0x5588e87a2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.739+0000 7f42b29f5740 1 --2- 172.21.15.116:0/4258864740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796af0 0x5588e87a1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.739+0000 7f42b29f5740 1 --2- 172.21.15.116:0/4258864740 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e8796100 0x5588e879e090 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.739+0000 7f42b29f5740 1 -- 172.21.15.116:0/4258864740 >> 172.21.15.116:0/4258864740 conn(0x5588e8791670 msgr2=0x5588e8790220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.740+0000 7f42b29f5740 1 -- 172.21.15.116:0/4258864740 shutdown_connections 2024-03-19T01:12:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.740+0000 7f42b29f5740 1 -- 172.21.15.116:0/4258864740 wait complete. 2024-03-19T01:12:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.740+0000 7f42b29f5740 1 -- start start 2024-03-19T01:12:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.740+0000 7f42b29f5740 1 Processor -- start 2024-03-19T01:12:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.740+0000 7f42b29f5740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796100 0x5588e87c5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.740+0000 7f42b29f5740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796af0 0x5588e87c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.740+0000 7f42b29f5740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e879bc60 0x5588e87c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42b29f5740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f42a81a80c0 con 0x5588e8796100 2024-03-19T01:12:59.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42b29f5740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f42a81a79c0 con 0x5588e8796af0 2024-03-19T01:12:59.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42b29f5740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f42a81a4bd0 con 0x5588e879bc60 2024-03-19T01:12:59.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42ad3a6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796af0 0x5588e87c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42ad3a6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796af0 0x5588e87c5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:33566/0 (socket says 172.21.15.116:33566) 2024-03-19T01:12:59.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42adba7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796100 0x5588e87c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42ad3a6640 1 -- 172.21.15.116:0/2212649315 learned_addr learned my addr 172.21.15.116:0/2212649315 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:12:59.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42ae3a8640 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e879bc60 0x5588e87c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42b29f5740 10 client.0 _ll_register_callbacks cb 0x7f42a402c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:12:59.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42adba7640 1 -- 172.21.15.116:0/2212649315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e879bc60 msgr2=0x5588e87c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:59.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42adba7640 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e879bc60 0x5588e87c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42adba7640 1 -- 172.21.15.116:0/2212649315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796af0 msgr2=0x5588e87c5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:12:59.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42adba7640 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796af0 0x5588e87c5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42adba7640 1 -- 172.21.15.116:0/2212649315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588e87c7180 con 0x5588e8796100 2024-03-19T01:12:59.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42ad3a6640 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796af0 0x5588e87c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:59.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42ae3a8640 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e879bc60 0x5588e87c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:12:59.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42adba7640 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796100 0x5588e87c5640 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f42a4003bb0 tx=0x7f42a4005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6e1fe83029f712 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:12:59.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42917fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:12:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.741+0000 7f42917fa640 1 -- 172.21.15.116:0/2212649315 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a40347e0 con 0x5588e8796100 2024-03-19T01:12:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.742+0000 7f42917fa640 1 -- 172.21.15.116:0/2212649315 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f42a4034980 con 0x5588e8796100 2024-03-19T01:12:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.742+0000 7f42b29f5740 1 -- 172.21.15.116:0/2212649315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f42a4003da0 con 0x5588e8796100 2024-03-19T01:12:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.742+0000 7f42917fa640 1 -- 172.21.15.116:0/2212649315 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f42a4034b80 con 0x5588e8796100 2024-03-19T01:12:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.742+0000 7f42b29f5740 10 client.15375 fetch_fsmap learned FSMap version 29 2024-03-19T01:12:59.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.742+0000 7f42b29f5740 1 -- 172.21.15.116:0/2212649315 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5588e881a6e0 con 0x5588e8796100 2024-03-19T01:12:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.743+0000 7f42917fa640 1 -- 172.21.15.116:0/2212649315 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f42a4003da0 con 0x5588e8796100 2024-03-19T01:12:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.743+0000 7f42917fa640 1 -- 172.21.15.116:0/2212649315 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 29) ==== 8378+0+0 (secure 0 0 0) 0x7f42a4054b20 con 0x5588e8796100 2024-03-19T01:12:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.743+0000 7f42b29f5740 10 client.15375 fetch_fsmap finished waiting for FSMap version 29 2024-03-19T01:12:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.743+0000 7f42b29f5740 10 client.15375 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:12:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.743+0000 7f42b29f5740 20 client.15375 populate_metadata read hostname 'smithi116' 2024-03-19T01:12:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.743+0000 7f42b29f5740 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5588e881b080 0x5588e881d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:12:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.743+0000 7f42b29f5740 4 client.15375 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:12:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.743+0000 7f42b29f5740 1 -- 172.21.15.116:0/2212649315 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5588e8702470 con 0x5588e881b080 2024-03-19T01:12:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.744+0000 7f42ad3a6640 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5588e881b080 0x5588e881d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:12:59.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.745+0000 7f42ad3a6640 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5588e881b080 0x5588e881d4a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3aa2ccd0c1dee6dc server_cookie=dabc818d30cc4c43 in_seq=0 out_seq=0 2024-03-19T01:12:59.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.745+0000 7f42917fa640 10 client.15375 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42917fa640 1 -- 172.21.15.116:0/2212649315 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5588e8702470 con 0x5588e881b080 2024-03-19T01:12:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42917fa640 10 client.15375 handle_command_reply: tid=0 2024-03-19T01:12:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42b29f5740 1 client.15375 shutdown 2024-03-19T01:12:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42b29f5740 1 -- 172.21.15.116:0/2212649315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796100 msgr2=0x5588e87c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42b29f5740 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796100 0x5588e87c5640 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f42a4003bb0 tx=0x7f42a4005780 comp rx=0 tx=0).stop 2024-03-19T01:12:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42b29f5740 1 -- 172.21.15.116:0/2212649315 shutdown_connections 2024-03-19T01:12:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42b29f5740 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5588e881b080 0x5588e881d4a0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42917fa640 0 client.15375 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:12:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42b29f5740 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5588e879bc60 0x5588e87c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.771 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42b29f5740 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5588e8796af0 0x5588e87c5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42b29f5740 1 --2- 172.21.15.116:0/2212649315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5588e8796100 0x5588e87c5640 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42b29f5740 1 -- 172.21.15.116:0/2212649315 >> 172.21.15.116:0/2212649315 conn(0x5588e8791670 msgr2=0x5588e87a1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42b29f5740 1 -- 172.21.15.116:0/2212649315 shutdown_connections 2024-03-19T01:12:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.769+0000 7f42b29f5740 1 -- 172.21.15.116:0/2212649315 wait complete. 2024-03-19T01:12:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.770+0000 7f42b29f5740 20 client.15375 trim_cache size 0 max 16384 2024-03-19T01:12:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:12:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:12:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:12:59.772 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:12:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.770+0000 7f42afe0b640 1 -- 172.21.15.116:0/510277759 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f42940a2970 msgr2=0x7f42940a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:59.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.770+0000 7f42afe0b640 1 --2- 172.21.15.116:0/510277759 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f42940a2970 0x7f42940a4e30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x5588e8797350 tx=0x7f42a0004ab0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.770+0000 7f42afe0b640 1 -- 172.21.15.116:0/510277759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42a8104c10 msgr2=0x7f42a8198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:12:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.770+0000 7f42afe0b640 1 --2- 172.21.15.116:0/510277759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42a8104c10 0x7f42a8198030 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f429800e9c0 tx=0x7f429800ee80 comp rx=0 tx=0).stop 2024-03-19T01:12:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.771+0000 7f42afe0b640 1 -- 172.21.15.116:0/510277759 shutdown_connections 2024-03-19T01:12:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.771+0000 7f42afe0b640 1 --2- 172.21.15.116:0/510277759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a8107540 0x7f42a819f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.771+0000 7f42afe0b640 1 --2- 172.21.15.116:0/510277759 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f42a8104c10 0x7f42a8198030 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.771+0000 7f42afe0b640 1 --2- 172.21.15.116:0/510277759 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f42940a2970 0x7f42940a4e30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.771+0000 7f42afe0b640 1 --2- 172.21.15.116:0/510277759 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42a81022e0 0x7f42a8197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:12:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.771+0000 7f42afe0b640 1 -- 172.21.15.116:0/510277759 >> 172.21.15.116:0/510277759 conn(0x7f42a80fbcb0 msgr2=0x7f42a8102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:12:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.771+0000 7f42afe0b640 1 -- 172.21.15.116:0/510277759 shutdown_connections 2024-03-19T01:12:59.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:12:59.771+0000 7f42afe0b640 1 -- 172.21.15.116:0/510277759 wait complete. 2024-03-19T01:13:00.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:00 smithi162 ceph-mon[67178]: pgmap v363: 97 pgs: 97 active+clean; 919 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 31 MiB/s wr, 704 op/s 2024-03-19T01:13:00.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:00 smithi116 ceph-mon[62701]: pgmap v363: 97 pgs: 97 active+clean; 919 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 31 MiB/s wr, 704 op/s 2024-03-19T01:13:00.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:00 smithi156 ceph-mon[66799]: pgmap v363: 97 pgs: 97 active+clean; 919 MiB data, 845 MiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 31 MiB/s wr, 704 op/s 2024-03-19T01:13:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 15.0 s, 3687.0 tps, lat 27.014 ms stddev 29.127 2024-03-19T01:13:02.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:02 smithi162 ceph-mon[67178]: pgmap v364: 97 pgs: 97 active+clean; 925 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 15 MiB/s wr, 677 op/s 2024-03-19T01:13:02.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:02 smithi116 ceph-mon[62701]: pgmap v364: 97 pgs: 97 active+clean; 925 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 15 MiB/s wr, 677 op/s 2024-03-19T01:13:02.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:02 smithi156 ceph-mon[66799]: pgmap v364: 97 pgs: 97 active+clean; 925 MiB data, 848 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 15 MiB/s wr, 677 op/s 2024-03-19T01:13:04.408 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:04 smithi116 ceph-mon[62701]: pgmap v365: 97 pgs: 97 active+clean; 928 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.7 MiB/s wr, 667 op/s 2024-03-19T01:13:04.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:04 smithi156 ceph-mon[66799]: pgmap v365: 97 pgs: 97 active+clean; 928 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.7 MiB/s wr, 667 op/s 2024-03-19T01:13:04.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:04 smithi162 ceph-mon[67178]: pgmap v365: 97 pgs: 97 active+clean; 928 MiB data, 857 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 8.7 MiB/s wr, 667 op/s 2024-03-19T01:13:06.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:06 smithi116 ceph-mon[62701]: pgmap v366: 97 pgs: 97 active+clean; 930 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.6 MiB/s wr, 817 op/s 2024-03-19T01:13:06.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:06 smithi156 ceph-mon[66799]: pgmap v366: 97 pgs: 97 active+clean; 930 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.6 MiB/s wr, 817 op/s 2024-03-19T01:13:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 20.0 s, 4031.4 tps, lat 24.894 ms stddev 26.969 2024-03-19T01:13:06.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:06 smithi162 ceph-mon[67178]: pgmap v366: 97 pgs: 97 active+clean; 930 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 9.6 MiB/s wr, 817 op/s 2024-03-19T01:13:08.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:08 smithi156 ceph-mon[66799]: pgmap v367: 97 pgs: 97 active+clean; 941 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 10 MiB/s wr, 812 op/s 2024-03-19T01:13:08.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:08 smithi162 ceph-mon[67178]: pgmap v367: 97 pgs: 97 active+clean; 941 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 10 MiB/s wr, 812 op/s 2024-03-19T01:13:08.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:08 smithi116 ceph-mon[62701]: pgmap v367: 97 pgs: 97 active+clean; 941 MiB data, 873 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 10 MiB/s wr, 812 op/s 2024-03-19T01:13:10.451 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:10 smithi162 ceph-mon[67178]: pgmap v368: 97 pgs: 97 active+clean; 950 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 9.5 MiB/s wr, 796 op/s 2024-03-19T01:13:10.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:10 smithi156 ceph-mon[66799]: pgmap v368: 97 pgs: 97 active+clean; 950 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 9.5 MiB/s wr, 796 op/s 2024-03-19T01:13:10.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:10 smithi116 ceph-mon[62701]: pgmap v368: 97 pgs: 97 active+clean; 950 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 9.5 MiB/s wr, 796 op/s 2024-03-19T01:13:11.544 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 25.0 s, 3805.0 tps, lat 26.284 ms stddev 27.093 2024-03-19T01:13:12.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:12 smithi156 ceph-mon[66799]: pgmap v369: 97 pgs: 97 active+clean; 952 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 9.3 MiB/s wr, 790 op/s 2024-03-19T01:13:12.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:12 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:13:12.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:12 smithi162 ceph-mon[67178]: pgmap v369: 97 pgs: 97 active+clean; 952 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 9.3 MiB/s wr, 790 op/s 2024-03-19T01:13:12.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:12 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:13:12.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:12 smithi116 ceph-mon[62701]: pgmap v369: 97 pgs: 97 active+clean; 952 MiB data, 935 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 9.3 MiB/s wr, 790 op/s 2024-03-19T01:13:12.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:12 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:13:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.890+0000 7f2bde55b640 1 -- 172.21.15.116:0/4257094143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bd8109990 msgr2=0x7f2bd81102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.890+0000 7f2bde55b640 1 --2- 172.21.15.116:0/4257094143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bd8109990 0x7f2bd81102b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009ca0 tx=0x7f2bd002c100 comp rx=0 tx=0).stop 2024-03-19T01:13:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.890+0000 7f2bde55b640 1 -- 172.21.15.116:0/4257094143 shutdown_connections 2024-03-19T01:13:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.890+0000 7f2bde55b640 1 --2- 172.21.15.116:0/4257094143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bd8109990 0x7f2bd81102b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.892 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.890+0000 7f2bde55b640 1 --2- 172.21.15.116:0/4257094143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bd8108fd0 0x7f2bd8109450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.890+0000 7f2bde55b640 1 --2- 172.21.15.116:0/4257094143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2bd8107e80 0x7f2bd81082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.890+0000 7f2bde55b640 1 -- 172.21.15.116:0/4257094143 >> 172.21.15.116:0/4257094143 conn(0x7f2bd8076e30 msgr2=0x7f2bd8079250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.890+0000 7f2bde55b640 1 -- 172.21.15.116:0/4257094143 shutdown_connections 2024-03-19T01:13:12.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.890+0000 7f2bde55b640 1 -- 172.21.15.116:0/4257094143 wait complete. 2024-03-19T01:13:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.891+0000 7f2bde55b640 1 Processor -- start 2024-03-19T01:13:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.891+0000 7f2bde55b640 1 -- start start 2024-03-19T01:13:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.891+0000 7f2bde55b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bd8107e80 0x7f2bd819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.891+0000 7f2bde55b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2bd8108fd0 0x7f2bd819e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.891+0000 7f2bde55b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bd8109990 0x7f2bd81a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.891+0000 7f2bcf7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2bd8108fd0 0x7f2bd819e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.891+0000 7f2bcf7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2bd8108fd0 0x7f2bd819e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:54850/0 (socket says 172.21.15.116:54850) 2024-03-19T01:13:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.892+0000 7f2bdcaf8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bd8109990 0x7f2bd81a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.896 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.892+0000 7f2bdcaf8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bd8109990 0x7f2bd81a5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37130/0 (socket says 172.21.15.116:37130) 2024-03-19T01:13:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.892+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/2100512824 learned_addr learned my addr 172.21.15.116:0/2100512824 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.892+0000 7f2bde55b640 1 -- 172.21.15.116:0/2100512824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd8113b10 con 0x7f2bd8107e80 2024-03-19T01:13:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.892+0000 7f2bde55b640 1 -- 172.21.15.116:0/2100512824 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd8113990 con 0x7f2bd8108fd0 2024-03-19T01:13:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.892+0000 7f2bde55b640 1 -- 172.21.15.116:0/2100512824 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd8113c90 con 0x7f2bd8109990 2024-03-19T01:13:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.892+0000 7f2bd7fff640 1 --2- 172.21.15.116:0/2100512824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bd8107e80 0x7f2bd819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.892+0000 7f2bdcaf8640 1 -- 172.21.15.116:0/2100512824 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2bd8108fd0 msgr2=0x7f2bd819e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.892+0000 7f2bdcaf8640 1 --2- 172.21.15.116:0/2100512824 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2bd8108fd0 0x7f2bd819e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.892+0000 7f2bdcaf8640 1 -- 172.21.15.116:0/2100512824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bd8107e80 msgr2=0x7f2bd819dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.892+0000 7f2bdcaf8640 1 --2- 172.21.15.116:0/2100512824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bd8107e80 0x7f2bd819dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.897 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.892+0000 7f2bdcaf8640 1 -- 172.21.15.116:0/2100512824 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd81a5bd0 con 0x7f2bd8109990 2024-03-19T01:13:12.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.893+0000 7f2bdcaf8640 1 --2- 172.21.15.116:0/2100512824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bd8109990 0x7f2bd81a5560 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2bd002c640 tx=0x7f2bd0035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:12.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.893+0000 7f2bd5ffb640 1 -- 172.21.15.116:0/2100512824 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd0039040 con 0x7f2bd8109990 2024-03-19T01:13:12.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.893+0000 7f2bd5ffb640 1 -- 172.21.15.116:0/2100512824 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bd0044070 con 0x7f2bd8109990 2024-03-19T01:13:12.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.893+0000 7f2bde55b640 1 -- 172.21.15.116:0/2100512824 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bd81a5e60 con 0x7f2bd8109990 2024-03-19T01:13:12.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.893+0000 7f2bd5ffb640 1 -- 172.21.15.116:0/2100512824 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd00044a0 con 0x7f2bd8109990 2024-03-19T01:13:12.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.893+0000 7f2bde55b640 1 -- 172.21.15.116:0/2100512824 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bd81a62c0 con 0x7f2bd8109990 2024-03-19T01:13:12.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.897+0000 7f2be1222740 1 Processor -- start 2024-03-19T01:13:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.898+0000 7f2bd5ffb640 1 -- 172.21.15.116:0/2100512824 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f2bd0004640 con 0x7f2bd8109990 2024-03-19T01:13:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.898+0000 7f2be1222740 1 -- start start 2024-03-19T01:13:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.898+0000 7f2be1222740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2c1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.898+0000 7f2be1222740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.898+0000 7f2be1222740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 0x55ce3a2c90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.898+0000 7f2be1222740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ce3a225be0 con 0x55ce3a2c1100 2024-03-19T01:13:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.898+0000 7f2be1222740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ce3a1ce960 con 0x55ce3a2c6c60 2024-03-19T01:13:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.898+0000 7f2be1222740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ce3a221c80 con 0x55ce3a2c1af0 2024-03-19T01:13:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.898+0000 7f2bcf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2c1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.898+0000 7f2bd7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.898+0000 7f2bcf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2c1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60810/0 (socket says 172.21.15.116:60810) 2024-03-19T01:13:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.898+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/1799813218 learned_addr learned my addr 172.21.15.116:0/1799813218 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.898+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/1799813218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 0x55ce3a2c90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.899+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/1799813218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 msgr2=0x55ce3a2c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.899+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/1799813218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.899+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/1799813218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 msgr2=0x55ce3a2c90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.899+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/1799813218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 0x55ce3a2c90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.899+0000 7f2bd5ffb640 1 --2- 172.21.15.116:0/2100512824 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2bc40a2920 0x7f2bc40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.899+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/1799813218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce3a1c5d20 con 0x55ce3a2c1100 2024-03-19T01:13:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.899+0000 7f2bd5ffb640 1 -- 172.21.15.116:0/2100512824 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2bd00f11a0 con 0x7f2bd8109990 2024-03-19T01:13:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.899+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/1799813218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2c1520 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0013a00 tx=0x7f2bc000a790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06ada852cb90230 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.900+0000 7f2bd7fff640 1 --2- 172.21.15.116:0/2100512824 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2bc40a2920 0x7f2bc40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.900+0000 7f2bce7fc640 1 -- 172.21.15.116:0/1799813218 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc000db50 con 0x55ce3a2c1100 2024-03-19T01:13:12.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.900+0000 7f2bce7fc640 1 -- 172.21.15.116:0/1799813218 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bc0005500 con 0x55ce3a2c1100 2024-03-19T01:13:12.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.900+0000 7f2bce7fc640 1 -- 172.21.15.116:0/1799813218 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0004d90 con 0x55ce3a2c1100 2024-03-19T01:13:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.901+0000 7f2be1222740 1 -- 172.21.15.116:0/1799813218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 msgr2=0x55ce3a2c1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:12.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.901+0000 7f2bd7fff640 1 --2- 172.21.15.116:0/2100512824 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2bc40a2920 0x7f2bc40a4de0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55ce3a2c1ef0 tx=0x7f2bc800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.901+0000 7f2be1222740 1 --2- 172.21.15.116:0/1799813218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2c1520 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0013a00 tx=0x7f2bc000a790 comp rx=0 tx=0).stop 2024-03-19T01:13:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.901+0000 7f2be1222740 1 -- 172.21.15.116:0/1799813218 shutdown_connections 2024-03-19T01:13:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.901+0000 7f2be1222740 1 --2- 172.21.15.116:0/1799813218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 0x55ce3a2c90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.901+0000 7f2be1222740 1 --2- 172.21.15.116:0/1799813218 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.901+0000 7f2be1222740 1 --2- 172.21.15.116:0/1799813218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2c1520 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.901+0000 7f2be1222740 1 -- 172.21.15.116:0/1799813218 >> 172.21.15.116:0/1799813218 conn(0x55ce3a2bc670 msgr2=0x55ce3a2bca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.901+0000 7f2be1222740 1 -- 172.21.15.116:0/1799813218 shutdown_connections 2024-03-19T01:13:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.901+0000 7f2be1222740 1 -- 172.21.15.116:0/1799813218 wait complete. 2024-03-19T01:13:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.902+0000 7f2be1222740 1 -- start start 2024-03-19T01:13:12.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.902+0000 7f2be1222740 1 Processor -- start 2024-03-19T01:13:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.902+0000 7f2be1222740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.902+0000 7f2be1222740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2cb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.902+0000 7f2be1222740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c6c60 0x55ce3a2d2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.902+0000 7f2be1222740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ce3a2c9bb0 con 0x55ce3a2c6c60 2024-03-19T01:13:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.902+0000 7f2be1222740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ce3a1e1db0 con 0x55ce3a2c1100 2024-03-19T01:13:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.902+0000 7f2be1222740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ce3a2c9d30 con 0x55ce3a2c1af0 2024-03-19T01:13:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.903+0000 7f2bcf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.903+0000 7f2bcf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2cb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37156/0 (socket says 172.21.15.116:37156) 2024-03-19T01:13:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.903+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/2504204382 learned_addr learned my addr 172.21.15.116:0/2504204382 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.903+0000 7f2be1222740 10 client.0 _ll_register_callbacks cb 0x55ce3a1e6e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:13:12.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.903+0000 7f2bdcaf8640 1 --2- 172.21.15.116:0/2504204382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c6c60 0x55ce3a2d2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.903+0000 7f2bd7fff640 1 --2- 172.21.15.116:0/2504204382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.903+0000 7f2bd7fff640 1 -- 172.21.15.116:0/2504204382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 msgr2=0x55ce3a2cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.903+0000 7f2bd7fff640 1 --2- 172.21.15.116:0/2504204382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2cb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.903+0000 7f2bd7fff640 1 -- 172.21.15.116:0/2504204382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c6c60 msgr2=0x55ce3a2d2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.903+0000 7f2bd7fff640 1 --2- 172.21.15.116:0/2504204382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c6c60 0x55ce3a2d2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.903+0000 7f2bd7fff640 1 -- 172.21.15.116:0/2504204382 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce3a1e0c60 con 0x55ce3a2c1100 2024-03-19T01:13:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.904+0000 7f2bd7fff640 1 --2- 172.21.15.116:0/2504204382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cb0d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55ce3a2c2560 tx=0x7f2bc80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=310d64c00871ac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.904+0000 7f2bb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:13:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.904+0000 7f2bb77fe640 1 -- 172.21.15.116:0/2504204382 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc802c060 con 0x55ce3a2c1100 2024-03-19T01:13:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.904+0000 7f2bb77fe640 1 -- 172.21.15.116:0/2504204382 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2bc8017030 con 0x55ce3a2c1100 2024-03-19T01:13:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.904+0000 7f2be1222740 1 -- 172.21.15.116:0/2504204382 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bc0005500 con 0x55ce3a2c1100 2024-03-19T01:13:12.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.905+0000 7f2bb77fe640 1 -- 172.21.15.116:0/2504204382 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2bc801fc60 con 0x55ce3a2c1100 2024-03-19T01:13:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.905+0000 7f2be1222740 10 client.25226 fetch_fsmap learned FSMap version 30 2024-03-19T01:13:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.905+0000 7f2be1222740 1 -- 172.21.15.116:0/2504204382 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ce39f3a210 con 0x55ce3a2c1100 2024-03-19T01:13:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.905+0000 7f2bb77fe640 1 -- 172.21.15.116:0/2504204382 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bc0005500 con 0x55ce3a2c1100 2024-03-19T01:13:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.905+0000 7f2bb77fe640 1 -- 172.21.15.116:0/2504204382 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f2bc80114d0 con 0x55ce3a2c1100 2024-03-19T01:13:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.905+0000 7f2be1222740 10 client.25226 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:13:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.905+0000 7f2be1222740 10 client.25226 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:13:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.905+0000 7f2be1222740 20 client.25226 populate_metadata read hostname 'smithi116' 2024-03-19T01:13:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.905+0000 7f2be1222740 1 --2- 172.21.15.116:0/2504204382 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ce3a2d7450 0x55ce3a2d98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.905+0000 7f2be1222740 4 client.25226 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:13:12.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.905+0000 7f2be1222740 1 -- 172.21.15.116:0/2504204382 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce3a233120 con 0x55ce3a2d7450 2024-03-19T01:13:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.905+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/2504204382 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ce3a2d7450 0x55ce3a2d98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.906+0000 7f2bb77fe640 10 client.25226 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.906+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/2504204382 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ce3a2d7450 0x55ce3a2d98f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3412eeb9bda2e87b server_cookie=ffed2075522b5b1e in_seq=0 out_seq=0 2024-03-19T01:13:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.907+0000 7f2bb77fe640 1 -- 172.21.15.116:0/2504204382 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ce3a233120 con 0x55ce3a2d7450 2024-03-19T01:13:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.907+0000 7f2bb77fe640 10 client.25226 handle_command_reply: tid=0 2024-03-19T01:13:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.907+0000 7f2be1222740 1 client.25226 shutdown 2024-03-19T01:13:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.907+0000 7f2be1222740 1 -- 172.21.15.116:0/2504204382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c1100 msgr2=0x55ce3a2cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.907+0000 7f2be1222740 1 --2- 172.21.15.116:0/2504204382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cb0d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55ce3a2c2560 tx=0x7f2bc80046c0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.907+0000 7f2be1222740 1 -- 172.21.15.116:0/2504204382 shutdown_connections 2024-03-19T01:13:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.907+0000 7f2be1222740 1 --2- 172.21.15.116:0/2504204382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c6c60 0x55ce3a2d2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.907+0000 7f2be1222740 1 --2- 172.21.15.116:0/2504204382 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ce3a2d7450 0x55ce3a2d98f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.907+0000 7f2be1222740 1 --2- 172.21.15.116:0/2504204382 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.907+0000 7f2be1222740 1 --2- 172.21.15.116:0/2504204382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cb0d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.907+0000 7f2be1222740 1 -- 172.21.15.116:0/2504204382 >> 172.21.15.116:0/2504204382 conn(0x55ce3a2bc670 msgr2=0x55ce3a2bd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.907+0000 7f2bb77fe640 0 client.25226 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.908+0000 7f2be1222740 1 -- 172.21.15.116:0/2504204382 shutdown_connections 2024-03-19T01:13:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.908+0000 7f2be1222740 1 -- 172.21.15.116:0/2504204382 wait complete. 2024-03-19T01:13:12.916 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.908+0000 7f2be1222740 20 client.25226 trim_cache size 0 max 16384 2024-03-19T01:13:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.934+0000 7f2be1222740 1 Processor -- start 2024-03-19T01:13:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.934+0000 7f2be1222740 1 -- start start 2024-03-19T01:13:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.934+0000 7f2be1222740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.934+0000 7f2be1222740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2d2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.934+0000 7f2be1222740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 0x55ce3a2bb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.934+0000 7f2be1222740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd81af300 con 0x55ce3a2c1100 2024-03-19T01:13:12.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.934+0000 7f2be1222740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd81b0f90 con 0x55ce3a2c6c60 2024-03-19T01:13:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.934+0000 7f2be1222740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd81af9d0 con 0x55ce3a2c1af0 2024-03-19T01:13:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.934+0000 7f2bd7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.934+0000 7f2bd7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60826/0 (socket says 172.21.15.116:60826) 2024-03-19T01:13:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.934+0000 7f2bd7fff640 1 -- 172.21.15.116:0/745758468 learned_addr learned my addr 172.21.15.116:0/745758468 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:12.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.934+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/745758468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2d2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.935+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/745758468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 msgr2=0x55ce3a2bb930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:13:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.935+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/745758468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 0x55ce3a2bb930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.935+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/745758468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 msgr2=0x55ce3a2cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.935+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/745758468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.935+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/745758468 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce3a006620 con 0x55ce3a2c1af0 2024-03-19T01:13:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.935+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/745758468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2d2610 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55ce3a2da1a0 tx=0x7f2bc000a9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2e50e2c8c7a36c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.935+0000 7f2bcdffb640 1 -- 172.21.15.116:0/745758468 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc000ab40 con 0x55ce3a2c1af0 2024-03-19T01:13:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.935+0000 7f2bcdffb640 1 -- 172.21.15.116:0/745758468 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bc0013240 con 0x55ce3a2c1af0 2024-03-19T01:13:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.935+0000 7f2bcdffb640 1 -- 172.21.15.116:0/745758468 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc000dbd0 con 0x55ce3a2c1af0 2024-03-19T01:13:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.936+0000 7f2be1222740 1 -- 172.21.15.116:0/745758468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 msgr2=0x55ce3a2d2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:12.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.936+0000 7f2be1222740 1 --2- 172.21.15.116:0/745758468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2d2610 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x55ce3a2da1a0 tx=0x7f2bc000a9d0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.937+0000 7f2be1222740 1 -- 172.21.15.116:0/745758468 shutdown_connections 2024-03-19T01:13:12.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.937+0000 7f2be1222740 1 --2- 172.21.15.116:0/745758468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 0x55ce3a2bb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.937+0000 7f2be1222740 1 --2- 172.21.15.116:0/745758468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2d2610 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.937+0000 7f2be1222740 1 --2- 172.21.15.116:0/745758468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.938+0000 7f2be1222740 1 -- 172.21.15.116:0/745758468 >> 172.21.15.116:0/745758468 conn(0x55ce3a2bc670 msgr2=0x55ce3a2bb220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.938+0000 7f2be1222740 1 -- 172.21.15.116:0/745758468 shutdown_connections 2024-03-19T01:13:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.938+0000 7f2be1222740 1 -- 172.21.15.116:0/745758468 wait complete. 2024-03-19T01:13:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.939+0000 7f2be1222740 1 -- start start 2024-03-19T01:13:12.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.939+0000 7f2be1222740 1 Processor -- start 2024-03-19T01:13:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2be1222740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2be1222740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2be1222740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 0x55ce3a33b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2bd7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2bd7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60828/0 (socket says 172.21.15.116:60828) 2024-03-19T01:13:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2bcf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2bcf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37172/0 (socket says 172.21.15.116:37172) 2024-03-19T01:13:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2bd7fff640 1 -- 172.21.15.116:0/1486845697 learned_addr learned my addr 172.21.15.116:0/1486845697 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2be1222740 1 -- 172.21.15.116:0/1486845697 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd81b0800 con 0x55ce3a2c1100 2024-03-19T01:13:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2be1222740 1 -- 172.21.15.116:0/1486845697 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd81b0100 con 0x55ce3a2c6c60 2024-03-19T01:13:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2be1222740 1 -- 172.21.15.116:0/1486845697 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd81ad2e0 con 0x55ce3a2c1af0 2024-03-19T01:13:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2be1222740 10 client.0 _ll_register_callbacks cb 0x7f2bc800bf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:13:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2bdcaf8640 1 --2- 172.21.15.116:0/1486845697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 0x55ce3a33b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/1486845697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 msgr2=0x55ce3a33b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/1486845697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 0x55ce3a33b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/1486845697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 msgr2=0x55ce3a2cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:12.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/1486845697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.940+0000 7f2bcf7fe640 1 -- 172.21.15.116:0/1486845697 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce3a33bd60 con 0x55ce3a2c1af0 2024-03-19T01:13:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.941+0000 7f2bcf7fe640 1 --2- 172.21.15.116:0/1486845697 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2cfaf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0006010 tx=0x7f2bc000d9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c37eba175bdee8aa server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:12.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.941+0000 7f2bb77fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:13:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.941+0000 7f2bb77fe640 1 -- 172.21.15.116:0/1486845697 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc000ab40 con 0x55ce3a2c1af0 2024-03-19T01:13:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.941+0000 7f2bb77fe640 1 -- 172.21.15.116:0/1486845697 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2bc000db70 con 0x55ce3a2c1af0 2024-03-19T01:13:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.941+0000 7f2be1222740 1 -- 172.21.15.116:0/1486845697 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2bc0013240 con 0x55ce3a2c1af0 2024-03-19T01:13:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.941+0000 7f2bb77fe640 1 -- 172.21.15.116:0/1486845697 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2bc000dd90 con 0x55ce3a2c1af0 2024-03-19T01:13:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.941+0000 7f2be1222740 10 client.25207 fetch_fsmap learned FSMap version 30 2024-03-19T01:13:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.941+0000 7f2be1222740 1 -- 172.21.15.116:0/1486845697 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ce3a33cf70 con 0x55ce3a2c1af0 2024-03-19T01:13:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.942+0000 7f2bb77fe640 1 -- 172.21.15.116:0/1486845697 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bc0013240 con 0x55ce3a2c1af0 2024-03-19T01:13:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.942+0000 7f2bb77fe640 1 -- 172.21.15.116:0/1486845697 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f2bc0027070 con 0x55ce3a2c1af0 2024-03-19T01:13:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.943+0000 7f2be1222740 10 client.25207 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:13:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.943+0000 7f2be1222740 10 client.25207 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:13:12.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.943+0000 7f2be1222740 20 client.25207 populate_metadata read hostname 'smithi116' 2024-03-19T01:13:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.943+0000 7f2be1222740 1 --2- 172.21.15.116:0/1486845697 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ce3a33d7d0 0x55ce3a33fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.943+0000 7f2be1222740 4 client.25207 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:13:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.943+0000 7f2be1222740 1 -- 172.21.15.116:0/1486845697 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ce3a22d470 con 0x55ce3a33d7d0 2024-03-19T01:13:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.943+0000 7f2bd7fff640 1 --2- 172.21.15.116:0/1486845697 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ce3a33d7d0 0x55ce3a33fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.943+0000 7f2bd7fff640 1 --2- 172.21.15.116:0/1486845697 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ce3a33d7d0 0x55ce3a33fbf0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=472c49903c76571 server_cookie=9b214338dd060425 in_seq=0 out_seq=0 2024-03-19T01:13:12.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.943+0000 7f2bb77fe640 10 client.25207 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.957+0000 7f2bb77fe640 1 -- 172.21.15.116:0/1486845697 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ce3a22d470 con 0x55ce3a33d7d0 2024-03-19T01:13:12.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.957+0000 7f2bb77fe640 10 client.25207 handle_command_reply: tid=0 2024-03-19T01:13:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.957+0000 7f2be1222740 1 client.25207 shutdown 2024-03-19T01:13:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.958+0000 7f2be1222740 1 -- 172.21.15.116:0/1486845697 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 msgr2=0x55ce3a2cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.958+0000 7f2be1222740 1 --2- 172.21.15.116:0/1486845697 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2cfaf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0006010 tx=0x7f2bc000d9e0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.958+0000 7f2be1222740 1 -- 172.21.15.116:0/1486845697 shutdown_connections 2024-03-19T01:13:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.958+0000 7f2be1222740 1 --2- 172.21.15.116:0/1486845697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ce3a2c6c60 0x55ce3a33b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.958+0000 7f2be1222740 1 --2- 172.21.15.116:0/1486845697 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ce3a33d7d0 0x55ce3a33fbf0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.958+0000 7f2be1222740 1 --2- 172.21.15.116:0/1486845697 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ce3a2c1af0 0x55ce3a2cfaf0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.958+0000 7f2be1222740 1 --2- 172.21.15.116:0/1486845697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ce3a2c1100 0x55ce3a2cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.958+0000 7f2be1222740 1 -- 172.21.15.116:0/1486845697 >> 172.21.15.116:0/1486845697 conn(0x55ce3a2bc670 msgr2=0x55ce3a2bf8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.958+0000 7f2bb77fe640 0 client.25207 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.958+0000 7f2be1222740 1 -- 172.21.15.116:0/1486845697 shutdown_connections 2024-03-19T01:13:12.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.958+0000 7f2be1222740 1 -- 172.21.15.116:0/1486845697 wait complete. 2024-03-19T01:13:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.958+0000 7f2be1222740 20 client.25207 trim_cache size 0 max 16384 2024-03-19T01:13:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:13:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:13:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:13:12.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:13:12.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.960+0000 7f2bde55b640 1 -- 172.21.15.116:0/2100512824 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2bc40a2920 msgr2=0x7f2bc40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:12.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.960+0000 7f2bde55b640 1 --2- 172.21.15.116:0/2100512824 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2bc40a2920 0x7f2bc40a4de0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55ce3a2c1ef0 tx=0x7f2bc800b390 comp rx=0 tx=0).stop 2024-03-19T01:13:12.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.960+0000 7f2bde55b640 1 -- 172.21.15.116:0/2100512824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bd8109990 msgr2=0x7f2bd81a5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:12.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.960+0000 7f2bde55b640 1 --2- 172.21.15.116:0/2100512824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bd8109990 0x7f2bd81a5560 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2bd002c640 tx=0x7f2bd0035d90 comp rx=0 tx=0).stop 2024-03-19T01:13:12.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.960+0000 7f2bde55b640 1 -- 172.21.15.116:0/2100512824 shutdown_connections 2024-03-19T01:13:12.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.960+0000 7f2bde55b640 1 --2- 172.21.15.116:0/2100512824 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2bd8109990 0x7f2bd81a5560 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.960+0000 7f2bde55b640 1 --2- 172.21.15.116:0/2100512824 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2bc40a2920 0x7f2bc40a4de0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.960+0000 7f2bde55b640 1 --2- 172.21.15.116:0/2100512824 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2bd8108fd0 0x7f2bd819e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.960+0000 7f2bde55b640 1 --2- 172.21.15.116:0/2100512824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bd8107e80 0x7f2bd819dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:12.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.960+0000 7f2bde55b640 1 -- 172.21.15.116:0/2100512824 >> 172.21.15.116:0/2100512824 conn(0x7f2bd8076e30 msgr2=0x7f2bd810c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:12.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.960+0000 7f2bde55b640 1 -- 172.21.15.116:0/2100512824 shutdown_connections 2024-03-19T01:13:12.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:12.960+0000 7f2bde55b640 1 -- 172.21.15.116:0/2100512824 wait complete. 2024-03-19T01:13:14.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:14 smithi156 ceph-mon[66799]: pgmap v370: 97 pgs: 97 active+clean; 954 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 9.6 MiB/s wr, 790 op/s 2024-03-19T01:13:14.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:14 smithi162 ceph-mon[67178]: pgmap v370: 97 pgs: 97 active+clean; 954 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 9.6 MiB/s wr, 790 op/s 2024-03-19T01:13:14.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:14 smithi116 ceph-mon[62701]: pgmap v370: 97 pgs: 97 active+clean; 954 MiB data, 969 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 9.6 MiB/s wr, 790 op/s 2024-03-19T01:13:16.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:16 smithi156 ceph-mon[66799]: pgmap v371: 97 pgs: 97 active+clean; 971 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 12 MiB/s wr, 884 op/s 2024-03-19T01:13:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 30.0 s, 3845.0 tps, lat 25.980 ms stddev 27.813 2024-03-19T01:13:16.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:16 smithi162 ceph-mon[67178]: pgmap v371: 97 pgs: 97 active+clean; 971 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 12 MiB/s wr, 884 op/s 2024-03-19T01:13:16.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:16 smithi116 ceph-mon[62701]: pgmap v371: 97 pgs: 97 active+clean; 971 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 12 MiB/s wr, 884 op/s 2024-03-19T01:13:18.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:18 smithi156 ceph-mon[66799]: pgmap v372: 97 pgs: 97 active+clean; 974 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 760 op/s 2024-03-19T01:13:18.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:18 smithi162 ceph-mon[67178]: pgmap v372: 97 pgs: 97 active+clean; 974 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 760 op/s 2024-03-19T01:13:18.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:18 smithi116 ceph-mon[62701]: pgmap v372: 97 pgs: 97 active+clean; 974 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 11 MiB/s wr, 760 op/s 2024-03-19T01:13:19.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:19 smithi162 ceph-mon[67178]: pgmap v373: 97 pgs: 97 active+clean; 975 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.9 MiB/s wr, 753 op/s 2024-03-19T01:13:19.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:19 smithi116 ceph-mon[62701]: pgmap v373: 97 pgs: 97 active+clean; 975 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.9 MiB/s wr, 753 op/s 2024-03-19T01:13:19.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:19 smithi156 ceph-mon[66799]: pgmap v373: 97 pgs: 97 active+clean; 975 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 9.9 MiB/s wr, 753 op/s 2024-03-19T01:13:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 35.0 s, 4043.4 tps, lat 24.761 ms stddev 26.641 2024-03-19T01:13:22.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:22 smithi116 ceph-mon[62701]: pgmap v374: 97 pgs: 97 active+clean; 987 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 797 op/s 2024-03-19T01:13:22.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:22 smithi156 ceph-mon[66799]: pgmap v374: 97 pgs: 97 active+clean; 987 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 797 op/s 2024-03-19T01:13:22.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:22 smithi162 ceph-mon[67178]: pgmap v374: 97 pgs: 97 active+clean; 987 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 797 op/s 2024-03-19T01:13:24.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:24 smithi116 ceph-mon[62701]: pgmap v375: 97 pgs: 97 active+clean; 1003 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 MiB/s wr, 734 op/s 2024-03-19T01:13:24.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:24 smithi156 ceph-mon[66799]: pgmap v375: 97 pgs: 97 active+clean; 1003 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 MiB/s wr, 734 op/s 2024-03-19T01:13:24.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:24 smithi162 ceph-mon[67178]: pgmap v375: 97 pgs: 97 active+clean; 1003 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 MiB/s wr, 734 op/s 2024-03-19T01:13:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.094+0000 7f7afd57e640 1 -- 172.21.15.116:0/2201410202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af8108fd0 msgr2=0x7f7af8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.094+0000 7f7afd57e640 1 --2- 172.21.15.116:0/2201410202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af8108fd0 0x7f7af8109450 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009900 tx=0x7f7aec02bc30 comp rx=0 tx=0).stop 2024-03-19T01:13:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.094+0000 7f7afd57e640 1 -- 172.21.15.116:0/2201410202 shutdown_connections 2024-03-19T01:13:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.094+0000 7f7afd57e640 1 --2- 172.21.15.116:0/2201410202 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af8109990 0x7f7af81102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.094+0000 7f7afd57e640 1 --2- 172.21.15.116:0/2201410202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af8108fd0 0x7f7af8109450 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.094+0000 7f7afd57e640 1 --2- 172.21.15.116:0/2201410202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af8107e80 0x7f7af81082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.094+0000 7f7afd57e640 1 -- 172.21.15.116:0/2201410202 >> 172.21.15.116:0/2201410202 conn(0x7f7af8076e30 msgr2=0x7f7af8079250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.094+0000 7f7afd57e640 1 -- 172.21.15.116:0/2201410202 shutdown_connections 2024-03-19T01:13:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.094+0000 7f7afd57e640 1 -- 172.21.15.116:0/2201410202 wait complete. 2024-03-19T01:13:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.095+0000 7f7afd57e640 1 Processor -- start 2024-03-19T01:13:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.095+0000 7f7afd57e640 1 -- start start 2024-03-19T01:13:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.095+0000 7f7afd57e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af8107e80 0x7f7af819e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7afd57e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af8108fd0 0x7f7af819e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7af67fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af8108fd0 0x7f7af819e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7afd57e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af8109990 0x7f7af81a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7af67fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af8108fd0 0x7f7af819e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:47786/0 (socket says 172.21.15.116:47786) 2024-03-19T01:13:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7af6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af8107e80 0x7f7af819e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7af67fc640 1 -- 172.21.15.116:0/2978922044 learned_addr learned my addr 172.21.15.116:0/2978922044 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7afd57e640 1 -- 172.21.15.116:0/2978922044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7af8113b20 con 0x7f7af8109990 2024-03-19T01:13:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7afd57e640 1 -- 172.21.15.116:0/2978922044 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7af81139a0 con 0x7f7af8108fd0 2024-03-19T01:13:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7afd57e640 1 -- 172.21.15.116:0/2978922044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7af8113ca0 con 0x7f7af8107e80 2024-03-19T01:13:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7af77fe640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af8109990 0x7f7af81a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7af6ffd640 1 -- 172.21.15.116:0/2978922044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af8108fd0 msgr2=0x7f7af819e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7af6ffd640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af8108fd0 0x7f7af819e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7af6ffd640 1 -- 172.21.15.116:0/2978922044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af8109990 msgr2=0x7f7af81a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7af6ffd640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af8109990 0x7f7af81a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7af6ffd640 1 -- 172.21.15.116:0/2978922044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af81a5cd0 con 0x7f7af8107e80 2024-03-19T01:13:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7af67fc640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af8108fd0 0x7f7af819e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:13:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.096+0000 7f7af77fe640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af8109990 0x7f7af81a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:13:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.097+0000 7f7af6ffd640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af8107e80 0x7f7af819e0b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800e9f0 tx=0x7f7ae800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.097+0000 7f7adffff640 1 -- 172.21.15.116:0/2978922044 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae800ccc0 con 0x7f7af8107e80 2024-03-19T01:13:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.097+0000 7f7afd57e640 1 -- 172.21.15.116:0/2978922044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af81a5fc0 con 0x7f7af8107e80 2024-03-19T01:13:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.097+0000 7f7adffff640 1 -- 172.21.15.116:0/2978922044 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ae8010910 con 0x7f7af8107e80 2024-03-19T01:13:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.097+0000 7f7adffff640 1 -- 172.21.15.116:0/2978922044 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8018960 con 0x7f7af8107e80 2024-03-19T01:13:26.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.097+0000 7f7afd57e640 1 -- 172.21.15.116:0/2978922044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af81a6430 con 0x7f7af8107e80 2024-03-19T01:13:26.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.100+0000 7f7b0016d740 1 Processor -- start 2024-03-19T01:13:26.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.101+0000 7f7adffff640 1 -- 172.21.15.116:0/2978922044 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7ae8010ac0 con 0x7f7af8107e80 2024-03-19T01:13:26.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7b0016d740 1 -- start start 2024-03-19T01:13:26.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7b0016d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b90e100 0x564a3b90e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7b0016d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b9135f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7b0016d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b913c60 0x564a3b9160f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7b0016d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564a3b872be0 con 0x564a3b90eaf0 2024-03-19T01:13:26.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7b0016d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564a3b81b960 con 0x564a3b90e100 2024-03-19T01:13:26.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7b0016d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564a3b86ec80 con 0x564a3b913c60 2024-03-19T01:13:26.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7af6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b9135f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7af67fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b913c60 0x564a3b9160f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7af67fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b913c60 0x564a3b9160f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48126/0 (socket says 172.21.15.116:48126) 2024-03-19T01:13:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7af67fc640 1 -- 172.21.15.116:0/664557613 learned_addr learned my addr 172.21.15.116:0/664557613 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7af77fe640 1 --2- 172.21.15.116:0/664557613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b90e100 0x564a3b90e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7af6ffd640 1 -- 172.21.15.116:0/664557613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b913c60 msgr2=0x564a3b9160f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:26.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7af6ffd640 1 --2- 172.21.15.116:0/664557613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b913c60 0x564a3b9160f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7af6ffd640 1 -- 172.21.15.116:0/664557613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b90e100 msgr2=0x564a3b90e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:26.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7af6ffd640 1 --2- 172.21.15.116:0/664557613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b90e100 0x564a3b90e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7af6ffd640 1 -- 172.21.15.116:0/664557613 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a3b812d20 con 0x564a3b90eaf0 2024-03-19T01:13:26.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.102+0000 7f7af67fc640 1 --2- 172.21.15.116:0/664557613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b913c60 0x564a3b9160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:13:26.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.103+0000 7f7af6ffd640 1 --2- 172.21.15.116:0/664557613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b9135f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80cca10 tx=0x7f7ae80d7570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40dcad3f2c729a57 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.103+0000 7f7add7fa640 1 -- 172.21.15.116:0/664557613 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae80e2070 con 0x564a3b90eaf0 2024-03-19T01:13:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.103+0000 7f7add7fa640 1 -- 172.21.15.116:0/664557613 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ae80ddae0 con 0x564a3b90eaf0 2024-03-19T01:13:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.103+0000 7f7adffff640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ae00a2920 0x7f7ae00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.103+0000 7f7af77fe640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ae00a2920 0x7f7ae00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.104+0000 7f7add7fa640 1 -- 172.21.15.116:0/664557613 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae80dcb60 con 0x564a3b90eaf0 2024-03-19T01:13:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.104+0000 7f7adffff640 1 -- 172.21.15.116:0/2978922044 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7ae80ceff0 con 0x7f7af8107e80 2024-03-19T01:13:26.109 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.104+0000 7f7af77fe640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ae00a2920 0x7f7ae00a4de0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x564a3b90f350 tx=0x7f7af0009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.104+0000 7f7b0016d740 1 -- 172.21.15.116:0/664557613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 msgr2=0x564a3b9135f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.104+0000 7f7b0016d740 1 --2- 172.21.15.116:0/664557613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b9135f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80cca10 tx=0x7f7ae80d7570 comp rx=0 tx=0).stop 2024-03-19T01:13:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.104+0000 7f7b0016d740 1 -- 172.21.15.116:0/664557613 shutdown_connections 2024-03-19T01:13:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.104+0000 7f7b0016d740 1 --2- 172.21.15.116:0/664557613 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b913c60 0x564a3b9160f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.104+0000 7f7b0016d740 1 --2- 172.21.15.116:0/664557613 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b9135f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.104+0000 7f7b0016d740 1 --2- 172.21.15.116:0/664557613 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b90e100 0x564a3b90e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.111 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.104+0000 7f7b0016d740 1 -- 172.21.15.116:0/664557613 >> 172.21.15.116:0/664557613 conn(0x564a3b909670 msgr2=0x564a3b909a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:26.111 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.104+0000 7f7b0016d740 1 -- 172.21.15.116:0/664557613 shutdown_connections 2024-03-19T01:13:26.111 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.104+0000 7f7b0016d740 1 -- 172.21.15.116:0/664557613 wait complete. 2024-03-19T01:13:26.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.105+0000 7f7b0016d740 1 -- start start 2024-03-19T01:13:26.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.105+0000 7f7b0016d740 1 Processor -- start 2024-03-19T01:13:26.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.106+0000 7f7b0016d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90e100 0x564a3b918170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.106+0000 7f7b0016d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b90eaf0 0x564a3b9186b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.106+0000 7f7b0016d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b913c60 0x564a3b91f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.106+0000 7f7b0016d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564a3b916c60 con 0x564a3b90e100 2024-03-19T01:13:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.106+0000 7f7b0016d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564a3b82edb0 con 0x564a3b90eaf0 2024-03-19T01:13:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.107+0000 7f7b0016d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564a3b916de0 con 0x564a3b913c60 2024-03-19T01:13:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.107+0000 7f7b0016d740 10 client.0 _ll_register_callbacks cb 0x564a3b833e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:13:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.107+0000 7f7af67fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b90eaf0 0x564a3b9186b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.107+0000 7f7af67fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b90eaf0 0x564a3b9186b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:47814/0 (socket says 172.21.15.116:47814) 2024-03-19T01:13:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.107+0000 7f7af67fc640 1 -- 172.21.15.116:0/3719385807 learned_addr learned my addr 172.21.15.116:0/3719385807 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.107+0000 7f7af6ffd640 1 --2- 172.21.15.116:0/3719385807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90e100 0x564a3b918170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.107+0000 7f7af67fc640 1 -- 172.21.15.116:0/3719385807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b913c60 msgr2=0x564a3b91f6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:13:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.107+0000 7f7af67fc640 1 --2- 172.21.15.116:0/3719385807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b913c60 0x564a3b91f6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.107+0000 7f7af67fc640 1 -- 172.21.15.116:0/3719385807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90e100 msgr2=0x564a3b918170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.107+0000 7f7af67fc640 1 --2- 172.21.15.116:0/3719385807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90e100 0x564a3b918170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.107+0000 7f7af67fc640 1 -- 172.21.15.116:0/3719385807 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a3b82dc60 con 0x564a3b90eaf0 2024-03-19T01:13:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.108+0000 7f7af67fc640 1 --2- 172.21.15.116:0/3719385807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b90eaf0 0x564a3b9186b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7aec02cfd0 tx=0x7f7aec004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f694886748cf14 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.108+0000 7f7adaff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:13:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.108+0000 7f7adaff5640 1 -- 172.21.15.116:0/3719385807 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec005d50 con 0x564a3b90eaf0 2024-03-19T01:13:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.108+0000 7f7adaff5640 1 -- 172.21.15.116:0/3719385807 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7aec005230 con 0x564a3b90eaf0 2024-03-19T01:13:26.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.108+0000 7f7b0016d740 1 -- 172.21.15.116:0/3719385807 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ae80ddae0 con 0x564a3b90eaf0 2024-03-19T01:13:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.109+0000 7f7adaff5640 1 -- 172.21.15.116:0/3719385807 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7aec005430 con 0x564a3b90eaf0 2024-03-19T01:13:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.109+0000 7f7b0016d740 10 client.25241 fetch_fsmap learned FSMap version 30 2024-03-19T01:13:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.109+0000 7f7b0016d740 1 -- 172.21.15.116:0/3719385807 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564a3b587210 con 0x564a3b90eaf0 2024-03-19T01:13:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.109+0000 7f7adaff5640 1 -- 172.21.15.116:0/3719385807 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ae80ddae0 con 0x564a3b90eaf0 2024-03-19T01:13:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.109+0000 7f7adaff5640 1 -- 172.21.15.116:0/3719385807 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f7aec003ae0 con 0x564a3b90eaf0 2024-03-19T01:13:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.109+0000 7f7b0016d740 10 client.25241 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:13:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.109+0000 7f7b0016d740 10 client.25241 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:13:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.109+0000 7f7b0016d740 20 client.25241 populate_metadata read hostname 'smithi116' 2024-03-19T01:13:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.109+0000 7f7b0016d740 1 --2- 172.21.15.116:0/3719385807 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564a3b935540 0x564a3b9379a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.109+0000 7f7b0016d740 4 client.25241 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:13:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.109+0000 7f7b0016d740 1 -- 172.21.15.116:0/3719385807 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a3b880120 con 0x564a3b935540 2024-03-19T01:13:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.110+0000 7f7af6ffd640 1 --2- 172.21.15.116:0/3719385807 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564a3b935540 0x564a3b9379a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.111+0000 7f7af6ffd640 1 --2- 172.21.15.116:0/3719385807 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564a3b935540 0x564a3b9379a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fe8df80a3b169bd server_cookie=8dd5b1bbd9baf160 in_seq=0 out_seq=0 2024-03-19T01:13:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.111+0000 7f7adaff5640 10 client.25241 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:26.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.112+0000 7f7adaff5640 1 -- 172.21.15.116:0/3719385807 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564a3b880120 con 0x564a3b935540 2024-03-19T01:13:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.112+0000 7f7adaff5640 10 client.25241 handle_command_reply: tid=0 2024-03-19T01:13:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.112+0000 7f7b0016d740 1 client.25241 shutdown 2024-03-19T01:13:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.112+0000 7f7b0016d740 1 -- 172.21.15.116:0/3719385807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b90eaf0 msgr2=0x564a3b9186b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.112+0000 7f7b0016d740 1 --2- 172.21.15.116:0/3719385807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b90eaf0 0x564a3b9186b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7aec02cfd0 tx=0x7f7aec004270 comp rx=0 tx=0).stop 2024-03-19T01:13:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.113+0000 7f7b0016d740 1 -- 172.21.15.116:0/3719385807 shutdown_connections 2024-03-19T01:13:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.113+0000 7f7b0016d740 1 --2- 172.21.15.116:0/3719385807 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564a3b935540 0x564a3b9379a0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.113+0000 7f7b0016d740 1 --2- 172.21.15.116:0/3719385807 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b913c60 0x564a3b91f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.113+0000 7f7b0016d740 1 --2- 172.21.15.116:0/3719385807 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b90eaf0 0x564a3b9186b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.113+0000 7f7b0016d740 1 --2- 172.21.15.116:0/3719385807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90e100 0x564a3b918170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.113+0000 7f7b0016d740 1 -- 172.21.15.116:0/3719385807 >> 172.21.15.116:0/3719385807 conn(0x564a3b909670 msgr2=0x564a3b90a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.113+0000 7f7adaff5640 0 client.25241 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.113+0000 7f7b0016d740 1 -- 172.21.15.116:0/3719385807 shutdown_connections 2024-03-19T01:13:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.114+0000 7f7b0016d740 1 -- 172.21.15.116:0/3719385807 wait complete. 2024-03-19T01:13:26.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.114+0000 7f7b0016d740 20 client.25241 trim_cache size 0 max 16384 2024-03-19T01:13:26.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.135+0000 7f7b0016d740 1 Processor -- start 2024-03-19T01:13:26.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.135+0000 7f7b0016d740 1 -- start start 2024-03-19T01:13:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.135+0000 7f7b0016d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b90e100 0x564a3b908220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.135+0000 7f7b0016d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b91f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.136+0000 7f7b0016d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b913c60 0x564a3b908760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.136+0000 7f7b0016d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7af81a6b10 con 0x564a3b90eaf0 2024-03-19T01:13:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.136+0000 7f7b0016d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7af81a87a0 con 0x564a3b913c60 2024-03-19T01:13:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.136+0000 7f7b0016d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7af81a71e0 con 0x564a3b90e100 2024-03-19T01:13:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.136+0000 7f7af77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b913c60 0x564a3b908760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.136+0000 7f7af67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b91f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.136+0000 7f7af77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b913c60 0x564a3b908760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:47822/0 (socket says 172.21.15.116:47822) 2024-03-19T01:13:26.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.136+0000 7f7af77fe640 1 -- 172.21.15.116:0/45153201 learned_addr learned my addr 172.21.15.116:0/45153201 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.137+0000 7f7af77fe640 1 -- 172.21.15.116:0/45153201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b90e100 msgr2=0x564a3b908220 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:13:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.137+0000 7f7af77fe640 1 --2- 172.21.15.116:0/45153201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b90e100 0x564a3b908220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.137+0000 7f7af77fe640 1 -- 172.21.15.116:0/45153201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 msgr2=0x564a3b91f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:26.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.137+0000 7f7af77fe640 1 --2- 172.21.15.116:0/45153201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b91f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.137+0000 7f7af77fe640 1 -- 172.21.15.116:0/45153201 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a3b653620 con 0x564a3b913c60 2024-03-19T01:13:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.137+0000 7f7af77fe640 1 --2- 172.21.15.116:0/45153201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b913c60 0x564a3b908760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7af0010810 tx=0x7f7af0015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21bc083af890f64f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.137+0000 7f7adcff9640 1 -- 172.21.15.116:0/45153201 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af001d8b0 con 0x564a3b913c60 2024-03-19T01:13:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.137+0000 7f7adcff9640 1 -- 172.21.15.116:0/45153201 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7af0029480 con 0x564a3b913c60 2024-03-19T01:13:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.137+0000 7f7adcff9640 1 -- 172.21.15.116:0/45153201 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af001e680 con 0x564a3b913c60 2024-03-19T01:13:26.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 -- 172.21.15.116:0/45153201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b913c60 msgr2=0x564a3b908760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 --2- 172.21.15.116:0/45153201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b913c60 0x564a3b908760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7af0010810 tx=0x7f7af0015000 comp rx=0 tx=0).stop 2024-03-19T01:13:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 -- 172.21.15.116:0/45153201 shutdown_connections 2024-03-19T01:13:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 --2- 172.21.15.116:0/45153201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b913c60 0x564a3b908760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 --2- 172.21.15.116:0/45153201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b91f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 --2- 172.21.15.116:0/45153201 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b90e100 0x564a3b908220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 -- 172.21.15.116:0/45153201 >> 172.21.15.116:0/45153201 conn(0x564a3b909670 msgr2=0x564a3b918170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:26.151 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 -- 172.21.15.116:0/45153201 shutdown_connections 2024-03-19T01:13:26.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 -- 172.21.15.116:0/45153201 wait complete. 2024-03-19T01:13:26.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 -- start start 2024-03-19T01:13:26.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 Processor -- start 2024-03-19T01:13:26.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b90e100 0x564a3b91cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b91d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b913c60 0x564a3b91e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7af81a8010 con 0x564a3b90eaf0 2024-03-19T01:13:26.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7af81a7910 con 0x564a3b913c60 2024-03-19T01:13:26.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.138+0000 7f7b0016d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7af8071ea0 con 0x564a3b90e100 2024-03-19T01:13:26.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7b0016d740 10 client.0 _ll_register_callbacks cb 0x7f7af00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:13:26.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7af67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b91d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7af6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b90e100 0x564a3b91cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7af67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b91d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49414/0 (socket says 172.21.15.116:49414) 2024-03-19T01:13:26.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7af67fc640 1 -- 172.21.15.116:0/406506193 learned_addr learned my addr 172.21.15.116:0/406506193 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:26.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7af77fe640 1 --2- 172.21.15.116:0/406506193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b913c60 0x564a3b91e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7af67fc640 1 -- 172.21.15.116:0/406506193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b90e100 msgr2=0x564a3b91cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:26.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7af67fc640 1 --2- 172.21.15.116:0/406506193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b90e100 0x564a3b91cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7af67fc640 1 -- 172.21.15.116:0/406506193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b913c60 msgr2=0x564a3b91e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:26.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7af67fc640 1 --2- 172.21.15.116:0/406506193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b913c60 0x564a3b91e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7af67fc640 1 -- 172.21.15.116:0/406506193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a3b91e760 con 0x564a3b90eaf0 2024-03-19T01:13:26.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7af67fc640 1 --2- 172.21.15.116:0/406506193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b91d1d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7aec02c710 tx=0x7f7aec005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191daf64f355bd9c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:26.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7adaff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:13:26.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7adaff5640 1 -- 172.21.15.116:0/406506193 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aec05d060 con 0x564a3b90eaf0 2024-03-19T01:13:26.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7adaff5640 1 -- 172.21.15.116:0/406506193 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7aec0037f0 con 0x564a3b90eaf0 2024-03-19T01:13:26.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7b0016d740 1 -- 172.21.15.116:0/406506193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7af0029480 con 0x564a3b90eaf0 2024-03-19T01:13:26.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7adaff5640 1 -- 172.21.15.116:0/406506193 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7aec054040 con 0x564a3b90eaf0 2024-03-19T01:13:26.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.139+0000 7f7b0016d740 10 client.15411 fetch_fsmap learned FSMap version 30 2024-03-19T01:13:26.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.140+0000 7f7b0016d740 1 -- 172.21.15.116:0/406506193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x564a3b99b8a0 con 0x564a3b90eaf0 2024-03-19T01:13:26.155 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.140+0000 7f7adaff5640 1 -- 172.21.15.116:0/406506193 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7af0029480 con 0x564a3b90eaf0 2024-03-19T01:13:26.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.140+0000 7f7adaff5640 1 -- 172.21.15.116:0/406506193 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f7aec057070 con 0x564a3b90eaf0 2024-03-19T01:13:26.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.140+0000 7f7b0016d740 10 client.15411 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:13:26.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.140+0000 7f7b0016d740 10 client.15411 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:13:26.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.141+0000 7f7b0016d740 20 client.15411 populate_metadata read hostname 'smithi116' 2024-03-19T01:13:26.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.141+0000 7f7b0016d740 1 --2- 172.21.15.116:0/406506193 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564a3b99c210 0x564a3b99e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:26.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.141+0000 7f7b0016d740 4 client.15411 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:13:26.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.141+0000 7f7b0016d740 1 -- 172.21.15.116:0/406506193 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564a3b87a470 con 0x564a3b99c210 2024-03-19T01:13:26.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.141+0000 7f7af6ffd640 1 --2- 172.21.15.116:0/406506193 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564a3b99c210 0x564a3b99e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:26.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.142+0000 7f7adaff5640 10 client.15411 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:26.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.142+0000 7f7af6ffd640 1 --2- 172.21.15.116:0/406506193 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564a3b99c210 0x564a3b99e630 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b75b642644ad6fe6 server_cookie=58a0a1acd12ba484 in_seq=0 out_seq=0 2024-03-19T01:13:26.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.174+0000 7f7adaff5640 1 -- 172.21.15.116:0/406506193 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564a3b87a470 con 0x564a3b99c210 2024-03-19T01:13:26.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.174+0000 7f7adaff5640 10 client.15411 handle_command_reply: tid=0 2024-03-19T01:13:26.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.175+0000 7f7b0016d740 1 client.15411 shutdown 2024-03-19T01:13:26.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.175+0000 7f7b0016d740 1 -- 172.21.15.116:0/406506193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 msgr2=0x564a3b91d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:26.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.175+0000 7f7b0016d740 1 --2- 172.21.15.116:0/406506193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b91d1d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7aec02c710 tx=0x7f7aec005fd0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.175+0000 7f7b0016d740 1 -- 172.21.15.116:0/406506193 shutdown_connections 2024-03-19T01:13:26.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.175+0000 7f7b0016d740 1 --2- 172.21.15.116:0/406506193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564a3b913c60 0x564a3b91e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.175+0000 7f7b0016d740 1 --2- 172.21.15.116:0/406506193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564a3b90eaf0 0x564a3b91d1d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.175+0000 7f7b0016d740 1 --2- 172.21.15.116:0/406506193 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564a3b99c210 0x564a3b99e630 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.175+0000 7f7b0016d740 1 --2- 172.21.15.116:0/406506193 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564a3b90e100 0x564a3b91cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.175+0000 7f7b0016d740 1 -- 172.21.15.116:0/406506193 >> 172.21.15.116:0/406506193 conn(0x564a3b909670 msgr2=0x564a3b908220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.175+0000 7f7adaff5640 0 client.15411 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.175+0000 7f7b0016d740 1 -- 172.21.15.116:0/406506193 shutdown_connections 2024-03-19T01:13:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.175+0000 7f7b0016d740 1 -- 172.21.15.116:0/406506193 wait complete. 2024-03-19T01:13:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.175+0000 7f7b0016d740 20 client.15411 trim_cache size 0 max 16384 2024-03-19T01:13:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:13:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:13:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:13:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:13:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.176+0000 7f7afd57e640 1 -- 172.21.15.116:0/2978922044 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ae00a2920 msgr2=0x7f7ae00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.176+0000 7f7afd57e640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ae00a2920 0x7f7ae00a4de0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x564a3b90f350 tx=0x7f7af0009500 comp rx=0 tx=0).stop 2024-03-19T01:13:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.176+0000 7f7afd57e640 1 -- 172.21.15.116:0/2978922044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af8107e80 msgr2=0x7f7af819e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.176+0000 7f7afd57e640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af8107e80 0x7f7af819e0b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800e9f0 tx=0x7f7ae800eeb0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.176+0000 7f7afd57e640 1 -- 172.21.15.116:0/2978922044 shutdown_connections 2024-03-19T01:13:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.176+0000 7f7afd57e640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7af8109990 0x7f7af81a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.176+0000 7f7afd57e640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7ae00a2920 0x7f7ae00a4de0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.176+0000 7f7afd57e640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7af8108fd0 0x7f7af819e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.176+0000 7f7afd57e640 1 --2- 172.21.15.116:0/2978922044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7af8107e80 0x7f7af819e0b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.176+0000 7f7afd57e640 1 -- 172.21.15.116:0/2978922044 >> 172.21.15.116:0/2978922044 conn(0x7f7af8076e30 msgr2=0x7f7af810c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.176+0000 7f7afd57e640 1 -- 172.21.15.116:0/2978922044 shutdown_connections 2024-03-19T01:13:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:26.176+0000 7f7afd57e640 1 -- 172.21.15.116:0/2978922044 wait complete. 2024-03-19T01:13:26.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:26 smithi156 ceph-mon[66799]: pgmap v376: 97 pgs: 97 active+clean; 1011 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 848 op/s 2024-03-19T01:13:26.544 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 40.0 s, 3942.6 tps, lat 25.356 ms stddev 25.947 2024-03-19T01:13:26.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:26 smithi162 ceph-mon[67178]: pgmap v376: 97 pgs: 97 active+clean; 1011 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 848 op/s 2024-03-19T01:13:26.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:26 smithi116 ceph-mon[62701]: pgmap v376: 97 pgs: 97 active+clean; 1011 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 848 op/s 2024-03-19T01:13:28.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:28 smithi156 ceph-mon[66799]: pgmap v377: 97 pgs: 97 active+clean; 995 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 819 op/s 2024-03-19T01:13:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:28 smithi162 ceph-mon[67178]: pgmap v377: 97 pgs: 97 active+clean; 995 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 819 op/s 2024-03-19T01:13:28.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:28 smithi116 ceph-mon[62701]: pgmap v377: 97 pgs: 97 active+clean; 995 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 MiB/s wr, 819 op/s 2024-03-19T01:13:30.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:30 smithi156 ceph-mon[66799]: pgmap v378: 97 pgs: 97 active+clean; 995 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 941 op/s 2024-03-19T01:13:30.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:30 smithi162 ceph-mon[67178]: pgmap v378: 97 pgs: 97 active+clean; 995 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 941 op/s 2024-03-19T01:13:30.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:30 smithi116 ceph-mon[62701]: pgmap v378: 97 pgs: 97 active+clean; 995 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 941 op/s 2024-03-19T01:13:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 45.0 s, 3886.4 tps, lat 25.689 ms stddev 26.640 2024-03-19T01:13:32.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:32 smithi156 ceph-mon[66799]: pgmap v379: 97 pgs: 97 active+clean; 999 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 MiB/s wr, 876 op/s 2024-03-19T01:13:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:32 smithi162 ceph-mon[67178]: pgmap v379: 97 pgs: 97 active+clean; 999 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 MiB/s wr, 876 op/s 2024-03-19T01:13:32.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:32 smithi116 ceph-mon[62701]: pgmap v379: 97 pgs: 97 active+clean; 999 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 MiB/s wr, 876 op/s 2024-03-19T01:13:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:34 smithi156 ceph-mon[66799]: pgmap v380: 97 pgs: 97 active+clean; 1009 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 737 op/s 2024-03-19T01:13:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:34 smithi162 ceph-mon[67178]: pgmap v380: 97 pgs: 97 active+clean; 1009 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 737 op/s 2024-03-19T01:13:34.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:34 smithi116 ceph-mon[62701]: pgmap v380: 97 pgs: 97 active+clean; 1009 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 737 op/s 2024-03-19T01:13:36.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:36 smithi156 ceph-mon[66799]: pgmap v381: 97 pgs: 97 active+clean; 1013 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 15 MiB/s wr, 905 op/s 2024-03-19T01:13:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 50.0 s, 3547.2 tps, lat 28.178 ms stddev 28.642 2024-03-19T01:13:36.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:36 smithi162 ceph-mon[67178]: pgmap v381: 97 pgs: 97 active+clean; 1013 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 15 MiB/s wr, 905 op/s 2024-03-19T01:13:36.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:36 smithi116 ceph-mon[62701]: pgmap v381: 97 pgs: 97 active+clean; 1013 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 15 MiB/s wr, 905 op/s 2024-03-19T01:13:38.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:38 smithi156 ceph-mon[66799]: pgmap v382: 97 pgs: 97 active+clean; 1013 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 829 op/s 2024-03-19T01:13:38.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:38 smithi162 ceph-mon[67178]: pgmap v382: 97 pgs: 97 active+clean; 1013 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 829 op/s 2024-03-19T01:13:38.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:38 smithi116 ceph-mon[62701]: pgmap v382: 97 pgs: 97 active+clean; 1013 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 829 op/s 2024-03-19T01:13:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.312+0000 7fc3d627e640 1 -- 172.21.15.116:0/2425868995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0101950 msgr2=0x7fc3d0101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.312+0000 7fc3d627e640 1 --2- 172.21.15.116:0/2425868995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0101950 0x7fc3d0101dd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009900 tx=0x7fc3b802bc40 comp rx=0 tx=0).stop 2024-03-19T01:13:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.312+0000 7fc3d627e640 1 -- 172.21.15.116:0/2425868995 shutdown_connections 2024-03-19T01:13:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.312+0000 7fc3d627e640 1 --2- 172.21.15.116:0/2425868995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d0102310 0x7fc3d0108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.312+0000 7fc3d627e640 1 --2- 172.21.15.116:0/2425868995 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0101950 0x7fc3d0101dd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.312+0000 7fc3d627e640 1 --2- 172.21.15.116:0/2425868995 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3d0100760 0x7fc3d0100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.312+0000 7fc3d627e640 1 -- 172.21.15.116:0/2425868995 >> 172.21.15.116:0/2425868995 conn(0x7fc3d00fbcf0 msgr2=0x7fc3d00fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.312+0000 7fc3d627e640 1 -- 172.21.15.116:0/2425868995 shutdown_connections 2024-03-19T01:13:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.312+0000 7fc3d627e640 1 -- 172.21.15.116:0/2425868995 wait complete. 2024-03-19T01:13:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.313+0000 7fc3d627e640 1 Processor -- start 2024-03-19T01:13:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.313+0000 7fc3d627e640 1 -- start start 2024-03-19T01:13:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.313+0000 7fc3d627e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3d0100760 0x7fc3d01958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.313+0000 7fc3d627e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0101950 0x7fc3d0195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.313+0000 7fc3d627e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d0102310 0x7fc3d019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.313+0000 7fc3d627e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d010b390 con 0x7fc3d0102310 2024-03-19T01:13:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.313+0000 7fc3d627e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d010b210 con 0x7fc3d0101950 2024-03-19T01:13:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.313+0000 7fc3d627e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d010b510 con 0x7fc3d0100760 2024-03-19T01:13:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.314+0000 7fc3cf7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0101950 0x7fc3d0195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.314+0000 7fc3cffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3d0100760 0x7fc3d01958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.314+0000 7fc3cf7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0101950 0x7fc3d0195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:41062/0 (socket says 172.21.15.116:41062) 2024-03-19T01:13:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.314+0000 7fc3cf7fe640 1 -- 172.21.15.116:0/506455286 learned_addr learned my addr 172.21.15.116:0/506455286 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.314+0000 7fc3d481b640 1 --2- 172.21.15.116:0/506455286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d0102310 0x7fc3d019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.314+0000 7fc3cffff640 1 -- 172.21.15.116:0/506455286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0101950 msgr2=0x7fc3d0195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.314+0000 7fc3cffff640 1 --2- 172.21.15.116:0/506455286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0101950 0x7fc3d0195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.314+0000 7fc3cffff640 1 -- 172.21.15.116:0/506455286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d0102310 msgr2=0x7fc3d019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.314+0000 7fc3cffff640 1 --2- 172.21.15.116:0/506455286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d0102310 0x7fc3d019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.314+0000 7fc3cffff640 1 -- 172.21.15.116:0/506455286 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d019d510 con 0x7fc3d0100760 2024-03-19T01:13:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.315+0000 7fc3cffff640 1 --2- 172.21.15.116:0/506455286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3d0100760 0x7fc3d01958f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000b3c0 tx=0x7fc3c000b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.315+0000 7fc3cd7fa640 1 -- 172.21.15.116:0/506455286 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0011830 con 0x7fc3d0100760 2024-03-19T01:13:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.315+0000 7fc3cd7fa640 1 -- 172.21.15.116:0/506455286 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3c0016070 con 0x7fc3d0100760 2024-03-19T01:13:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.315+0000 7fc3cd7fa640 1 -- 172.21.15.116:0/506455286 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c000f2e0 con 0x7fc3d0100760 2024-03-19T01:13:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.315+0000 7fc3d627e640 1 -- 172.21.15.116:0/506455286 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3d019d800 con 0x7fc3d0100760 2024-03-19T01:13:39.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.315+0000 7fc3d627e640 1 -- 172.21.15.116:0/506455286 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3d019dd00 con 0x7fc3d0100760 2024-03-19T01:13:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.319+0000 7fc3cd7fa640 1 -- 172.21.15.116:0/506455286 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fc3c00119d0 con 0x7fc3d0100760 2024-03-19T01:13:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3d8e69740 1 Processor -- start 2024-03-19T01:13:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3d8e69740 1 -- start start 2024-03-19T01:13:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3d8e69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4786910 0x559ff4786d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3d8e69740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4787300 0x559ff478be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3d8e69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff478c470 0x559ff478e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3d8e69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559ff46e2be0 con 0x559ff4786910 2024-03-19T01:13:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3d8e69740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559ff468b960 con 0x559ff4787300 2024-03-19T01:13:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3d8e69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559ff46dec80 con 0x559ff478c470 2024-03-19T01:13:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3cd7fa640 1 --2- 172.21.15.116:0/506455286 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc3bc0a2a60 0x7fc3bc0a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3cf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff478c470 0x559ff478e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3cffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4787300 0x559ff478be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3cf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff478c470 0x559ff478e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45176/0 (socket says 172.21.15.116:45176) 2024-03-19T01:13:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3d481b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4786910 0x559ff4786d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3cf7fe640 1 -- 172.21.15.116:0/4160801136 learned_addr learned my addr 172.21.15.116:0/4160801136 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3d481b640 1 --2- 172.21.15.116:0/506455286 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc3bc0a2a60 0x7fc3bc0a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3cd7fa640 1 -- 172.21.15.116:0/506455286 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc3c00cdee0 con 0x7fc3d0100760 2024-03-19T01:13:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3cf7fe640 1 -- 172.21.15.116:0/4160801136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4787300 msgr2=0x559ff478be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3cf7fe640 1 --2- 172.21.15.116:0/4160801136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4787300 0x559ff478be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3cf7fe640 1 -- 172.21.15.116:0/4160801136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4786910 msgr2=0x559ff4786d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.320+0000 7fc3cf7fe640 1 --2- 172.21.15.116:0/4160801136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4786910 0x559ff4786d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.321+0000 7fc3cf7fe640 1 -- 172.21.15.116:0/4160801136 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff4682d20 con 0x559ff478c470 2024-03-19T01:13:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.321+0000 7fc3d481b640 1 --2- 172.21.15.116:0/506455286 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc3bc0a2a60 0x7fc3bc0a4f20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x559ff4787b60 tx=0x7fc3c8009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.321+0000 7fc3cf7fe640 1 --2- 172.21.15.116:0/4160801136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff478c470 0x559ff478e900 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8036040 tx=0x7fc3b8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb393f696a021e3b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.321+0000 7fc3c67fc640 1 -- 172.21.15.116:0/4160801136 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8042070 con 0x559ff478c470 2024-03-19T01:13:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.321+0000 7fc3c67fc640 1 -- 172.21.15.116:0/4160801136 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3b802cb90 con 0x559ff478c470 2024-03-19T01:13:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.321+0000 7fc3c67fc640 1 -- 172.21.15.116:0/4160801136 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8004920 con 0x559ff478c470 2024-03-19T01:13:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.322+0000 7fc3d8e69740 1 -- 172.21.15.116:0/4160801136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff478c470 msgr2=0x559ff478e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.322+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/4160801136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff478c470 0x559ff478e900 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8036040 tx=0x7fc3b8000bc0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.322+0000 7fc3d8e69740 1 -- 172.21.15.116:0/4160801136 shutdown_connections 2024-03-19T01:13:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.322+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/4160801136 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff478c470 0x559ff478e900 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.322+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/4160801136 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4787300 0x559ff478be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.322+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/4160801136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4786910 0x559ff4786d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.322+0000 7fc3d8e69740 1 -- 172.21.15.116:0/4160801136 >> 172.21.15.116:0/4160801136 conn(0x559ff4779670 msgr2=0x559ff4779a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.322+0000 7fc3d8e69740 1 -- 172.21.15.116:0/4160801136 shutdown_connections 2024-03-19T01:13:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.322+0000 7fc3d8e69740 1 -- 172.21.15.116:0/4160801136 wait complete. 2024-03-19T01:13:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.322+0000 7fc3d8e69740 1 -- start start 2024-03-19T01:13:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3d8e69740 1 Processor -- start 2024-03-19T01:13:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3d8e69740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4786910 0x559ff4790910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3d8e69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4787300 0x559ff4790e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3d8e69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff478c470 0x559ff4797e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3d8e69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559ff478f3c0 con 0x559ff4787300 2024-03-19T01:13:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3d8e69740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559ff469edb0 con 0x559ff4786910 2024-03-19T01:13:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3d8e69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559ff478f540 con 0x559ff478c470 2024-03-19T01:13:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3d8e69740 10 client.0 _ll_register_callbacks cb 0x559ff46a3e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:13:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3cffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4786910 0x559ff4790910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3cffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4786910 0x559ff4790910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:41084/0 (socket says 172.21.15.116:41084) 2024-03-19T01:13:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3cffff640 1 -- 172.21.15.116:0/547454882 learned_addr learned my addr 172.21.15.116:0/547454882 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3d481b640 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff478c470 0x559ff4797e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3cf7fe640 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4787300 0x559ff4790e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3cffff640 1 -- 172.21.15.116:0/547454882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff478c470 msgr2=0x559ff4797e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3cffff640 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff478c470 0x559ff4797e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3cffff640 1 -- 172.21.15.116:0/547454882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4787300 msgr2=0x559ff4790e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.323+0000 7fc3cffff640 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4787300 0x559ff4790e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.324+0000 7fc3cffff640 1 -- 172.21.15.116:0/547454882 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ff469dc60 con 0x559ff4786910 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.324+0000 7fc3cf7fe640 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4787300 0x559ff4790e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.324+0000 7fc3cffff640 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4786910 0x559ff4790910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000bd80 tx=0x7fc3c0010450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a686a2350809e91 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.324+0000 7fc3af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.324+0000 7fc3af7fe640 1 -- 172.21.15.116:0/547454882 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c009af60 con 0x559ff4786910 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.324+0000 7fc3af7fe640 1 -- 172.21.15.116:0/547454882 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc3c0011cc0 con 0x559ff4786910 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.324+0000 7fc3d8e69740 1 -- 172.21.15.116:0/547454882 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3b802cb90 con 0x559ff4786910 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.324+0000 7fc3af7fe640 1 -- 172.21.15.116:0/547454882 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc3c009a860 con 0x559ff4786910 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.324+0000 7fc3d8e69740 10 client.25259 fetch_fsmap learned FSMap version 30 2024-03-19T01:13:39.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.324+0000 7fc3d8e69740 1 -- 172.21.15.116:0/547454882 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559ff43f7210 con 0x559ff4786910 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.324+0000 7fc3af7fe640 1 -- 172.21.15.116:0/547454882 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3b802cb90 con 0x559ff4786910 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.325+0000 7fc3af7fe640 1 -- 172.21.15.116:0/547454882 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7fc3c007ebc0 con 0x559ff4786910 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.325+0000 7fc3d8e69740 10 client.25259 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.325+0000 7fc3d8e69740 10 client.25259 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.325+0000 7fc3d8e69740 20 client.25259 populate_metadata read hostname 'smithi116' 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.325+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559ff479cac0 0x559ff479ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.325+0000 7fc3d8e69740 4 client.25259 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.325+0000 7fc3d8e69740 1 -- 172.21.15.116:0/547454882 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ff46f0120 con 0x559ff479cac0 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.325+0000 7fc3cf7fe640 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559ff479cac0 0x559ff479ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.325+0000 7fc3af7fe640 10 client.25259 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.325+0000 7fc3cf7fe640 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559ff479cac0 0x559ff479ef80 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dee26e08cbc5deb7 server_cookie=f2dcf64945ea9900 in_seq=0 out_seq=0 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3af7fe640 1 -- 172.21.15.116:0/547454882 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fc3b80539c0 con 0x559ff479cac0 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3af7fe640 10 client.25259 handle_command_reply: tid=0 2024-03-19T01:13:39.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3d8e69740 1 client.25259 shutdown 2024-03-19T01:13:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3d8e69740 1 -- 172.21.15.116:0/547454882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4786910 msgr2=0x559ff4790910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4786910 0x559ff4790910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000bd80 tx=0x7fc3c0010450 comp rx=0 tx=0).stop 2024-03-19T01:13:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3d8e69740 1 -- 172.21.15.116:0/547454882 shutdown_connections 2024-03-19T01:13:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559ff479cac0 0x559ff479ef80 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff478c470 0x559ff4797e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4787300 0x559ff4790e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/547454882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4786910 0x559ff4790910 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3d8e69740 1 -- 172.21.15.116:0/547454882 >> 172.21.15.116:0/547454882 conn(0x559ff4779670 msgr2=0x559ff477a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3af7fe640 0 client.25259 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3d8e69740 1 -- 172.21.15.116:0/547454882 shutdown_connections 2024-03-19T01:13:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3d8e69740 1 -- 172.21.15.116:0/547454882 wait complete. 2024-03-19T01:13:39.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.326+0000 7fc3d8e69740 20 client.25259 trim_cache size 0 max 16384 2024-03-19T01:13:39.348 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.346+0000 7fc3d8e69740 1 Processor -- start 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3d8e69740 1 -- start start 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3d8e69740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4786910 0x559ff4790db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3d8e69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4787300 0x559ff4797e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3d8e69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff478c470 0x559ff477cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3d8e69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d01a6ac0 con 0x559ff478c470 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3d8e69740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d01a8750 con 0x559ff4786910 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3d8e69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d01a7190 con 0x559ff4787300 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3cffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4786910 0x559ff4790db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3cf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4787300 0x559ff4797e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3d481b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff478c470 0x559ff477cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3cffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4786910 0x559ff4790db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:41096/0 (socket says 172.21.15.116:41096) 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3cf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4787300 0x559ff4797e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45196/0 (socket says 172.21.15.116:45196) 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3cffff640 1 -- 172.21.15.116:0/4272726362 learned_addr learned my addr 172.21.15.116:0/4272726362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:39.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3cf7fe640 1 -- 172.21.15.116:0/4272726362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4786910 msgr2=0x559ff4790db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3cf7fe640 1 --2- 172.21.15.116:0/4272726362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4786910 0x559ff4790db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3cf7fe640 1 -- 172.21.15.116:0/4272726362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff478c470 msgr2=0x559ff477cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3cf7fe640 1 --2- 172.21.15.116:0/4272726362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff478c470 0x559ff477cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.347+0000 7fc3cf7fe640 1 -- 172.21.15.116:0/4272726362 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ff44c3620 con 0x559ff4787300 2024-03-19T01:13:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.348+0000 7fc3cf7fe640 1 --2- 172.21.15.116:0/4272726362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4787300 0x559ff4797e50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80096c0 tx=0x7fc3b802bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fbf5e94dab85ac2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.348+0000 7fc3c5ffb640 1 -- 172.21.15.116:0/4272726362 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b804c040 con 0x559ff4787300 2024-03-19T01:13:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.348+0000 7fc3c5ffb640 1 -- 172.21.15.116:0/4272726362 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3b804e040 con 0x559ff4787300 2024-03-19T01:13:39.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.348+0000 7fc3c5ffb640 1 -- 172.21.15.116:0/4272726362 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8042040 con 0x559ff4787300 2024-03-19T01:13:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.349+0000 7fc3d8e69740 1 -- 172.21.15.116:0/4272726362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4787300 msgr2=0x559ff4797e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.349+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/4272726362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4787300 0x559ff4797e50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80096c0 tx=0x7fc3b802bc40 comp rx=0 tx=0).stop 2024-03-19T01:13:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.349+0000 7fc3d8e69740 1 -- 172.21.15.116:0/4272726362 shutdown_connections 2024-03-19T01:13:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.349+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/4272726362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff478c470 0x559ff477cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.349+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/4272726362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4787300 0x559ff4797e50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.349+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/4272726362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff4786910 0x559ff4790db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.349+0000 7fc3d8e69740 1 -- 172.21.15.116:0/4272726362 >> 172.21.15.116:0/4272726362 conn(0x559ff4779670 msgr2=0x559ff477c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.349+0000 7fc3d8e69740 1 -- 172.21.15.116:0/4272726362 shutdown_connections 2024-03-19T01:13:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.349+0000 7fc3d8e69740 1 -- 172.21.15.116:0/4272726362 wait complete. 2024-03-19T01:13:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.349+0000 7fc3d8e69740 1 -- start start 2024-03-19T01:13:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.349+0000 7fc3d8e69740 1 Processor -- start 2024-03-19T01:13:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3d8e69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4786910 0x559ff4795b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3d8e69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4787300 0x559ff4796050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3d8e69740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff478c470 0x559ff4797060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3d8e69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d01a7fc0 con 0x559ff4787300 2024-03-19T01:13:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3d8e69740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d01a78c0 con 0x559ff478c470 2024-03-19T01:13:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3d8e69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d01a4aa0 con 0x559ff4786910 2024-03-19T01:13:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3cf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4787300 0x559ff4796050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3cffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4786910 0x559ff4795b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3d8e69740 10 client.0 _ll_register_callbacks cb 0x7fc3c000bf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:13:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3cffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4786910 0x559ff4795b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45198/0 (socket says 172.21.15.116:45198) 2024-03-19T01:13:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3cffff640 1 -- 172.21.15.116:0/163514448 learned_addr learned my addr 172.21.15.116:0/163514448 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3d481b640 1 --2- 172.21.15.116:0/163514448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff478c470 0x559ff4797060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3cffff640 1 -- 172.21.15.116:0/163514448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff478c470 msgr2=0x559ff4797060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3cffff640 1 --2- 172.21.15.116:0/163514448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff478c470 0x559ff4797060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3cffff640 1 -- 172.21.15.116:0/163514448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4787300 msgr2=0x559ff4796050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3cffff640 1 --2- 172.21.15.116:0/163514448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4787300 0x559ff4796050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.350+0000 7fc3cffff640 1 -- 172.21.15.116:0/163514448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ff4797650 con 0x559ff4786910 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.351+0000 7fc3cffff640 1 --2- 172.21.15.116:0/163514448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4786910 0x559ff4795b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000bd80 tx=0x7fc3c000d0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=413b2b102224ec22 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.351+0000 7fc3af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.351+0000 7fc3af7fe640 1 -- 172.21.15.116:0/163514448 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c009af60 con 0x559ff4786910 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.351+0000 7fc3af7fe640 1 -- 172.21.15.116:0/163514448 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc3c0099cb0 con 0x559ff4786910 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.351+0000 7fc3d8e69740 1 -- 172.21.15.116:0/163514448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3b804e040 con 0x559ff4786910 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.352+0000 7fc3af7fe640 1 -- 172.21.15.116:0/163514448 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fc3c0009180 con 0x559ff4786910 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.352+0000 7fc3d8e69740 10 client.25234 fetch_fsmap learned FSMap version 30 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.352+0000 7fc3d8e69740 1 -- 172.21.15.116:0/163514448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559ff4803490 con 0x559ff4786910 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.352+0000 7fc3af7fe640 1 -- 172.21.15.116:0/163514448 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3b804e040 con 0x559ff4786910 2024-03-19T01:13:39.355 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.352+0000 7fc3af7fe640 1 -- 172.21.15.116:0/163514448 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7fc3c0099720 con 0x559ff4786910 2024-03-19T01:13:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.352+0000 7fc3d8e69740 10 client.25234 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:13:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.352+0000 7fc3d8e69740 10 client.25234 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:13:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.352+0000 7fc3d8e69740 20 client.25234 populate_metadata read hostname 'smithi116' 2024-03-19T01:13:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.352+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/163514448 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559ff4803f70 0x559ff4806390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:39.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.352+0000 7fc3d8e69740 4 client.25234 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:13:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.352+0000 7fc3d8e69740 1 -- 172.21.15.116:0/163514448 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559ff46ea470 con 0x559ff4803f70 2024-03-19T01:13:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.353+0000 7fc3cf7fe640 1 --2- 172.21.15.116:0/163514448 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559ff4803f70 0x559ff4806390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.353+0000 7fc3cf7fe640 1 --2- 172.21.15.116:0/163514448 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559ff4803f70 0x559ff4806390 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3186e7cd599de001 server_cookie=fcc51588eb012a04 in_seq=0 out_seq=0 2024-03-19T01:13:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.353+0000 7fc3af7fe640 10 client.25234 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:39.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.375+0000 7fc3af7fe640 1 -- 172.21.15.116:0/163514448 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559ff46ea470 con 0x559ff4803f70 2024-03-19T01:13:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.375+0000 7fc3af7fe640 10 client.25234 handle_command_reply: tid=0 2024-03-19T01:13:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.375+0000 7fc3d8e69740 1 client.25234 shutdown 2024-03-19T01:13:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.375+0000 7fc3d8e69740 1 -- 172.21.15.116:0/163514448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4786910 msgr2=0x559ff4795b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.375+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/163514448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4786910 0x559ff4795b10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000bd80 tx=0x7fc3c000d0c0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.375+0000 7fc3d8e69740 1 -- 172.21.15.116:0/163514448 shutdown_connections 2024-03-19T01:13:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.375+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/163514448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559ff478c470 0x559ff4797060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.375+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/163514448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559ff4787300 0x559ff4796050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.375+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/163514448 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559ff4803f70 0x559ff4806390 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.375+0000 7fc3d8e69740 1 --2- 172.21.15.116:0/163514448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559ff4786910 0x559ff4795b10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.375+0000 7fc3af7fe640 0 client.25234 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.375+0000 7fc3d8e69740 1 -- 172.21.15.116:0/163514448 >> 172.21.15.116:0/163514448 conn(0x559ff4779670 msgr2=0x559ff477d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.376+0000 7fc3d8e69740 1 -- 172.21.15.116:0/163514448 shutdown_connections 2024-03-19T01:13:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.376+0000 7fc3d8e69740 1 -- 172.21.15.116:0/163514448 wait complete. 2024-03-19T01:13:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.376+0000 7fc3d8e69740 20 client.25234 trim_cache size 0 max 16384 2024-03-19T01:13:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:13:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:13:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:13:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:13:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.377+0000 7fc3d627e640 1 -- 172.21.15.116:0/506455286 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc3bc0a2a60 msgr2=0x7fc3bc0a4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.377+0000 7fc3d627e640 1 --2- 172.21.15.116:0/506455286 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc3bc0a2a60 0x7fc3bc0a4f20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x559ff4787b60 tx=0x7fc3c8009500 comp rx=0 tx=0).stop 2024-03-19T01:13:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.377+0000 7fc3d627e640 1 -- 172.21.15.116:0/506455286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3d0100760 msgr2=0x7fc3d01958f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.377+0000 7fc3d627e640 1 --2- 172.21.15.116:0/506455286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3d0100760 0x7fc3d01958f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc3c000b3c0 tx=0x7fc3c000b880 comp rx=0 tx=0).stop 2024-03-19T01:13:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.378+0000 7fc3d627e640 1 -- 172.21.15.116:0/506455286 shutdown_connections 2024-03-19T01:13:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.378+0000 7fc3d627e640 1 --2- 172.21.15.116:0/506455286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3d0102310 0x7fc3d019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.378+0000 7fc3d627e640 1 --2- 172.21.15.116:0/506455286 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc3bc0a2a60 0x7fc3bc0a4f20 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.378+0000 7fc3d627e640 1 --2- 172.21.15.116:0/506455286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3d0101950 0x7fc3d0195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.378+0000 7fc3d627e640 1 --2- 172.21.15.116:0/506455286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc3d0100760 0x7fc3d01958f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:39.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.378+0000 7fc3d627e640 1 -- 172.21.15.116:0/506455286 >> 172.21.15.116:0/506455286 conn(0x7fc3d00fbcf0 msgr2=0x7fc3d0104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:39.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.378+0000 7fc3d627e640 1 -- 172.21.15.116:0/506455286 shutdown_connections 2024-03-19T01:13:39.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:39.378+0000 7fc3d627e640 1 -- 172.21.15.116:0/506455286 wait complete. 2024-03-19T01:13:40.451 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:40 smithi162 ceph-mon[67178]: pgmap v383: 97 pgs: 97 active+clean; 1014 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.5 MiB/s wr, 752 op/s 2024-03-19T01:13:40.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:40 smithi156 ceph-mon[66799]: pgmap v383: 97 pgs: 97 active+clean; 1014 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.5 MiB/s wr, 752 op/s 2024-03-19T01:13:40.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:40 smithi116 ceph-mon[62701]: pgmap v383: 97 pgs: 97 active+clean; 1014 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.5 MiB/s wr, 752 op/s 2024-03-19T01:13:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 55.0 s, 4106.6 tps, lat 24.364 ms stddev 26.045 2024-03-19T01:13:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:42 smithi156 ceph-mon[66799]: pgmap v384: 97 pgs: 97 active+clean; 1018 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 720 op/s 2024-03-19T01:13:42.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:42 smithi162 ceph-mon[67178]: pgmap v384: 97 pgs: 97 active+clean; 1018 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 720 op/s 2024-03-19T01:13:42.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:42 smithi116 ceph-mon[62701]: pgmap v384: 97 pgs: 97 active+clean; 1018 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 720 op/s 2024-03-19T01:13:44.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:44 smithi156 ceph-mon[66799]: pgmap v385: 97 pgs: 97 active+clean; 1024 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.6 MiB/s wr, 700 op/s 2024-03-19T01:13:44.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:44 smithi162 ceph-mon[67178]: pgmap v385: 97 pgs: 97 active+clean; 1024 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.6 MiB/s wr, 700 op/s 2024-03-19T01:13:44.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:44 smithi116 ceph-mon[62701]: pgmap v385: 97 pgs: 97 active+clean; 1024 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 8.6 MiB/s wr, 700 op/s 2024-03-19T01:13:45.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:45 smithi156 ceph-mon[66799]: pgmap v386: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 9.8 MiB/s wr, 833 op/s 2024-03-19T01:13:45.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:45 smithi162 ceph-mon[67178]: pgmap v386: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 9.8 MiB/s wr, 833 op/s 2024-03-19T01:13:45.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:45 smithi116 ceph-mon[62701]: pgmap v386: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 9.8 MiB/s wr, 833 op/s 2024-03-19T01:13:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 60.0 s, 3355.4 tps, lat 29.818 ms stddev 29.953 2024-03-19T01:13:48.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:48 smithi156 ceph-mon[66799]: pgmap v387: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 8.8 MiB/s wr, 756 op/s 2024-03-19T01:13:48.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:48 smithi162 ceph-mon[67178]: pgmap v387: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 8.8 MiB/s wr, 756 op/s 2024-03-19T01:13:48.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:48 smithi116 ceph-mon[62701]: pgmap v387: 97 pgs: 97 active+clean; 1.0 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 8.8 MiB/s wr, 756 op/s 2024-03-19T01:13:50.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:50 smithi156 ceph-mon[66799]: pgmap v388: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 9.5 MiB/s wr, 793 op/s 2024-03-19T01:13:50.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:50 smithi162 ceph-mon[67178]: pgmap v388: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 9.5 MiB/s wr, 793 op/s 2024-03-19T01:13:50.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:50 smithi116 ceph-mon[62701]: pgmap v388: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 9.5 MiB/s wr, 793 op/s 2024-03-19T01:13:51.544 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 65.0 s, 3265.0 tps, lat 30.553 ms stddev 33.147 2024-03-19T01:13:52.457 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:52 smithi116 ceph-mon[62701]: pgmap v389: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 11 MiB/s wr, 837 op/s 2024-03-19T01:13:52.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:52 smithi156 ceph-mon[66799]: pgmap v389: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 11 MiB/s wr, 837 op/s 2024-03-19T01:13:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.507+0000 7f39095a9640 1 -- 172.21.15.116:0/1770148098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3904102a70 msgr2=0x7f3904102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.507+0000 7f39095a9640 1 --2- 172.21.15.116:0/1770148098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3904102a70 0x7f3904102ef0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009ca0 tx=0x7f38fc02c0f0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.507+0000 7f39095a9640 1 -- 172.21.15.116:0/1770148098 shutdown_connections 2024-03-19T01:13:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.507+0000 7f39095a9640 1 --2- 172.21.15.116:0/1770148098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3904103430 0x7f3904109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.507+0000 7f39095a9640 1 --2- 172.21.15.116:0/1770148098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3904102a70 0x7f3904102ef0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.507+0000 7f39095a9640 1 --2- 172.21.15.116:0/1770148098 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3904101880 0x7f3904101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.507+0000 7f39095a9640 1 -- 172.21.15.116:0/1770148098 >> 172.21.15.116:0/1770148098 conn(0x7f39040fcdf0 msgr2=0x7f39040ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:52.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.507+0000 7f39095a9640 1 -- 172.21.15.116:0/1770148098 shutdown_connections 2024-03-19T01:13:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.507+0000 7f39095a9640 1 -- 172.21.15.116:0/1770148098 wait complete. 2024-03-19T01:13:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.507+0000 7f39095a9640 1 Processor -- start 2024-03-19T01:13:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39095a9640 1 -- start start 2024-03-19T01:13:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39095a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3904101880 0x7f3904197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39095a9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3904102a70 0x7f3904198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39095a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3904103430 0x7f390419f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39095a9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f390410d500 con 0x7f3904103430 2024-03-19T01:13:52.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39095a9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f390410d380 con 0x7f3904102a70 2024-03-19T01:13:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39095a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f390410d680 con 0x7f3904101880 2024-03-19T01:13:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3904103430 0x7f390419f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39027fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3904102a70 0x7f3904198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f3902ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3904101880 0x7f3904197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39027fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3904102a70 0x7f3904198020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:57460/0 (socket says 172.21.15.116:57460) 2024-03-19T01:13:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3904103430 0x7f390419f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49840/0 (socket says 172.21.15.116:49840) 2024-03-19T01:13:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39027fc640 1 -- 172.21.15.116:0/834476324 learned_addr learned my addr 172.21.15.116:0/834476324 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39027fc640 1 -- 172.21.15.116:0/834476324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3904101880 msgr2=0x7f3904197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39027fc640 1 --2- 172.21.15.116:0/834476324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3904101880 0x7f3904197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39027fc640 1 -- 172.21.15.116:0/834476324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3904103430 msgr2=0x7f390419f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39027fc640 1 --2- 172.21.15.116:0/834476324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3904103430 0x7f390419f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.512 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.508+0000 7f39027fc640 1 -- 172.21.15.116:0/834476324 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3904073b50 con 0x7f3904102a70 2024-03-19T01:13:52.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.509+0000 7f39027fc640 1 --2- 172.21.15.116:0/834476324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3904102a70 0x7f3904198020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009dc0 tx=0x7f38fc032f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.509+0000 7f38ebfff640 1 -- 172.21.15.116:0/834476324 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc041020 con 0x7f3904102a70 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.509+0000 7f38ebfff640 1 -- 172.21.15.116:0/834476324 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38fc034cd0 con 0x7f3904102a70 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.509+0000 7f38ebfff640 1 -- 172.21.15.116:0/834476324 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc039040 con 0x7f3904102a70 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.509+0000 7f39095a9640 1 -- 172.21.15.116:0/834476324 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3904073de0 con 0x7f3904102a70 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.509+0000 7f39095a9640 1 -- 172.21.15.116:0/834476324 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3904074280 con 0x7f3904102a70 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.512+0000 7f38ebfff640 1 -- 172.21.15.116:0/834476324 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f38fc044020 con 0x7f3904102a70 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.512+0000 7f390c191740 1 Processor -- start 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f390c191740 1 -- start start 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f390c191740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054100 0x555ebb054520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f390c191740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054af0 0x555ebb0595f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f390c191740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb05c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f390c191740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555ebafb8be0 con 0x555ebb054100 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f390c191740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555ebaf61960 con 0x555ebb054af0 2024-03-19T01:13:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f390c191740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555ebafb4c80 con 0x555ebb059c60 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f38ebfff640 1 --2- 172.21.15.116:0/834476324 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f38ec0a2920 0x7f38ec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f39027fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb05c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f39037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054100 0x555ebb054520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f39037fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054100 0x555ebb054520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49850/0 (socket says 172.21.15.116:49850) 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f39027fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb05c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37096/0 (socket says 172.21.15.116:37096) 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f39037fe640 1 -- 172.21.15.116:0/3509529645 learned_addr learned my addr 172.21.15.116:0/3509529645 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f39037fe640 1 --2- 172.21.15.116:0/834476324 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f38ec0a2920 0x7f38ec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f39027fc640 1 -- 172.21.15.116:0/3509529645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054af0 msgr2=0x555ebb0595f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f39027fc640 1 --2- 172.21.15.116:0/3509529645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054af0 0x555ebb0595f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f39027fc640 1 -- 172.21.15.116:0/3509529645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054100 msgr2=0x555ebb054520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f39027fc640 1 --2- 172.21.15.116:0/3509529645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054100 0x555ebb054520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f39027fc640 1 -- 172.21.15.116:0/3509529645 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ebaf58d20 con 0x555ebb059c60 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.513+0000 7f39037fe640 1 --2- 172.21.15.116:0/3509529645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054100 0x555ebb054520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:13:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.514+0000 7f38ebfff640 1 -- 172.21.15.116:0/834476324 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f38fc0f0830 con 0x7f3904102a70 2024-03-19T01:13:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.514+0000 7f39027fc640 1 --2- 172.21.15.116:0/3509529645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb05c0f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0ee360 tx=0x7f38fc04d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e022c9aa974442d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.514+0000 7f39037fe640 1 --2- 172.21.15.116:0/834476324 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f38ec0a2920 0x7f38ec0a4de0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x555ebb055350 tx=0x7f38f4004730 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.514+0000 7f38e97fa640 1 -- 172.21.15.116:0/3509529645 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc034920 con 0x555ebb059c60 2024-03-19T01:13:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.514+0000 7f38e97fa640 1 -- 172.21.15.116:0/3509529645 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38fc0bd8b0 con 0x555ebb059c60 2024-03-19T01:13:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.514+0000 7f38e97fa640 1 -- 172.21.15.116:0/3509529645 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0c1300 con 0x555ebb059c60 2024-03-19T01:13:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.515+0000 7f390c191740 1 -- 172.21.15.116:0/3509529645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 msgr2=0x555ebb05c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.515+0000 7f390c191740 1 --2- 172.21.15.116:0/3509529645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb05c0f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0ee360 tx=0x7f38fc04d000 comp rx=0 tx=0).stop 2024-03-19T01:13:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.516+0000 7f390c191740 1 -- 172.21.15.116:0/3509529645 shutdown_connections 2024-03-19T01:13:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.516+0000 7f390c191740 1 --2- 172.21.15.116:0/3509529645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb05c0f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.516+0000 7f390c191740 1 --2- 172.21.15.116:0/3509529645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054af0 0x555ebb0595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.516+0000 7f390c191740 1 --2- 172.21.15.116:0/3509529645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054100 0x555ebb054520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.516+0000 7f390c191740 1 -- 172.21.15.116:0/3509529645 >> 172.21.15.116:0/3509529645 conn(0x555ebb04f670 msgr2=0x555ebb04fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.516+0000 7f390c191740 1 -- 172.21.15.116:0/3509529645 shutdown_connections 2024-03-19T01:13:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.516+0000 7f390c191740 1 -- 172.21.15.116:0/3509529645 wait complete. 2024-03-19T01:13:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.517+0000 7f390c191740 1 -- start start 2024-03-19T01:13:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.517+0000 7f390c191740 1 Processor -- start 2024-03-19T01:13:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.517+0000 7f390c191740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054100 0x555ebb05dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.517+0000 7f390c191740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054af0 0x555ebb05e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.517+0000 7f390c191740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb0652c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.517+0000 7f390c191740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x555ebb05c9d0 con 0x555ebb054af0 2024-03-19T01:13:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.517+0000 7f390c191740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x555ebaf74db0 con 0x555ebb054100 2024-03-19T01:13:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.517+0000 7f390c191740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x555ebb05cb50 con 0x555ebb059c60 2024-03-19T01:13:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.517+0000 7f39037fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb0652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.517+0000 7f39037fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb0652c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37112/0 (socket says 172.21.15.116:37112) 2024-03-19T01:13:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.517+0000 7f39037fe640 1 -- 172.21.15.116:0/2353602828 learned_addr learned my addr 172.21.15.116:0/2353602828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.518+0000 7f3902ffd640 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054100 0x555ebb05dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.518+0000 7f390c191740 10 client.0 _ll_register_callbacks cb 0x555ebaf79e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:13:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.518+0000 7f39027fc640 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054af0 0x555ebb05e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.518+0000 7f39037fe640 1 -- 172.21.15.116:0/2353602828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054100 msgr2=0x555ebb05dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.518+0000 7f39037fe640 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054100 0x555ebb05dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.518+0000 7f39037fe640 1 -- 172.21.15.116:0/2353602828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054af0 msgr2=0x555ebb05e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.518+0000 7f39037fe640 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054af0 0x555ebb05e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.518+0000 7f39037fe640 1 -- 172.21.15.116:0/2353602828 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ebaf73c60 con 0x555ebb059c60 2024-03-19T01:13:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.518+0000 7f39027fc640 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054af0 0x555ebb05e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:13:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.518+0000 7f39037fe640 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb0652c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f38f4013840 tx=0x7f38f40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4938652506423d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.518+0000 7f3902ffd640 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054100 0x555ebb05dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:13:52.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.519+0000 7f38e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.519+0000 7f38e6ff5640 1 -- 172.21.15.116:0/2353602828 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f4026030 con 0x555ebb059c60 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.519+0000 7f390c191740 1 -- 172.21.15.116:0/2353602828 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38fc0bd8b0 con 0x555ebb059c60 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.519+0000 7f38e6ff5640 1 -- 172.21.15.116:0/2353602828 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f38f401c070 con 0x555ebb059c60 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.519+0000 7f38e6ff5640 1 -- 172.21.15.116:0/2353602828 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f38f4031070 con 0x555ebb059c60 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.519+0000 7f390c191740 10 client.25246 fetch_fsmap learned FSMap version 30 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.519+0000 7f390c191740 1 -- 172.21.15.116:0/2353602828 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555ebaccd210 con 0x555ebb059c60 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.519+0000 7f38e6ff5640 1 -- 172.21.15.116:0/2353602828 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38fc0bd8b0 con 0x555ebb059c60 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.520+0000 7f38e6ff5640 1 -- 172.21.15.116:0/2353602828 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f38f401f050 con 0x555ebb059c60 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.520+0000 7f390c191740 10 client.25246 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.520+0000 7f390c191740 10 client.25246 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.520+0000 7f390c191740 20 client.25246 populate_metadata read hostname 'smithi116' 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.520+0000 7f390c191740 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x555ebb069c80 0x555ebb06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.520+0000 7f390c191740 4 client.25246 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.520+0000 7f390c191740 1 -- 172.21.15.116:0/2353602828 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ebafc6120 con 0x555ebb069c80 2024-03-19T01:13:52.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.520+0000 7f3902ffd640 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x555ebb069c80 0x555ebb06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.520+0000 7f3902ffd640 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x555ebb069c80 0x555ebb06c110 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96aef47c15b085f3 server_cookie=c5670b0aca72eb82 in_seq=0 out_seq=0 2024-03-19T01:13:52.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.520+0000 7f38e6ff5640 10 client.25246 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f38e6ff5640 1 -- 172.21.15.116:0/2353602828 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555ebafc6120 con 0x555ebb069c80 2024-03-19T01:13:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f38e6ff5640 10 client.25246 handle_command_reply: tid=0 2024-03-19T01:13:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f390c191740 1 client.25246 shutdown 2024-03-19T01:13:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f390c191740 1 -- 172.21.15.116:0/2353602828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 msgr2=0x555ebb0652c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f390c191740 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb0652c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f38f4013840 tx=0x7f38f40046c0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f390c191740 1 -- 172.21.15.116:0/2353602828 shutdown_connections 2024-03-19T01:13:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f390c191740 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x555ebb069c80 0x555ebb06c110 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f390c191740 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb0652c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f390c191740 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054af0 0x555ebb05e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f390c191740 1 --2- 172.21.15.116:0/2353602828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054100 0x555ebb05dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f38e6ff5640 0 client.25246 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:52.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f390c191740 1 -- 172.21.15.116:0/2353602828 >> 172.21.15.116:0/2353602828 conn(0x555ebb04f670 msgr2=0x555ebb0508c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:52.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f390c191740 1 -- 172.21.15.116:0/2353602828 shutdown_connections 2024-03-19T01:13:52.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f390c191740 1 -- 172.21.15.116:0/2353602828 wait complete. 2024-03-19T01:13:52.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.521+0000 7f390c191740 20 client.25246 trim_cache size 0 max 16384 2024-03-19T01:13:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.544+0000 7f390c191740 1 Processor -- start 2024-03-19T01:13:52.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.544+0000 7f390c191740 1 -- start start 2024-03-19T01:13:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.544+0000 7f390c191740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054100 0x555ebb055590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.544+0000 7f390c191740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb054af0 0x555ebb065280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f390c191740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb059c60 0x555ebb055ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f390c191740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f39041ab020 con 0x555ebb059c60 2024-03-19T01:13:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f390c191740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f39041accb0 con 0x555ebb054100 2024-03-19T01:13:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f390c191740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f39041ab6f0 con 0x555ebb054af0 2024-03-19T01:13:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f39027fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb054af0 0x555ebb065280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f39027fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb054af0 0x555ebb065280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37118/0 (socket says 172.21.15.116:37118) 2024-03-19T01:13:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f39027fc640 1 -- 172.21.15.116:0/3942783295 learned_addr learned my addr 172.21.15.116:0/3942783295 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f39037fe640 1 --2- 172.21.15.116:0/3942783295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb059c60 0x555ebb055ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f3902ffd640 1 --2- 172.21.15.116:0/3942783295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054100 0x555ebb055590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f39027fc640 1 -- 172.21.15.116:0/3942783295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054100 msgr2=0x555ebb055590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f39027fc640 1 --2- 172.21.15.116:0/3942783295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054100 0x555ebb055590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f39027fc640 1 -- 172.21.15.116:0/3942783295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb059c60 msgr2=0x555ebb055ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f39027fc640 1 --2- 172.21.15.116:0/3942783295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb059c60 0x555ebb055ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f39027fc640 1 -- 172.21.15.116:0/3942783295 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ebad99620 con 0x555ebb054af0 2024-03-19T01:13:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.545+0000 7f39037fe640 1 --2- 172.21.15.116:0/3942783295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb059c60 0x555ebb055ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:13:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.546+0000 7f39027fc640 1 --2- 172.21.15.116:0/3942783295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb054af0 0x555ebb065280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f38fc032420 tx=0x7f38fc02c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26210e586fe6197e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.546+0000 7f38e8ff9640 1 -- 172.21.15.116:0/3942783295 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0b9450 con 0x555ebb054af0 2024-03-19T01:13:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.546+0000 7f38e8ff9640 1 -- 172.21.15.116:0/3942783295 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38fc034920 con 0x555ebb054af0 2024-03-19T01:13:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.546+0000 7f38e8ff9640 1 -- 172.21.15.116:0/3942783295 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0c1300 con 0x555ebb054af0 2024-03-19T01:13:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.547+0000 7f390c191740 1 -- 172.21.15.116:0/3942783295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb054af0 msgr2=0x555ebb065280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.547+0000 7f390c191740 1 --2- 172.21.15.116:0/3942783295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb054af0 0x555ebb065280 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f38fc032420 tx=0x7f38fc02c0f0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.547+0000 7f390c191740 1 -- 172.21.15.116:0/3942783295 shutdown_connections 2024-03-19T01:13:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.547+0000 7f390c191740 1 --2- 172.21.15.116:0/3942783295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb059c60 0x555ebb055ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.547+0000 7f390c191740 1 --2- 172.21.15.116:0/3942783295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb054af0 0x555ebb065280 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.547+0000 7f390c191740 1 --2- 172.21.15.116:0/3942783295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054100 0x555ebb055590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.547+0000 7f390c191740 1 -- 172.21.15.116:0/3942783295 >> 172.21.15.116:0/3942783295 conn(0x555ebb04f670 msgr2=0x555ebb04e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.547+0000 7f390c191740 1 -- 172.21.15.116:0/3942783295 shutdown_connections 2024-03-19T01:13:52.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.547+0000 7f390c191740 1 -- 172.21.15.116:0/3942783295 wait complete. 2024-03-19T01:13:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.547+0000 7f390c191740 1 -- start start 2024-03-19T01:13:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.548+0000 7f390c191740 1 Processor -- start 2024-03-19T01:13:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.548+0000 7f390c191740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054100 0x555ebb062d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.548+0000 7f390c191740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054af0 0x555ebb063250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.548+0000 7f390c191740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb064210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f390c191740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f39041ac520 con 0x555ebb054100 2024-03-19T01:13:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f390c191740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f39041abe20 con 0x555ebb054af0 2024-03-19T01:13:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f390c191740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f39041a9000 con 0x555ebb059c60 2024-03-19T01:13:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f39037fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb064210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f39037fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb064210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37132/0 (socket says 172.21.15.116:37132) 2024-03-19T01:13:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f39037fe640 1 -- 172.21.15.116:0/2722416646 learned_addr learned my addr 172.21.15.116:0/2722416646 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:13:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f390c191740 10 client.0 _ll_register_callbacks cb 0x7f38f4009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:13:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f39037fe640 1 -- 172.21.15.116:0/2722416646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054af0 msgr2=0x555ebb063250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:13:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f39037fe640 1 --2- 172.21.15.116:0/2722416646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054af0 0x555ebb063250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f39037fe640 1 -- 172.21.15.116:0/2722416646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054100 msgr2=0x555ebb062d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:13:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f39037fe640 1 --2- 172.21.15.116:0/2722416646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054100 0x555ebb062d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f39037fe640 1 -- 172.21.15.116:0/2722416646 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ebb064800 con 0x555ebb059c60 2024-03-19T01:13:52.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f39037fe640 1 --2- 172.21.15.116:0/2722416646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb064210 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f38f4000c00 tx=0x7f38f4021b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bbd643a4f0c49f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.549+0000 7f38e6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f38e6ff5640 1 -- 172.21.15.116:0/2722416646 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f403b060 con 0x555ebb059c60 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f390c191740 1 -- 172.21.15.116:0/2722416646 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38fc034920 con 0x555ebb059c60 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f38e6ff5640 1 -- 172.21.15.116:0/2722416646 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f38f4021bc0 con 0x555ebb059c60 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f38e6ff5640 1 -- 172.21.15.116:0/2722416646 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f38f4026040 con 0x555ebb059c60 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f390c191740 10 client.25252 fetch_fsmap learned FSMap version 30 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f390c191740 1 -- 172.21.15.116:0/2722416646 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555ebb0d8bb0 con 0x555ebb059c60 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f38e6ff5640 1 -- 172.21.15.116:0/2722416646 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38fc034920 con 0x555ebb059c60 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f38e6ff5640 1 -- 172.21.15.116:0/2722416646 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f38f401f040 con 0x555ebb059c60 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f390c191740 10 client.25252 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f390c191740 10 client.25252 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f390c191740 20 client.25252 populate_metadata read hostname 'smithi116' 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f390c191740 1 --2- 172.21.15.116:0/2722416646 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x555ebb0d9520 0x555ebb0db940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:13:52.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f390c191740 4 client.25252 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:13:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f390c191740 1 -- 172.21.15.116:0/2722416646 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555ebafc0470 con 0x555ebb0d9520 2024-03-19T01:13:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.550+0000 7f3902ffd640 1 --2- 172.21.15.116:0/2722416646 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x555ebb0d9520 0x555ebb0db940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:13:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.551+0000 7f38e6ff5640 10 client.25252 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.551+0000 7f3902ffd640 1 --2- 172.21.15.116:0/2722416646 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x555ebb0d9520 0x555ebb0db940 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42deedd6e826ceb1 server_cookie=cc5e9cad7d8974c6 in_seq=0 out_seq=0 2024-03-19T01:13:52.566 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f38e6ff5640 1 -- 172.21.15.116:0/2722416646 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555ebafc0470 con 0x555ebb0d9520 2024-03-19T01:13:52.566 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f38e6ff5640 10 client.25252 handle_command_reply: tid=0 2024-03-19T01:13:52.566 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f390c191740 1 client.25252 shutdown 2024-03-19T01:13:52.566 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f390c191740 1 -- 172.21.15.116:0/2722416646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 msgr2=0x555ebb064210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f390c191740 1 --2- 172.21.15.116:0/2722416646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb064210 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f38f4000c00 tx=0x7f38f4021b50 comp rx=0 tx=0).stop 2024-03-19T01:13:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f390c191740 1 -- 172.21.15.116:0/2722416646 shutdown_connections 2024-03-19T01:13:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f390c191740 1 --2- 172.21.15.116:0/2722416646 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x555ebb0d9520 0x555ebb0db940 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f390c191740 1 --2- 172.21.15.116:0/2722416646 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x555ebb059c60 0x555ebb064210 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f38e6ff5640 0 client.25252 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:13:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f390c191740 1 --2- 172.21.15.116:0/2722416646 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x555ebb054af0 0x555ebb063250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f390c191740 1 --2- 172.21.15.116:0/2722416646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x555ebb054100 0x555ebb062d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f390c191740 1 -- 172.21.15.116:0/2722416646 >> 172.21.15.116:0/2722416646 conn(0x555ebb04f670 msgr2=0x555ebb0528d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f390c191740 1 -- 172.21.15.116:0/2722416646 shutdown_connections 2024-03-19T01:13:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f390c191740 1 -- 172.21.15.116:0/2722416646 wait complete. 2024-03-19T01:13:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.565+0000 7f390c191740 20 client.25252 trim_cache size 0 max 16384 2024-03-19T01:13:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:13:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:13:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:13:52.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:13:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.566+0000 7f39095a9640 1 -- 172.21.15.116:0/834476324 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f38ec0a2920 msgr2=0x7f38ec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.566+0000 7f39095a9640 1 --2- 172.21.15.116:0/834476324 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f38ec0a2920 0x7f38ec0a4de0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x555ebb055350 tx=0x7f38f4004730 comp rx=0 tx=0).stop 2024-03-19T01:13:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.566+0000 7f39095a9640 1 -- 172.21.15.116:0/834476324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3904102a70 msgr2=0x7f3904198020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:13:52.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.566+0000 7f39095a9640 1 --2- 172.21.15.116:0/834476324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3904102a70 0x7f3904198020 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f38fc009dc0 tx=0x7f38fc032f20 comp rx=0 tx=0).stop 2024-03-19T01:13:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.566+0000 7f39095a9640 1 -- 172.21.15.116:0/834476324 shutdown_connections 2024-03-19T01:13:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.566+0000 7f39095a9640 1 --2- 172.21.15.116:0/834476324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3904103430 0x7f390419f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.566+0000 7f39095a9640 1 --2- 172.21.15.116:0/834476324 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f38ec0a2920 0x7f38ec0a4de0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.566+0000 7f39095a9640 1 --2- 172.21.15.116:0/834476324 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3904102a70 0x7f3904198020 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.566+0000 7f39095a9640 1 --2- 172.21.15.116:0/834476324 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3904101880 0x7f3904197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:13:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.566+0000 7f39095a9640 1 -- 172.21.15.116:0/834476324 >> 172.21.15.116:0/834476324 conn(0x7f39040fcdf0 msgr2=0x7f3904105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:13:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.566+0000 7f39095a9640 1 -- 172.21.15.116:0/834476324 shutdown_connections 2024-03-19T01:13:52.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:13:52.566+0000 7f39095a9640 1 -- 172.21.15.116:0/834476324 wait complete. 2024-03-19T01:13:52.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:52 smithi162 ceph-mon[67178]: pgmap v389: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 11 MiB/s wr, 837 op/s 2024-03-19T01:13:54.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:54 smithi156 ceph-mon[66799]: pgmap v390: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 11 MiB/s wr, 750 op/s 2024-03-19T01:13:54.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:54 smithi162 ceph-mon[67178]: pgmap v390: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 11 MiB/s wr, 750 op/s 2024-03-19T01:13:54.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:54 smithi116 ceph-mon[62701]: pgmap v390: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 11 MiB/s wr, 750 op/s 2024-03-19T01:13:55.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:55 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:13:55.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:55 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:13:55.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:55 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:13:56.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:56 smithi156 ceph-mon[66799]: pgmap v391: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 12 MiB/s wr, 832 op/s 2024-03-19T01:13:56.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:56 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:56.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:56 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:56.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:56 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:56.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:56 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 70.0 s, 3108.2 tps, lat 32.205 ms stddev 32.869 2024-03-19T01:13:56.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:56 smithi162 ceph-mon[67178]: pgmap v391: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 12 MiB/s wr, 832 op/s 2024-03-19T01:13:56.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:56 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:56.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:56 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:56.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:56 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:56.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:56 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:56.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:56 smithi116 ceph-mon[62701]: pgmap v391: 97 pgs: 97 active+clean; 1.0 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 12 MiB/s wr, 832 op/s 2024-03-19T01:13:56.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:56.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:56.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:56.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:56 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:57 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:13:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:57 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:13:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:57 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:57 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:13:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:57 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:13:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:57 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:57 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:13:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:57 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:13:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:57 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:13:58.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:58 smithi156 ceph-mon[66799]: pgmap v392: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 11 MiB/s wr, 789 op/s 2024-03-19T01:13:58.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:58 smithi162 ceph-mon[67178]: pgmap v392: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 11 MiB/s wr, 789 op/s 2024-03-19T01:13:58.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:58 smithi116 ceph-mon[62701]: pgmap v392: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 11 MiB/s wr, 789 op/s 2024-03-19T01:13:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:13:59 smithi162 ceph-mon[67178]: pgmap v393: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 11 MiB/s wr, 767 op/s 2024-03-19T01:13:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:13:59 smithi116 ceph-mon[62701]: pgmap v393: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 11 MiB/s wr, 767 op/s 2024-03-19T01:13:59.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:13:59 smithi156 ceph-mon[66799]: pgmap v393: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 11 MiB/s wr, 767 op/s 2024-03-19T01:14:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 75.0 s, 3801.0 tps, lat 26.284 ms stddev 29.719 2024-03-19T01:14:02.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:02 smithi156 ceph-mon[66799]: pgmap v394: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 804 op/s 2024-03-19T01:14:02.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:02 smithi162 ceph-mon[67178]: pgmap v394: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 804 op/s 2024-03-19T01:14:02.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:02 smithi116 ceph-mon[62701]: pgmap v394: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 804 op/s 2024-03-19T01:14:04.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:04 smithi156 ceph-mon[66799]: pgmap v395: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.9 MiB/s wr, 718 op/s 2024-03-19T01:14:04.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:04 smithi162 ceph-mon[67178]: pgmap v395: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.9 MiB/s wr, 718 op/s 2024-03-19T01:14:04.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:04 smithi116 ceph-mon[62701]: pgmap v395: 97 pgs: 97 active+clean; 1.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.9 MiB/s wr, 718 op/s 2024-03-19T01:14:05.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.690+0000 7f6b2d1c4640 1 -- 172.21.15.116:0/3852870687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b281013a0 msgr2=0x7f6b28108c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:05.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.690+0000 7f6b2d1c4640 1 --2- 172.21.15.116:0/3852870687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b281013a0 0x7f6b28108c40 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c009930 tx=0x7f6b1c02bc30 comp rx=0 tx=0).stop 2024-03-19T01:14:05.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.690+0000 7f6b2d1c4640 1 -- 172.21.15.116:0/3852870687 shutdown_connections 2024-03-19T01:14:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.690+0000 7f6b2d1c4640 1 --2- 172.21.15.116:0/3852870687 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b281013a0 0x7f6b28108c40 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.690+0000 7f6b2d1c4640 1 --2- 172.21.15.116:0/3852870687 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b281009e0 0x7f6b28100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.690+0000 7f6b2d1c4640 1 --2- 172.21.15.116:0/3852870687 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b28101d40 0x7f6b28102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.690+0000 7f6b2d1c4640 1 -- 172.21.15.116:0/3852870687 >> 172.21.15.116:0/3852870687 conn(0x7f6b280fb470 msgr2=0x7f6b280fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.690+0000 7f6b2d1c4640 1 -- 172.21.15.116:0/3852870687 shutdown_connections 2024-03-19T01:14:05.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.690+0000 7f6b2d1c4640 1 -- 172.21.15.116:0/3852870687 wait complete. 2024-03-19T01:14:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.691+0000 7f6b2d1c4640 1 Processor -- start 2024-03-19T01:14:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.691+0000 7f6b2d1c4640 1 -- start start 2024-03-19T01:14:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.691+0000 7f6b2d1c4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b281009e0 0x7f6b2810aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.691+0000 7f6b2d1c4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b281013a0 0x7f6b28109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.691+0000 7f6b2d1c4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b28101d40 0x7f6b28109a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.691+0000 7f6b2d1c4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2810d510 con 0x7f6b281013a0 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.691+0000 7f6b2d1c4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2810d390 con 0x7f6b28101d40 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.691+0000 7f6b2d1c4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b2810d690 con 0x7f6b281009e0 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.692+0000 7f6b2759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b28101d40 0x7f6b28109a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.692+0000 7f6b2759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b28101d40 0x7f6b28109a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:41478/0 (socket says 172.21.15.116:41478) 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.692+0000 7f6b2759e640 1 -- 172.21.15.116:0/3301104926 learned_addr learned my addr 172.21.15.116:0/3301104926 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.692+0000 7f6b2659c640 1 --2- 172.21.15.116:0/3301104926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b281013a0 0x7f6b28109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.692+0000 7f6b2759e640 1 -- 172.21.15.116:0/3301104926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b281009e0 msgr2=0x7f6b2810aed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.692+0000 7f6b2759e640 1 --2- 172.21.15.116:0/3301104926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b281009e0 0x7f6b2810aed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.692+0000 7f6b2759e640 1 -- 172.21.15.116:0/3301104926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b281013a0 msgr2=0x7f6b28109510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.692+0000 7f6b2759e640 1 --2- 172.21.15.116:0/3301104926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b281013a0 0x7f6b28109510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.692+0000 7f6b2759e640 1 -- 172.21.15.116:0/3301104926 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b28109f90 con 0x7f6b28101d40 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.693+0000 7f6b2759e640 1 --2- 172.21.15.116:0/3301104926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b28101d40 0x7f6b28109a50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c036040 tx=0x7f6b1c005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.693+0000 7f6b0ffff640 1 -- 172.21.15.116:0/3301104926 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c043020 con 0x7f6b28101d40 2024-03-19T01:14:05.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.693+0000 7f6b2d1c4640 1 -- 172.21.15.116:0/3301104926 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b2810a220 con 0x7f6b28101d40 2024-03-19T01:14:05.696 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.693+0000 7f6b0ffff640 1 -- 172.21.15.116:0/3301104926 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b1c0054e0 con 0x7f6b28101d40 2024-03-19T01:14:05.696 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.693+0000 7f6b0ffff640 1 -- 172.21.15.116:0/3301104926 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c003bc0 con 0x7f6b28101d40 2024-03-19T01:14:05.696 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.693+0000 7f6b2d1c4640 1 -- 172.21.15.116:0/3301104926 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b281a6510 con 0x7f6b28101d40 2024-03-19T01:14:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.696+0000 7f6b2fdad740 1 Processor -- start 2024-03-19T01:14:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.697+0000 7f6b0ffff640 1 -- 172.21.15.116:0/3301104926 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6b1c046020 con 0x7f6b28101d40 2024-03-19T01:14:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.697+0000 7f6b2fdad740 1 -- start start 2024-03-19T01:14:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.697+0000 7f6b2fdad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216bd100 0x55d2216bd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.697+0000 7f6b2fdad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bdaf0 0x55d2216c25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.697+0000 7f6b2fdad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216c2c60 0x55d2216c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.697+0000 7f6b2fdad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d221621be0 con 0x55d2216bdaf0 2024-03-19T01:14:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.697+0000 7f6b2fdad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d2215ca960 con 0x55d2216c2c60 2024-03-19T01:14:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.697+0000 7f6b2fdad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d22161dc80 con 0x55d2216bd100 2024-03-19T01:14:05.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.697+0000 7f6b2659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216bd100 0x55d2216bd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.697+0000 7f6b2659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216bd100 0x55d2216bd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38614/0 (socket says 172.21.15.116:38614) 2024-03-19T01:14:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.697+0000 7f6b2659c640 1 -- 172.21.15.116:0/3093208472 learned_addr learned my addr 172.21.15.116:0/3093208472 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.697+0000 7f6b26d9d640 1 --2- 172.21.15.116:0/3093208472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bdaf0 0x55d2216c25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.697+0000 7f6b2659c640 1 --2- 172.21.15.116:0/3093208472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216c2c60 0x55d2216c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.698+0000 7f6b0ffff640 1 --2- 172.21.15.116:0/3301104926 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b100a2970 0x7f6b100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.698+0000 7f6b2659c640 1 -- 172.21.15.116:0/3093208472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216c2c60 msgr2=0x55d2216c50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.698+0000 7f6b2659c640 1 --2- 172.21.15.116:0/3093208472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216c2c60 0x55d2216c50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.698+0000 7f6b2659c640 1 -- 172.21.15.116:0/3093208472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bdaf0 msgr2=0x55d2216c25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.698+0000 7f6b2659c640 1 --2- 172.21.15.116:0/3093208472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bdaf0 0x55d2216c25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.698+0000 7f6b2659c640 1 -- 172.21.15.116:0/3093208472 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2215c1d20 con 0x55d2216bd100 2024-03-19T01:14:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.698+0000 7f6b2759e640 1 --2- 172.21.15.116:0/3301104926 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b100a2970 0x7f6b100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.698+0000 7f6b0ffff640 1 -- 172.21.15.116:0/3301104926 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6b1c0f0b90 con 0x7f6b28101d40 2024-03-19T01:14:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.698+0000 7f6b2659c640 1 --2- 172.21.15.116:0/3093208472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216bd100 0x55d2216bd520 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6b180093e0 tx=0x7f6b18009370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1cc68e8bec8f60b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.698+0000 7f6b0d7fa640 1 -- 172.21.15.116:0/3093208472 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1800f7c0 con 0x55d2216bd100 2024-03-19T01:14:05.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.698+0000 7f6b2759e640 1 --2- 172.21.15.116:0/3301104926 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b100a2970 0x7f6b100a4e30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c0b9970 tx=0x7f6b1c097040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:05.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.699+0000 7f6b0d7fa640 1 -- 172.21.15.116:0/3093208472 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b1800fe40 con 0x55d2216bd100 2024-03-19T01:14:05.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.699+0000 7f6b0d7fa640 1 -- 172.21.15.116:0/3093208472 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1800e3b0 con 0x55d2216bd100 2024-03-19T01:14:05.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.700+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3093208472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216bd100 msgr2=0x55d2216bd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:05.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.700+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3093208472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216bd100 0x55d2216bd520 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6b180093e0 tx=0x7f6b18009370 comp rx=0 tx=0).stop 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.700+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3093208472 shutdown_connections 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.700+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3093208472 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216c2c60 0x55d2216c50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.700+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3093208472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bdaf0 0x55d2216c25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.700+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3093208472 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216bd100 0x55d2216bd520 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.700+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3093208472 >> 172.21.15.116:0/3093208472 conn(0x55d2216b8670 msgr2=0x55d2216b8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.700+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3093208472 shutdown_connections 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.700+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3093208472 wait complete. 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b2fdad740 1 -- start start 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b2fdad740 1 Processor -- start 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b2fdad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bd100 0x55d2216c7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b2fdad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bdaf0 0x55d2216c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b2fdad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216c2c60 0x55d2216ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b2fdad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55d2216c5c10 con 0x55d2216bd100 2024-03-19T01:14:05.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b2fdad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55d2215dddb0 con 0x55d2216bdaf0 2024-03-19T01:14:05.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b2fdad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55d2216c5d90 con 0x55d2216c2c60 2024-03-19T01:14:05.707 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b2fdad740 10 client.0 _ll_register_callbacks cb 0x55d2215e2e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:14:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b26d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bdaf0 0x55d2216c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b2659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bd100 0x55d2216c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b26d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bdaf0 0x55d2216c76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:41486/0 (socket says 172.21.15.116:41486) 2024-03-19T01:14:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b26d9d640 1 -- 172.21.15.116:0/2091971927 learned_addr learned my addr 172.21.15.116:0/2091971927 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.701+0000 7f6b2659c640 1 --2- 172.21.15.116:0/2091971927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216c2c60 0x55d2216ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.702+0000 7f6b26d9d640 1 -- 172.21.15.116:0/2091971927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216c2c60 msgr2=0x55d2216ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:05.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.702+0000 7f6b26d9d640 1 --2- 172.21.15.116:0/2091971927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216c2c60 0x55d2216ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.702+0000 7f6b26d9d640 1 -- 172.21.15.116:0/2091971927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bd100 msgr2=0x55d2216c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.702+0000 7f6b26d9d640 1 --2- 172.21.15.116:0/2091971927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bd100 0x55d2216c7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.702+0000 7f6b26d9d640 1 -- 172.21.15.116:0/2091971927 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2215dcc60 con 0x55d2216bdaf0 2024-03-19T01:14:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.702+0000 7f6b26d9d640 1 --2- 172.21.15.116:0/2091971927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bdaf0 0x55d2216c76d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6b200117e0 tx=0x7f6b20011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92d00d676702388e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.702+0000 7f6b0aff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:14:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.702+0000 7f6b0aff5640 1 -- 172.21.15.116:0/2091971927 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2000fdf0 con 0x55d2216bdaf0 2024-03-19T01:14:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.702+0000 7f6b2fdad740 1 -- 172.21.15.116:0/2091971927 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b1800fe40 con 0x55d2216bdaf0 2024-03-19T01:14:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.702+0000 7f6b0aff5640 1 -- 172.21.15.116:0/2091971927 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6b20004b20 con 0x55d2216bdaf0 2024-03-19T01:14:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.702+0000 7f6b0aff5640 1 -- 172.21.15.116:0/2091971927 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6b20004d20 con 0x55d2216bdaf0 2024-03-19T01:14:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.702+0000 7f6b2fdad740 10 client.25286 fetch_fsmap learned FSMap version 30 2024-03-19T01:14:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.702+0000 7f6b2fdad740 1 -- 172.21.15.116:0/2091971927 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d221336210 con 0x55d2216bdaf0 2024-03-19T01:14:05.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.703+0000 7f6b0aff5640 1 -- 172.21.15.116:0/2091971927 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b1800fe40 con 0x55d2216bdaf0 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.703+0000 7f6b0aff5640 1 -- 172.21.15.116:0/2091971927 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f6b20023610 con 0x55d2216bdaf0 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.704+0000 7f6b2fdad740 10 client.25286 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.704+0000 7f6b2fdad740 10 client.25286 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.704+0000 7f6b2fdad740 20 client.25286 populate_metadata read hostname 'smithi116' 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.704+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/2091971927 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55d2216d3070 0x55d2216d54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.704+0000 7f6b2fdad740 4 client.25286 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.704+0000 7f6b2fdad740 1 -- 172.21.15.116:0/2091971927 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d22162f120 con 0x55d2216d3070 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.704+0000 7f6b2659c640 1 --2- 172.21.15.116:0/2091971927 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55d2216d3070 0x55d2216d54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.705+0000 7f6b0aff5640 10 client.25286 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.705+0000 7f6b2659c640 1 --2- 172.21.15.116:0/2091971927 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55d2216d3070 0x55d2216d54a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a490990c10201776 server_cookie=5b807a10319b1eba in_seq=0 out_seq=0 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.705+0000 7f6b0aff5640 1 -- 172.21.15.116:0/2091971927 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d22162f120 con 0x55d2216d3070 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.705+0000 7f6b0aff5640 10 client.25286 handle_command_reply: tid=0 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.706+0000 7f6b2fdad740 1 client.25286 shutdown 2024-03-19T01:14:05.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.706+0000 7f6b2fdad740 1 -- 172.21.15.116:0/2091971927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bdaf0 msgr2=0x55d2216c76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.706+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/2091971927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bdaf0 0x55d2216c76d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6b200117e0 tx=0x7f6b20011ca0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.706+0000 7f6b2fdad740 1 -- 172.21.15.116:0/2091971927 shutdown_connections 2024-03-19T01:14:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.706+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/2091971927 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55d2216d3070 0x55d2216d54a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.706+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/2091971927 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216c2c60 0x55d2216ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.706+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/2091971927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bdaf0 0x55d2216c76d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.706+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/2091971927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bd100 0x55d2216c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.706+0000 7f6b2fdad740 1 -- 172.21.15.116:0/2091971927 >> 172.21.15.116:0/2091971927 conn(0x55d2216b8670 msgr2=0x55d2216b98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.706+0000 7f6b0aff5640 0 client.25286 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.706+0000 7f6b2fdad740 1 -- 172.21.15.116:0/2091971927 shutdown_connections 2024-03-19T01:14:05.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.706+0000 7f6b2fdad740 1 -- 172.21.15.116:0/2091971927 wait complete. 2024-03-19T01:14:05.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.706+0000 7f6b2fdad740 20 client.25286 trim_cache size 0 max 16384 2024-03-19T01:14:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.742+0000 7f6b2fdad740 1 Processor -- start 2024-03-19T01:14:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.742+0000 7f6b2fdad740 1 -- start start 2024-03-19T01:14:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.742+0000 7f6b2fdad740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bd100 0x55d2216c7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.742+0000 7f6b2fdad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bdaf0 0x55d2216ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.742+0000 7f6b2fdad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216c2c60 0x55d2216c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.742+0000 7f6b2fdad740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b281af3b0 con 0x55d2216bd100 2024-03-19T01:14:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.742+0000 7f6b2fdad740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6b281b1040 con 0x55d2216bdaf0 2024-03-19T01:14:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.742+0000 7f6b2fdad740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b281afa80 con 0x55d2216c2c60 2024-03-19T01:14:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.743+0000 7f6b2659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bd100 0x55d2216c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.743+0000 7f6b2659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bd100 0x55d2216c7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50268/0 (socket says 172.21.15.116:50268) 2024-03-19T01:14:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.743+0000 7f6b2659c640 1 -- 172.21.15.116:0/3033511539 learned_addr learned my addr 172.21.15.116:0/3033511539 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.743+0000 7f6b2659c640 1 --2- 172.21.15.116:0/3033511539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216c2c60 0x55d2216c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.743+0000 7f6b2659c640 1 -- 172.21.15.116:0/3033511539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216c2c60 msgr2=0x55d2216c76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.743+0000 7f6b2659c640 1 --2- 172.21.15.116:0/3033511539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216c2c60 0x55d2216c76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.743+0000 7f6b2659c640 1 -- 172.21.15.116:0/3033511539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bdaf0 msgr2=0x55d2216ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.743+0000 7f6b2659c640 1 --2- 172.21.15.116:0/3033511539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bdaf0 0x55d2216ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.743+0000 7f6b2659c640 1 -- 172.21.15.116:0/3033511539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d221402620 con 0x55d2216bd100 2024-03-19T01:14:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.743+0000 7f6b2659c640 1 --2- 172.21.15.116:0/3033511539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bd100 0x55d2216c7190 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6b1801f7e0 tx=0x7f6b18005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451889b1078eada3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.743+0000 7f6b0cff9640 1 -- 172.21.15.116:0/3033511539 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1801f890 con 0x55d2216bd100 2024-03-19T01:14:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.744+0000 7f6b0cff9640 1 -- 172.21.15.116:0/3033511539 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b180055b0 con 0x55d2216bd100 2024-03-19T01:14:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.744+0000 7f6b0cff9640 1 -- 172.21.15.116:0/3033511539 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1802f050 con 0x55d2216bd100 2024-03-19T01:14:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.744+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3033511539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bd100 msgr2=0x55d2216c7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.744+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3033511539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bd100 0x55d2216c7190 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6b1801f7e0 tx=0x7f6b18005850 comp rx=0 tx=0).stop 2024-03-19T01:14:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.744+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3033511539 shutdown_connections 2024-03-19T01:14:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.744+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3033511539 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216c2c60 0x55d2216c76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.744+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3033511539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bdaf0 0x55d2216ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.744+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3033511539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216bd100 0x55d2216c7190 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.744+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3033511539 >> 172.21.15.116:0/3033511539 conn(0x55d2216b8670 msgr2=0x55d2216b8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.745+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3033511539 shutdown_connections 2024-03-19T01:14:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.745+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3033511539 wait complete. 2024-03-19T01:14:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.745+0000 7f6b2fdad740 1 -- start start 2024-03-19T01:14:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.745+0000 7f6b2fdad740 1 Processor -- start 2024-03-19T01:14:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.745+0000 7f6b2fdad740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bd100 0x55d2216cb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2fdad740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216bdaf0 0x55d2216cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bd100 0x55d2216cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bd100 0x55d2216cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:41512/0 (socket says 172.21.15.116:41512) 2024-03-19T01:14:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2659c640 1 -- 172.21.15.116:0/3353504396 learned_addr learned my addr 172.21.15.116:0/3353504396 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216c2c60 0x55d2216ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b26d9d640 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216bdaf0 0x55d2216cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3353504396 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b281b08b0 con 0x55d2216c2c60 2024-03-19T01:14:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3353504396 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6b281b01b0 con 0x55d2216bd100 2024-03-19T01:14:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3353504396 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6b281ad390 con 0x55d2216bdaf0 2024-03-19T01:14:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2fdad740 10 client.0 _ll_register_callbacks cb 0x7f6b20009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:14:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2659c640 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216c2c60 0x55d2216ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2659c640 1 -- 172.21.15.116:0/3353504396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216bdaf0 msgr2=0x55d2216cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2659c640 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216bdaf0 0x55d2216cbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2659c640 1 -- 172.21.15.116:0/3353504396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216c2c60 msgr2=0x55d2216ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2659c640 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216c2c60 0x55d2216ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.746+0000 7f6b2659c640 1 -- 172.21.15.116:0/3353504396 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2216cd3d0 con 0x55d2216bd100 2024-03-19T01:14:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.747+0000 7f6b2659c640 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bd100 0x55d2216cb890 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6b180094b0 tx=0x7f6b18005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdc1132fa6a4f24d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:05.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.750+0000 7f6b0aff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:14:05.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.750+0000 7f6b0aff5640 1 -- 172.21.15.116:0/3353504396 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1800f790 con 0x55d2216bd100 2024-03-19T01:14:05.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.750+0000 7f6b0aff5640 1 -- 172.21.15.116:0/3353504396 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6b1800f930 con 0x55d2216bd100 2024-03-19T01:14:05.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.750+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3353504396 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b180055b0 con 0x55d2216bd100 2024-03-19T01:14:05.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.750+0000 7f6b0aff5640 1 -- 172.21.15.116:0/3353504396 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6b180050d0 con 0x55d2216bd100 2024-03-19T01:14:05.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.750+0000 7f6b2fdad740 10 client.25289 fetch_fsmap learned FSMap version 30 2024-03-19T01:14:05.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.750+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3353504396 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55d2216ce6e0 con 0x55d2216bd100 2024-03-19T01:14:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.750+0000 7f6b0aff5640 1 -- 172.21.15.116:0/3353504396 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b180055b0 con 0x55d2216bd100 2024-03-19T01:14:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.751+0000 7f6b0aff5640 1 -- 172.21.15.116:0/3353504396 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f6b1802d040 con 0x55d2216bd100 2024-03-19T01:14:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.751+0000 7f6b2fdad740 10 client.25289 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:14:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.751+0000 7f6b2fdad740 10 client.25289 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:14:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.751+0000 7f6b2fdad740 20 client.25289 populate_metadata read hostname 'smithi116' 2024-03-19T01:14:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.751+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55d2217396b0 0x55d22173bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.751+0000 7f6b2fdad740 4 client.25289 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:14:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.751+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3353504396 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d221629470 con 0x55d2217396b0 2024-03-19T01:14:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.751+0000 7f6b26d9d640 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55d2217396b0 0x55d22173bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.751+0000 7f6b26d9d640 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55d2217396b0 0x55d22173bad0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=429ef9359b7bb4e6 server_cookie=c6cb82161a6838c3 in_seq=0 out_seq=0 2024-03-19T01:14:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.751+0000 7f6b0aff5640 10 client.25289 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.764+0000 7f6b0aff5640 1 -- 172.21.15.116:0/3353504396 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d221629470 con 0x55d2217396b0 2024-03-19T01:14:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.764+0000 7f6b0aff5640 10 client.25289 handle_command_reply: tid=0 2024-03-19T01:14:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.764+0000 7f6b2fdad740 1 client.25289 shutdown 2024-03-19T01:14:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3353504396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bd100 msgr2=0x55d2216cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bd100 0x55d2216cb890 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6b180094b0 tx=0x7f6b18005f90 comp rx=0 tx=0).stop 2024-03-19T01:14:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3353504396 shutdown_connections 2024-03-19T01:14:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55d2216c2c60 0x55d2216ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55d2217396b0 0x55d22173bad0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55d2216bdaf0 0x55d2216cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2fdad740 1 --2- 172.21.15.116:0/3353504396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55d2216bd100 0x55d2216cb890 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b0aff5640 0 client.25289 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3353504396 >> 172.21.15.116:0/3353504396 conn(0x55d2216b8670 msgr2=0x55d2216c7150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3353504396 shutdown_connections 2024-03-19T01:14:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2fdad740 1 -- 172.21.15.116:0/3353504396 wait complete. 2024-03-19T01:14:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2fdad740 20 client.25289 trim_cache size 0 max 16384 2024-03-19T01:14:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:14:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:14:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:14:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:14:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2d1c4640 1 -- 172.21.15.116:0/3301104926 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b100a2970 msgr2=0x7f6b100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2d1c4640 1 --2- 172.21.15.116:0/3301104926 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b100a2970 0x7f6b100a4e30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c0b9970 tx=0x7f6b1c097040 comp rx=0 tx=0).stop 2024-03-19T01:14:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2d1c4640 1 -- 172.21.15.116:0/3301104926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b28101d40 msgr2=0x7f6b28109a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.765+0000 7f6b2d1c4640 1 --2- 172.21.15.116:0/3301104926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b28101d40 0x7f6b28109a50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c036040 tx=0x7f6b1c005a20 comp rx=0 tx=0).stop 2024-03-19T01:14:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.766+0000 7f6b2d1c4640 1 -- 172.21.15.116:0/3301104926 shutdown_connections 2024-03-19T01:14:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.766+0000 7f6b2d1c4640 1 --2- 172.21.15.116:0/3301104926 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6b100a2970 0x7f6b100a4e30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.766+0000 7f6b2d1c4640 1 --2- 172.21.15.116:0/3301104926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b28101d40 0x7f6b28109a50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.766+0000 7f6b2d1c4640 1 --2- 172.21.15.116:0/3301104926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b281013a0 0x7f6b28109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.766+0000 7f6b2d1c4640 1 --2- 172.21.15.116:0/3301104926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6b281009e0 0x7f6b2810aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.766+0000 7f6b2d1c4640 1 -- 172.21.15.116:0/3301104926 >> 172.21.15.116:0/3301104926 conn(0x7f6b280fb470 msgr2=0x7f6b28104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.766+0000 7f6b2d1c4640 1 -- 172.21.15.116:0/3301104926 shutdown_connections 2024-03-19T01:14:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:05.766+0000 7f6b2d1c4640 1 -- 172.21.15.116:0/3301104926 wait complete. 2024-03-19T01:14:06.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:06 smithi156 ceph-mon[66799]: pgmap v396: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.4 MiB/s wr, 789 op/s 2024-03-19T01:14:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 80.0 s, 3454.4 tps, lat 28.998 ms stddev 31.074 2024-03-19T01:14:06.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:06 smithi162 ceph-mon[67178]: pgmap v396: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.4 MiB/s wr, 789 op/s 2024-03-19T01:14:06.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:06 smithi116 ceph-mon[62701]: pgmap v396: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.4 MiB/s wr, 789 op/s 2024-03-19T01:14:08.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:08 smithi156 ceph-mon[66799]: pgmap v397: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.2 MiB/s wr, 762 op/s 2024-03-19T01:14:08.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:08 smithi162 ceph-mon[67178]: pgmap v397: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.2 MiB/s wr, 762 op/s 2024-03-19T01:14:08.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:08 smithi116 ceph-mon[62701]: pgmap v397: 97 pgs: 97 active+clean; 1.0 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.2 MiB/s wr, 762 op/s 2024-03-19T01:14:10.462 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:10 smithi162 ceph-mon[67178]: pgmap v398: 97 pgs: 97 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 MiB/s wr, 828 op/s 2024-03-19T01:14:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:10 smithi156 ceph-mon[66799]: pgmap v398: 97 pgs: 97 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 MiB/s wr, 828 op/s 2024-03-19T01:14:10.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:10 smithi116 ceph-mon[62701]: pgmap v398: 97 pgs: 97 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 MiB/s wr, 828 op/s 2024-03-19T01:14:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 85.0 s, 3681.8 tps, lat 27.179 ms stddev 28.487 2024-03-19T01:14:12.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:12 smithi156 ceph-mon[66799]: pgmap v399: 97 pgs: 97 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 829 op/s 2024-03-19T01:14:12.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:12 smithi162 ceph-mon[67178]: pgmap v399: 97 pgs: 97 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 829 op/s 2024-03-19T01:14:12.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:12 smithi116 ceph-mon[62701]: pgmap v399: 97 pgs: 97 active+clean; 1.0 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 MiB/s wr, 829 op/s 2024-03-19T01:14:13.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:13 smithi162 ceph-mon[67178]: pgmap v400: 97 pgs: 97 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 701 op/s 2024-03-19T01:14:13.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:13 smithi116 ceph-mon[62701]: pgmap v400: 97 pgs: 97 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 701 op/s 2024-03-19T01:14:13.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:13 smithi156 ceph-mon[66799]: pgmap v400: 97 pgs: 97 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 701 op/s 2024-03-19T01:14:16.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:16 smithi156 ceph-mon[66799]: pgmap v401: 97 pgs: 97 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 13 MiB/s wr, 857 op/s 2024-03-19T01:14:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 90.0 s, 3351.4 tps, lat 29.708 ms stddev 30.221 2024-03-19T01:14:16.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:16 smithi162 ceph-mon[67178]: pgmap v401: 97 pgs: 97 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 13 MiB/s wr, 857 op/s 2024-03-19T01:14:16.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:16 smithi116 ceph-mon[62701]: pgmap v401: 97 pgs: 97 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 13 MiB/s wr, 857 op/s 2024-03-19T01:14:18.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:18 smithi156 ceph-mon[66799]: pgmap v402: 97 pgs: 97 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 MiB/s wr, 799 op/s 2024-03-19T01:14:18.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:18 smithi162 ceph-mon[67178]: pgmap v402: 97 pgs: 97 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 MiB/s wr, 799 op/s 2024-03-19T01:14:18.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:18 smithi116 ceph-mon[62701]: pgmap v402: 97 pgs: 97 active+clean; 1.1 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 MiB/s wr, 799 op/s 2024-03-19T01:14:18.898 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.895+0000 7fad7aee8640 1 -- 172.21.15.116:0/992813420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad741022f0 msgr2=0x7fad74108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.895+0000 7fad7aee8640 1 --2- 172.21.15.116:0/992813420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad741022f0 0x7fad74108b80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009ca0 tx=0x7fad6c02dfe0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.895+0000 7fad7aee8640 1 -- 172.21.15.116:0/992813420 shutdown_connections 2024-03-19T01:14:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.895+0000 7fad7aee8640 1 --2- 172.21.15.116:0/992813420 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad741022f0 0x7fad74108b80 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.895+0000 7fad7aee8640 1 --2- 172.21.15.116:0/992813420 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad74101930 0x7fad74101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.895+0000 7fad7aee8640 1 --2- 172.21.15.116:0/992813420 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad74100740 0x7fad74100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.895+0000 7fad7aee8640 1 -- 172.21.15.116:0/992813420 >> 172.21.15.116:0/992813420 conn(0x7fad740fbcb0 msgr2=0x7fad740fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.896+0000 7fad7aee8640 1 -- 172.21.15.116:0/992813420 shutdown_connections 2024-03-19T01:14:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.896+0000 7fad7aee8640 1 -- 172.21.15.116:0/992813420 wait complete. 2024-03-19T01:14:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.896+0000 7fad7aee8640 1 Processor -- start 2024-03-19T01:14:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.896+0000 7fad7aee8640 1 -- start start 2024-03-19T01:14:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.896+0000 7fad7aee8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad74100740 0x7fad74195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.896+0000 7fad7aee8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad74101930 0x7fad74195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.896+0000 7fad7aee8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad741022f0 0x7fad7419cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.896+0000 7fad7aee8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad7410b2c0 con 0x7fad74100740 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.896+0000 7fad7aee8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad7410b140 con 0x7fad741022f0 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.896+0000 7fad73fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad74101930 0x7fad74195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.896+0000 7fad73fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad74101930 0x7fad74195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51880/0 (socket says 172.21.15.116:51880) 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.896+0000 7fad73fff640 1 -- 172.21.15.116:0/383729702 learned_addr learned my addr 172.21.15.116:0/383729702 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.897+0000 7fad7aee8640 1 -- 172.21.15.116:0/383729702 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad7410b440 con 0x7fad74101930 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.897+0000 7fad78c84640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad74100740 0x7fad74195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.897+0000 7fad79485640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad741022f0 0x7fad7419cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.897+0000 7fad73fff640 1 -- 172.21.15.116:0/383729702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad741022f0 msgr2=0x7fad7419cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.897+0000 7fad73fff640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad741022f0 0x7fad7419cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.897+0000 7fad73fff640 1 -- 172.21.15.116:0/383729702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad74100740 msgr2=0x7fad74195820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.897+0000 7fad73fff640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad74100740 0x7fad74195820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.900 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.897+0000 7fad73fff640 1 -- 172.21.15.116:0/383729702 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7419d440 con 0x7fad74101930 2024-03-19T01:14:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.897+0000 7fad78c84640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad74100740 0x7fad74195820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:14:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.897+0000 7fad73fff640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad74101930 0x7fad74195d60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fad6000e9f0 tx=0x7fad6000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.898+0000 7fad71ffb640 1 -- 172.21.15.116:0/383729702 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6000cd80 con 0x7fad74101930 2024-03-19T01:14:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.898+0000 7fad71ffb640 1 -- 172.21.15.116:0/383729702 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad60010910 con 0x7fad74101930 2024-03-19T01:14:18.901 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.898+0000 7fad71ffb640 1 -- 172.21.15.116:0/383729702 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad60018990 con 0x7fad74101930 2024-03-19T01:14:18.902 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.898+0000 7fad7aee8640 1 -- 172.21.15.116:0/383729702 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad7419d730 con 0x7fad74101930 2024-03-19T01:14:18.902 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.898+0000 7fad7aee8640 1 -- 172.21.15.116:0/383729702 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad7419db70 con 0x7fad74101930 2024-03-19T01:14:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.901+0000 7fad71ffb640 1 -- 172.21.15.116:0/383729702 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fad60018b30 con 0x7fad74101930 2024-03-19T01:14:18.903 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.902+0000 7fad7dbaa740 1 Processor -- start 2024-03-19T01:14:18.904 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.902+0000 7fad7dbaa740 1 -- start start 2024-03-19T01:14:18.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.902+0000 7fad7dbaa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f37c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.902+0000 7fad7dbaa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37cd70 0x55ba9f37f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.902+0000 7fad7dbaa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f391a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.902+0000 7fad7dbaa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ba9f2e5be0 con 0x55ba9f37cd70 2024-03-19T01:14:18.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.902+0000 7fad7dbaa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ba9f28e960 con 0x55ba9f37fed0 2024-03-19T01:14:18.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.902+0000 7fad7dbaa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ba9f2e1c80 con 0x55ba9f37c380 2024-03-19T01:14:18.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.902+0000 7fad71ffb640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fad640a2a10 0x7fad640a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.903+0000 7fad78c84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37cd70 0x55ba9f37f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.903+0000 7fad73fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f391a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.903+0000 7fad78c84640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37cd70 0x55ba9f37f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51022/0 (socket says 172.21.15.116:51022) 2024-03-19T01:14:18.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.903+0000 7fad78c84640 1 -- 172.21.15.116:0/157287598 learned_addr learned my addr 172.21.15.116:0/157287598 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:18.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.903+0000 7fad79485640 1 --2- 172.21.15.116:0/157287598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f37c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.905 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.903+0000 7fad79485640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fad640a2a10 0x7fad640a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.903+0000 7fad71ffb640 1 -- 172.21.15.116:0/383729702 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fad600ced60 con 0x7fad74101930 2024-03-19T01:14:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.903+0000 7fad78c84640 1 -- 172.21.15.116:0/157287598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 msgr2=0x55ba9f37c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.903+0000 7fad78c84640 1 --2- 172.21.15.116:0/157287598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f37c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.903+0000 7fad78c84640 1 -- 172.21.15.116:0/157287598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 msgr2=0x55ba9f391a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.903+0000 7fad78c84640 1 --2- 172.21.15.116:0/157287598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f391a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.903+0000 7fad78c84640 1 -- 172.21.15.116:0/157287598 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba9f285d20 con 0x55ba9f37cd70 2024-03-19T01:14:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.903+0000 7fad79485640 1 --2- 172.21.15.116:0/157287598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f37c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:14:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.904+0000 7fad79485640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fad640a2a10 0x7fad640a4ed0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fad6c02e520 tx=0x7fad6c0095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:18.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.905+0000 7fad78c84640 1 --2- 172.21.15.116:0/157287598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37cd70 0x55ba9f37f860 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fad680114f0 tx=0x7fad680119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a956c68e5ab30b95 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:18.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.905+0000 7fad5effd640 1 -- 172.21.15.116:0/157287598 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6801e070 con 0x55ba9f37cd70 2024-03-19T01:14:18.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.906+0000 7fad5effd640 1 -- 172.21.15.116:0/157287598 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad680035f0 con 0x55ba9f37cd70 2024-03-19T01:14:18.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.906+0000 7fad5effd640 1 -- 172.21.15.116:0/157287598 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6800f820 con 0x55ba9f37cd70 2024-03-19T01:14:18.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.906+0000 7fad7dbaa740 1 -- 172.21.15.116:0/157287598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37cd70 msgr2=0x55ba9f37f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:18.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.906+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/157287598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37cd70 0x55ba9f37f860 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fad680114f0 tx=0x7fad680119b0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.906+0000 7fad7dbaa740 1 -- 172.21.15.116:0/157287598 shutdown_connections 2024-03-19T01:14:18.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.907+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/157287598 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f391a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.907+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/157287598 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37cd70 0x55ba9f37f860 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.907+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/157287598 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f37c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.907+0000 7fad7dbaa740 1 -- 172.21.15.116:0/157287598 >> 172.21.15.116:0/157287598 conn(0x55ba9f380a80 msgr2=0x55ba9f380e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:18.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.907+0000 7fad7dbaa740 1 -- 172.21.15.116:0/157287598 shutdown_connections 2024-03-19T01:14:18.909 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.907+0000 7fad7dbaa740 1 -- 172.21.15.116:0/157287598 wait complete. 2024-03-19T01:14:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.907+0000 7fad7dbaa740 1 -- start start 2024-03-19T01:14:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.907+0000 7fad7dbaa740 1 Processor -- start 2024-03-19T01:14:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad7dbaa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f38b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad7dbaa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37cd70 0x55ba9f38b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad7dbaa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f390620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad7dbaa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55ba9f385610 con 0x55ba9f37cd70 2024-03-19T01:14:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad7dbaa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ba9f2a1db0 con 0x55ba9f37fed0 2024-03-19T01:14:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad7dbaa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55ba9f385790 con 0x55ba9f37c380 2024-03-19T01:14:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad78c84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f38b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37cd70 0x55ba9f38b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad78c84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f38b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51904/0 (socket says 172.21.15.116:51904) 2024-03-19T01:14:18.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad78c84640 1 -- 172.21.15.116:0/3549494594 learned_addr learned my addr 172.21.15.116:0/3549494594 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:18.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad79485640 1 --2- 172.21.15.116:0/3549494594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f390620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad7dbaa740 10 client.0 _ll_register_callbacks cb 0x55ba9f2a6e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:14:18.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad78c84640 1 -- 172.21.15.116:0/3549494594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 msgr2=0x55ba9f390620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:18.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad78c84640 1 --2- 172.21.15.116:0/3549494594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f390620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.911 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad78c84640 1 -- 172.21.15.116:0/3549494594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37cd70 msgr2=0x55ba9f38b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad78c84640 1 --2- 172.21.15.116:0/3549494594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37cd70 0x55ba9f38b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.908+0000 7fad78c84640 1 -- 172.21.15.116:0/3549494594 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba9f2a0c60 con 0x55ba9f37c380 2024-03-19T01:14:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.909+0000 7fad73fff640 1 --2- 172.21.15.116:0/3549494594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37cd70 0x55ba9f38b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:14:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.909+0000 7fad78c84640 1 --2- 172.21.15.116:0/3549494594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f38b0b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fad68011eb0 tx=0x7fad68005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=366f065a8c5b2fb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.909+0000 7fad53fff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:14:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.909+0000 7fad53fff640 1 -- 172.21.15.116:0/3549494594 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6801e070 con 0x55ba9f37c380 2024-03-19T01:14:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.909+0000 7fad53fff640 1 -- 172.21.15.116:0/3549494594 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fad680037a0 con 0x55ba9f37c380 2024-03-19T01:14:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.909+0000 7fad7dbaa740 1 -- 172.21.15.116:0/3549494594 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad680035f0 con 0x55ba9f37c380 2024-03-19T01:14:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.910+0000 7fad53fff640 1 -- 172.21.15.116:0/3549494594 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fad680055f0 con 0x55ba9f37c380 2024-03-19T01:14:18.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.910+0000 7fad7dbaa740 10 client.25279 fetch_fsmap learned FSMap version 30 2024-03-19T01:14:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.910+0000 7fad7dbaa740 1 -- 172.21.15.116:0/3549494594 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ba9effa210 con 0x55ba9f37c380 2024-03-19T01:14:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.910+0000 7fad53fff640 1 -- 172.21.15.116:0/3549494594 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad68028070 con 0x55ba9f37c380 2024-03-19T01:14:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.911+0000 7fad53fff640 1 -- 172.21.15.116:0/3549494594 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7fad6800f5c0 con 0x55ba9f37c380 2024-03-19T01:14:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.911+0000 7fad7dbaa740 10 client.25279 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:14:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.911+0000 7fad7dbaa740 10 client.25279 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:14:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.911+0000 7fad7dbaa740 20 client.25279 populate_metadata read hostname 'smithi116' 2024-03-19T01:14:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.911+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/3549494594 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ba9f3b8cd0 0x55ba9f3bb140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.911+0000 7fad7dbaa740 4 client.25279 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:14:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.911+0000 7fad7dbaa740 1 -- 172.21.15.116:0/3549494594 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fad680035f0 con 0x55ba9f3b8cd0 2024-03-19T01:14:18.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.911+0000 7fad73fff640 1 --2- 172.21.15.116:0/3549494594 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ba9f3b8cd0 0x55ba9f3bb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.916 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.912+0000 7fad73fff640 1 --2- 172.21.15.116:0/3549494594 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ba9f3b8cd0 0x55ba9f3bb140 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceeebb627ae8f6b9 server_cookie=756e51db65c8bca6 in_seq=0 out_seq=0 2024-03-19T01:14:18.916 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.912+0000 7fad53fff640 10 client.25279 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:18.916 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.913+0000 7fad53fff640 1 -- 172.21.15.116:0/3549494594 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fad680035f0 con 0x55ba9f3b8cd0 2024-03-19T01:14:18.916 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.913+0000 7fad53fff640 10 client.25279 handle_command_reply: tid=0 2024-03-19T01:14:18.916 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.913+0000 7fad7dbaa740 1 client.25279 shutdown 2024-03-19T01:14:18.916 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.913+0000 7fad7dbaa740 1 -- 172.21.15.116:0/3549494594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 msgr2=0x55ba9f38b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:18.916 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.913+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/3549494594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f38b0b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fad68011eb0 tx=0x7fad68005270 comp rx=0 tx=0).stop 2024-03-19T01:14:18.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.913+0000 7fad7dbaa740 1 -- 172.21.15.116:0/3549494594 shutdown_connections 2024-03-19T01:14:18.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.913+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/3549494594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f390620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.913+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/3549494594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37cd70 0x55ba9f38b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.913+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/3549494594 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ba9f3b8cd0 0x55ba9f3bb140 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.914+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/3549494594 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f38b0b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.914+0000 7fad7dbaa740 1 -- 172.21.15.116:0/3549494594 >> 172.21.15.116:0/3549494594 conn(0x55ba9f380a80 msgr2=0x55ba9f381cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:18.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.914+0000 7fad53fff640 0 client.25279 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:18.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.914+0000 7fad7dbaa740 1 -- 172.21.15.116:0/3549494594 shutdown_connections 2024-03-19T01:14:18.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.914+0000 7fad7dbaa740 1 -- 172.21.15.116:0/3549494594 wait complete. 2024-03-19T01:14:18.917 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.914+0000 7fad7dbaa740 20 client.25279 trim_cache size 0 max 16384 2024-03-19T01:14:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.936+0000 7fad7dbaa740 1 Processor -- start 2024-03-19T01:14:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.936+0000 7fad7dbaa740 1 -- start start 2024-03-19T01:14:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.936+0000 7fad7dbaa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f3bfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad7dbaa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37cd70 0x55ba9f3bff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad7dbaa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37fed0 0x55ba9f3c04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad78c84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f3bfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad7dbaa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad7419e3e0 con 0x55ba9f37fed0 2024-03-19T01:14:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad7dbaa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad741a0070 con 0x55ba9f37cd70 2024-03-19T01:14:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad7dbaa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad7419eab0 con 0x55ba9f37c380 2024-03-19T01:14:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad78c84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f3bfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51920/0 (socket says 172.21.15.116:51920) 2024-03-19T01:14:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad78c84640 1 -- 172.21.15.116:0/2232655853 learned_addr learned my addr 172.21.15.116:0/2232655853 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad73fff640 1 --2- 172.21.15.116:0/2232655853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37cd70 0x55ba9f3bff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad79485640 1 --2- 172.21.15.116:0/2232655853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37fed0 0x55ba9f3c04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad78c84640 1 -- 172.21.15.116:0/2232655853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37cd70 msgr2=0x55ba9f3bff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad78c84640 1 --2- 172.21.15.116:0/2232655853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37cd70 0x55ba9f3bff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad78c84640 1 -- 172.21.15.116:0/2232655853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37fed0 msgr2=0x55ba9f3c04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad78c84640 1 --2- 172.21.15.116:0/2232655853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37fed0 0x55ba9f3c04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad78c84640 1 -- 172.21.15.116:0/2232655853 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba9f0c6620 con 0x55ba9f37c380 2024-03-19T01:14:18.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.937+0000 7fad73fff640 1 --2- 172.21.15.116:0/2232655853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37cd70 0x55ba9f3bff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:14:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.938+0000 7fad78c84640 1 --2- 172.21.15.116:0/2232655853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f3bfa40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fad680059f0 tx=0x7fad680119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc455ee32b9f4ab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.938+0000 7fad5e7fc640 1 -- 172.21.15.116:0/2232655853 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6801e040 con 0x55ba9f37c380 2024-03-19T01:14:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.938+0000 7fad5e7fc640 1 -- 172.21.15.116:0/2232655853 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad68028070 con 0x55ba9f37c380 2024-03-19T01:14:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.938+0000 7fad5e7fc640 1 -- 172.21.15.116:0/2232655853 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad680035f0 con 0x55ba9f37c380 2024-03-19T01:14:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.940+0000 7fad7dbaa740 1 -- 172.21.15.116:0/2232655853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 msgr2=0x55ba9f3bfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.940+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/2232655853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f3bfa40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fad680059f0 tx=0x7fad680119b0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.940+0000 7fad7dbaa740 1 -- 172.21.15.116:0/2232655853 shutdown_connections 2024-03-19T01:14:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.940+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/2232655853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f37fed0 0x55ba9f3c04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.940+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/2232655853 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37cd70 0x55ba9f3bff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.940+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/2232655853 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37c380 0x55ba9f3bfa40 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fad680059f0 tx=0x7fad680119b0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.940+0000 7fad7dbaa740 1 -- 172.21.15.116:0/2232655853 >> 172.21.15.116:0/2232655853 conn(0x55ba9f380a80 msgr2=0x55ba9f383ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.940+0000 7fad7dbaa740 1 -- 172.21.15.116:0/2232655853 shutdown_connections 2024-03-19T01:14:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.940+0000 7fad7dbaa740 1 -- 172.21.15.116:0/2232655853 wait complete. 2024-03-19T01:14:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.941+0000 7fad7dbaa740 1 -- start start 2024-03-19T01:14:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.941+0000 7fad7dbaa740 1 Processor -- start 2024-03-19T01:14:18.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.941+0000 7fad7dbaa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37cd70 0x55ba9f418a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.941+0000 7fad7dbaa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f41af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.941+0000 7fad7dbaa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f41b4a0 0x55ba9f41d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad73fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f41af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad78c84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37cd70 0x55ba9f418a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad73fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f41af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:51514/0 (socket says 172.21.15.116:51514) 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad78c84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37cd70 0x55ba9f418a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51934/0 (socket says 172.21.15.116:51934) 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad7dbaa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad7419f8e0 con 0x55ba9f41b4a0 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad7dbaa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad7419f1e0 con 0x55ba9f37fed0 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad7dbaa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fad74071fd0 con 0x55ba9f37cd70 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad7dbaa740 10 client.0 _ll_register_callbacks cb 0x7fad6801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad73fff640 1 -- 172.21.15.116:0/1386160109 learned_addr learned my addr 172.21.15.116:0/1386160109 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad79485640 1 --2- 172.21.15.116:0/1386160109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f41b4a0 0x55ba9f41d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad73fff640 1 -- 172.21.15.116:0/1386160109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37cd70 msgr2=0x55ba9f418a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:18.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad73fff640 1 --2- 172.21.15.116:0/1386160109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37cd70 0x55ba9f418a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad73fff640 1 -- 172.21.15.116:0/1386160109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f41b4a0 msgr2=0x55ba9f41d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad73fff640 1 --2- 172.21.15.116:0/1386160109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f41b4a0 0x55ba9f41d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad73fff640 1 -- 172.21.15.116:0/1386160109 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba9f41ded0 con 0x55ba9f37fed0 2024-03-19T01:14:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.942+0000 7fad78c84640 1 --2- 172.21.15.116:0/1386160109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37cd70 0x55ba9f418a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:14:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad73fff640 1 --2- 172.21.15.116:0/1386160109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f41af60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fad6000e9c0 tx=0x7fad60018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1b31819831dcdee server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad53fff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:14:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad53fff640 1 -- 172.21.15.116:0/1386160109 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad600ae020 con 0x55ba9f37fed0 2024-03-19T01:14:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad53fff640 1 -- 172.21.15.116:0/1386160109 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fad600999e0 con 0x55ba9f37fed0 2024-03-19T01:14:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad7dbaa740 1 -- 172.21.15.116:0/1386160109 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad68028070 con 0x55ba9f37fed0 2024-03-19T01:14:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad53fff640 1 -- 172.21.15.116:0/1386160109 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fad600a3e80 con 0x55ba9f37fed0 2024-03-19T01:14:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad7dbaa740 10 client.25307 fetch_fsmap learned FSMap version 30 2024-03-19T01:14:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad7dbaa740 1 -- 172.21.15.116:0/1386160109 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ba9f41f2b0 con 0x55ba9f37fed0 2024-03-19T01:14:18.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad53fff640 1 -- 172.21.15.116:0/1386160109 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad68028070 con 0x55ba9f37fed0 2024-03-19T01:14:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad53fff640 1 -- 172.21.15.116:0/1386160109 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7fad60097430 con 0x55ba9f37fed0 2024-03-19T01:14:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad7dbaa740 10 client.25307 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:14:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad7dbaa740 10 client.25307 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:14:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad7dbaa740 20 client.25307 populate_metadata read hostname 'smithi116' 2024-03-19T01:14:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/1386160109 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ba9f41fcd0 0x55ba9f4220f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad7dbaa740 4 client.25307 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:14:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.943+0000 7fad7dbaa740 1 -- 172.21.15.116:0/1386160109 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ba9f2f3120 con 0x55ba9f41fcd0 2024-03-19T01:14:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.944+0000 7fad78c84640 1 --2- 172.21.15.116:0/1386160109 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ba9f41fcd0 0x55ba9f4220f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.946+0000 7fad78c84640 1 --2- 172.21.15.116:0/1386160109 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ba9f41fcd0 0x55ba9f4220f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40b21b23d7604bb8 server_cookie=c7f1ccdb07bca5e3 in_seq=0 out_seq=0 2024-03-19T01:14:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.946+0000 7fad53fff640 10 client.25307 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.957+0000 7fad53fff640 1 -- 172.21.15.116:0/1386160109 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ba9f2f3120 con 0x55ba9f41fcd0 2024-03-19T01:14:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.957+0000 7fad53fff640 10 client.25307 handle_command_reply: tid=0 2024-03-19T01:14:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.957+0000 7fad7dbaa740 1 client.25307 shutdown 2024-03-19T01:14:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.958+0000 7fad7dbaa740 1 -- 172.21.15.116:0/1386160109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 msgr2=0x55ba9f41af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:18.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.958+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/1386160109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f41af60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fad6000e9c0 tx=0x7fad60018420 comp rx=0 tx=0).stop 2024-03-19T01:14:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.958+0000 7fad7dbaa740 1 -- 172.21.15.116:0/1386160109 shutdown_connections 2024-03-19T01:14:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.958+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/1386160109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55ba9f41b4a0 0x55ba9f41d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.958+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/1386160109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ba9f37fed0 0x55ba9f41af60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.958+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/1386160109 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55ba9f41fcd0 0x55ba9f4220f0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.958+0000 7fad7dbaa740 1 --2- 172.21.15.116:0/1386160109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55ba9f37cd70 0x55ba9f418a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.958+0000 7fad7dbaa740 1 -- 172.21.15.116:0/1386160109 >> 172.21.15.116:0/1386160109 conn(0x55ba9f380a80 msgr2=0x55ba9f385b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.958+0000 7fad53fff640 0 client.25307 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.959+0000 7fad7dbaa740 1 -- 172.21.15.116:0/1386160109 shutdown_connections 2024-03-19T01:14:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.959+0000 7fad7dbaa740 1 -- 172.21.15.116:0/1386160109 wait complete. 2024-03-19T01:14:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.959+0000 7fad7dbaa740 20 client.25307 trim_cache size 0 max 16384 2024-03-19T01:14:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:14:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:14:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:14:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:14:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.960+0000 7fad7aee8640 1 -- 172.21.15.116:0/383729702 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fad640a2a10 msgr2=0x7fad640a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.960+0000 7fad7aee8640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fad640a2a10 0x7fad640a4ed0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fad6c02e520 tx=0x7fad6c0095c0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.960+0000 7fad7aee8640 1 -- 172.21.15.116:0/383729702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad74101930 msgr2=0x7fad74195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.960+0000 7fad7aee8640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad74101930 0x7fad74195d60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fad6000e9f0 tx=0x7fad6000eeb0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.960+0000 7fad7aee8640 1 -- 172.21.15.116:0/383729702 shutdown_connections 2024-03-19T01:14:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.960+0000 7fad7aee8640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fad640a2a10 0x7fad640a4ed0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.960+0000 7fad7aee8640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad741022f0 0x7fad7419cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.960+0000 7fad7aee8640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fad74101930 0x7fad74195d60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.960+0000 7fad7aee8640 1 --2- 172.21.15.116:0/383729702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad74100740 0x7fad74195820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.960+0000 7fad7aee8640 1 -- 172.21.15.116:0/383729702 >> 172.21.15.116:0/383729702 conn(0x7fad740fbcb0 msgr2=0x7fad74104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.960+0000 7fad7aee8640 1 -- 172.21.15.116:0/383729702 shutdown_connections 2024-03-19T01:14:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:18.960+0000 7fad7aee8640 1 -- 172.21.15.116:0/383729702 wait complete. 2024-03-19T01:14:20.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:20 smithi156 ceph-mon[66799]: pgmap v403: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 847 op/s 2024-03-19T01:14:20.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:20 smithi162 ceph-mon[67178]: pgmap v403: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 847 op/s 2024-03-19T01:14:20.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:20 smithi116 ceph-mon[62701]: pgmap v403: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 MiB/s wr, 847 op/s 2024-03-19T01:14:21.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:21 smithi156 ceph-mon[66799]: pgmap v404: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 721 op/s 2024-03-19T01:14:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 95.0 s, 2984.0 tps, lat 33.542 ms stddev 36.967 2024-03-19T01:14:21.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:21 smithi162 ceph-mon[67178]: pgmap v404: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 721 op/s 2024-03-19T01:14:21.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:21 smithi116 ceph-mon[62701]: pgmap v404: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.3 MiB/s wr, 721 op/s 2024-03-19T01:14:24.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:24 smithi156 ceph-mon[66799]: pgmap v405: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.6 MiB/s wr, 688 op/s 2024-03-19T01:14:24.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:24 smithi162 ceph-mon[67178]: pgmap v405: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.6 MiB/s wr, 688 op/s 2024-03-19T01:14:24.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:24 smithi116 ceph-mon[62701]: pgmap v405: 97 pgs: 97 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.6 MiB/s wr, 688 op/s 2024-03-19T01:14:26.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:26 smithi156 ceph-mon[66799]: pgmap v406: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.7 MiB/s wr, 811 op/s 2024-03-19T01:14:26.545 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 100.0 s, 3298.8 tps, lat 30.361 ms stddev 31.810 2024-03-19T01:14:26.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:26 smithi162 ceph-mon[67178]: pgmap v406: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.7 MiB/s wr, 811 op/s 2024-03-19T01:14:26.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:26 smithi116 ceph-mon[62701]: pgmap v406: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.7 MiB/s wr, 811 op/s 2024-03-19T01:14:28.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:28 smithi156 ceph-mon[66799]: pgmap v407: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 747 op/s 2024-03-19T01:14:28.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:28 smithi162 ceph-mon[67178]: pgmap v407: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 747 op/s 2024-03-19T01:14:28.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:28 smithi116 ceph-mon[62701]: pgmap v407: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.8 MiB/s wr, 747 op/s 2024-03-19T01:14:30.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:30 smithi156 ceph-mon[66799]: pgmap v408: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 803 op/s 2024-03-19T01:14:30.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:30 smithi162 ceph-mon[67178]: pgmap v408: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 803 op/s 2024-03-19T01:14:30.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:30 smithi116 ceph-mon[62701]: pgmap v408: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 803 op/s 2024-03-19T01:14:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 105.0 s, 3890.2 tps, lat 25.745 ms stddev 27.048 2024-03-19T01:14:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.089+0000 7f7fc24e8640 1 -- 172.21.15.116:0/758370568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fbc1075c0 msgr2=0x7f7fbc1099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.089+0000 7f7fc24e8640 1 --2- 172.21.15.116:0/758370568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fbc1075c0 0x7f7fbc1099b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009ca0 tx=0x7f7fb402c0f0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.089+0000 7f7fc24e8640 1 -- 172.21.15.116:0/758370568 shutdown_connections 2024-03-19T01:14:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.089+0000 7f7fc24e8640 1 --2- 172.21.15.116:0/758370568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fbc1075c0 0x7f7fbc1099b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.089+0000 7f7fc24e8640 1 --2- 172.21.15.116:0/758370568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fbc104c90 0x7f7fbc107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.089+0000 7f7fc24e8640 1 --2- 172.21.15.116:0/758370568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fbc102360 0x7f7fbc104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.089+0000 7f7fc24e8640 1 -- 172.21.15.116:0/758370568 >> 172.21.15.116:0/758370568 conn(0x7f7fbc0fbcf0 msgr2=0x7f7fbc0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.089+0000 7f7fc24e8640 1 -- 172.21.15.116:0/758370568 shutdown_connections 2024-03-19T01:14:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.089+0000 7f7fc24e8640 1 -- 172.21.15.116:0/758370568 wait complete. 2024-03-19T01:14:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.090+0000 7f7fc24e8640 1 Processor -- start 2024-03-19T01:14:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.090+0000 7f7fc24e8640 1 -- start start 2024-03-19T01:14:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.090+0000 7f7fc24e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fbc102360 0x7f7fbc071290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.090+0000 7f7fc24e8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fbc104c90 0x7f7fbc0717d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.090+0000 7f7fc24e8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fbc1075c0 0x7f7fbc0727c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.090+0000 7f7fc24e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc06b9a0 con 0x7f7fbc102360 2024-03-19T01:14:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.090+0000 7f7fc24e8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc06b820 con 0x7f7fbc104c90 2024-03-19T01:14:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.090+0000 7f7fc24e8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc06bb20 con 0x7f7fbc1075c0 2024-03-19T01:14:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.090+0000 7f7fbbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fbc102360 0x7f7fbc071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.090+0000 7f7fc0a85640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fbc1075c0 0x7f7fbc0727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.090+0000 7f7fc0a85640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fbc1075c0 0x7f7fbc0727c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45808/0 (socket says 172.21.15.116:45808) 2024-03-19T01:14:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.090+0000 7f7fc0a85640 1 -- 172.21.15.116:0/980431644 learned_addr learned my addr 172.21.15.116:0/980431644 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.090+0000 7f7fbb7fe640 1 --2- 172.21.15.116:0/980431644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fbc104c90 0x7f7fbc0717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.091+0000 7f7fc0a85640 1 -- 172.21.15.116:0/980431644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fbc104c90 msgr2=0x7f7fbc0717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.091+0000 7f7fc0a85640 1 --2- 172.21.15.116:0/980431644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fbc104c90 0x7f7fbc0717d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.091+0000 7f7fc0a85640 1 -- 172.21.15.116:0/980431644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fbc102360 msgr2=0x7f7fbc071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.091+0000 7f7fc0a85640 1 --2- 172.21.15.116:0/980431644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fbc102360 0x7f7fbc071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.091+0000 7f7fc0a85640 1 -- 172.21.15.116:0/980431644 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fbc072e30 con 0x7f7fbc1075c0 2024-03-19T01:14:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.091+0000 7f7fc0a85640 1 --2- 172.21.15.116:0/980431644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fbc1075c0 0x7f7fbc0727c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009bf0 tx=0x7f7fb4032ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.091+0000 7f7fb97fa640 1 -- 172.21.15.116:0/980431644 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4041020 con 0x7f7fbc1075c0 2024-03-19T01:14:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.091+0000 7f7fc24e8640 1 -- 172.21.15.116:0/980431644 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fbc0730c0 con 0x7f7fbc1075c0 2024-03-19T01:14:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.091+0000 7f7fc24e8640 1 -- 172.21.15.116:0/980431644 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fbc073600 con 0x7f7fbc1075c0 2024-03-19T01:14:32.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.091+0000 7f7fb97fa640 1 -- 172.21.15.116:0/980431644 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fb4034cd0 con 0x7f7fbc1075c0 2024-03-19T01:14:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.091+0000 7f7fb97fa640 1 -- 172.21.15.116:0/980431644 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4039040 con 0x7f7fbc1075c0 2024-03-19T01:14:32.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.095+0000 7f7fc51af740 1 Processor -- start 2024-03-19T01:14:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.095+0000 7f7fb97fa640 1 -- 172.21.15.116:0/980431644 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7fb4044020 con 0x7f7fbc1075c0 2024-03-19T01:14:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.095+0000 7f7fc51af740 1 -- start start 2024-03-19T01:14:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.095+0000 7f7fc51af740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7d910 0x55a3bde7dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.095+0000 7f7fc51af740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7e300 0x55a3bde82e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.095+0000 7f7fc51af740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde83470 0x55a3bde85900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.095+0000 7f7fc51af740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a3bddd9be0 con 0x55a3bde83470 2024-03-19T01:14:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.095+0000 7f7fc51af740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a3bdd82960 con 0x55a3bde7d910 2024-03-19T01:14:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.095+0000 7f7fc51af740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a3bddd5c80 con 0x55a3bde7e300 2024-03-19T01:14:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.096+0000 7f7fbb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde83470 0x55a3bde85900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.096+0000 7f7fbb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde83470 0x55a3bde85900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45554/0 (socket says 172.21.15.116:45554) 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.096+0000 7f7fbb7fe640 1 -- 172.21.15.116:0/3523939058 learned_addr learned my addr 172.21.15.116:0/3523939058 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.096+0000 7f7fbb7fe640 1 --2- 172.21.15.116:0/3523939058 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7d910 0x55a3bde7dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.096+0000 7f7fbbfff640 1 --2- 172.21.15.116:0/3523939058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7e300 0x55a3bde82e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.096+0000 7f7fbb7fe640 1 -- 172.21.15.116:0/3523939058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7e300 msgr2=0x55a3bde82e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.096+0000 7f7fbb7fe640 1 --2- 172.21.15.116:0/3523939058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7e300 0x55a3bde82e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.096+0000 7f7fbb7fe640 1 -- 172.21.15.116:0/3523939058 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7d910 msgr2=0x55a3bde7dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.096+0000 7f7fbb7fe640 1 --2- 172.21.15.116:0/3523939058 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7d910 0x55a3bde7dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.096+0000 7f7fbb7fe640 1 -- 172.21.15.116:0/3523939058 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3bdd79d20 con 0x55a3bde83470 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.096+0000 7f7fbbfff640 1 --2- 172.21.15.116:0/3523939058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7e300 0x55a3bde82e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.097+0000 7f7fbb7fe640 1 --2- 172.21.15.116:0/3523939058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde83470 0x55a3bde85900 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009fd0 tx=0x7f7fb0005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c5948f0625dc29 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.097+0000 7f7fa27fc640 1 -- 172.21.15.116:0/3523939058 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0004030 con 0x55a3bde83470 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.097+0000 7f7fa27fc640 1 -- 172.21.15.116:0/3523939058 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fb000fc70 con 0x55a3bde83470 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.097+0000 7f7fa27fc640 1 -- 172.21.15.116:0/3523939058 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0003a10 con 0x55a3bde83470 2024-03-19T01:14:32.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.097+0000 7f7fb97fa640 1 --2- 172.21.15.116:0/980431644 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7fa40a29f0 0x7f7fa40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.097+0000 7f7fbbfff640 1 --2- 172.21.15.116:0/980431644 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7fa40a29f0 0x7f7fa40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.097+0000 7f7fb97fa640 1 -- 172.21.15.116:0/980431644 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7fb40f0780 con 0x7f7fbc1075c0 2024-03-19T01:14:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.097+0000 7f7fc51af740 1 -- 172.21.15.116:0/3523939058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde83470 msgr2=0x55a3bde85900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.097+0000 7f7fbbfff640 1 --2- 172.21.15.116:0/980431644 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7fa40a29f0 0x7f7fa40a4eb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x55a3bde7e700 tx=0x7f7fac005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.097+0000 7f7fc51af740 1 --2- 172.21.15.116:0/3523939058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde83470 0x55a3bde85900 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0009fd0 tx=0x7f7fb0005850 comp rx=0 tx=0).stop 2024-03-19T01:14:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.098+0000 7f7fc51af740 1 -- 172.21.15.116:0/3523939058 shutdown_connections 2024-03-19T01:14:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.098+0000 7f7fc51af740 1 --2- 172.21.15.116:0/3523939058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde83470 0x55a3bde85900 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.098+0000 7f7fc51af740 1 --2- 172.21.15.116:0/3523939058 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7e300 0x55a3bde82e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.098+0000 7f7fc51af740 1 --2- 172.21.15.116:0/3523939058 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7d910 0x55a3bde7dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.098+0000 7f7fc51af740 1 -- 172.21.15.116:0/3523939058 >> 172.21.15.116:0/3523939058 conn(0x55a3bde78e80 msgr2=0x55a3bde79280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:32.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.098+0000 7f7fc51af740 1 -- 172.21.15.116:0/3523939058 shutdown_connections 2024-03-19T01:14:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.098+0000 7f7fc51af740 1 -- 172.21.15.116:0/3523939058 wait complete. 2024-03-19T01:14:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.098+0000 7f7fc51af740 1 -- start start 2024-03-19T01:14:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.098+0000 7f7fc51af740 1 Processor -- start 2024-03-19T01:14:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fc51af740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7d910 0x55a3bde87910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fc51af740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7e300 0x55a3bde87e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fc51af740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde83470 0x55a3bde8ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fc51af740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a3bde86420 con 0x55a3bde83470 2024-03-19T01:14:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fc51af740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a3bdd95db0 con 0x55a3bde7e300 2024-03-19T01:14:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fc51af740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a3bde865a0 con 0x55a3bde7d910 2024-03-19T01:14:32.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fc51af740 10 client.0 _ll_register_callbacks cb 0x55a3bdd9ae70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:14:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fbbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7d910 0x55a3bde87910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fbbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7d910 0x55a3bde87910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45826/0 (socket says 172.21.15.116:45826) 2024-03-19T01:14:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fbbfff640 1 -- 172.21.15.116:0/2156274036 learned_addr learned my addr 172.21.15.116:0/2156274036 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fc0a85640 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde83470 0x55a3bde8ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fbb7fe640 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7e300 0x55a3bde87e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fbbfff640 1 -- 172.21.15.116:0/2156274036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7e300 msgr2=0x55a3bde87e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fbbfff640 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7e300 0x55a3bde87e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fbbfff640 1 -- 172.21.15.116:0/2156274036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde83470 msgr2=0x55a3bde8ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fbbfff640 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde83470 0x55a3bde8ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fbbfff640 1 -- 172.21.15.116:0/2156274036 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3bdd94c60 con 0x55a3bde7d910 2024-03-19T01:14:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.099+0000 7f7fbb7fe640 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7e300 0x55a3bde87e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:14:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.100+0000 7f7fbbfff640 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7d910 0x55a3bde87910 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7fac011c10 tx=0x7f7fac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d92b5cd4f70d43d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:32.104 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.100+0000 7f7f9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.100+0000 7f7f9b7fe640 1 -- 172.21.15.116:0/2156274036 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac010cb0 con 0x55a3bde7d910 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.100+0000 7f7fc51af740 1 -- 172.21.15.116:0/2156274036 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fb000fc70 con 0x55a3bde7d910 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.100+0000 7f7f9b7fe640 1 -- 172.21.15.116:0/2156274036 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7fac028400 con 0x55a3bde7d910 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.100+0000 7f7f9b7fe640 1 -- 172.21.15.116:0/2156274036 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7fac010e50 con 0x55a3bde7d910 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.100+0000 7f7fc51af740 10 client.25294 fetch_fsmap learned FSMap version 30 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.100+0000 7f7fc51af740 1 -- 172.21.15.116:0/2156274036 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a3bdaee210 con 0x55a3bde7d910 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.100+0000 7f7f9b7fe640 1 -- 172.21.15.116:0/2156274036 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fb000fc70 con 0x55a3bde7d910 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.101+0000 7f7f9b7fe640 1 -- 172.21.15.116:0/2156274036 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f7fac0337b0 con 0x55a3bde7d910 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.101+0000 7f7fc51af740 10 client.25294 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.101+0000 7f7fc51af740 10 client.25294 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.101+0000 7f7fc51af740 20 client.25294 populate_metadata read hostname 'smithi116' 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.101+0000 7f7fc51af740 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a3bdeb5c70 0x55a3bdeb80f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.101+0000 7f7fc51af740 4 client.25294 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.101+0000 7f7fc51af740 1 -- 172.21.15.116:0/2156274036 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a3bdde7120 con 0x55a3bdeb5c70 2024-03-19T01:14:32.105 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.102+0000 7f7fbb7fe640 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a3bdeb5c70 0x55a3bdeb80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.102+0000 7f7f9b7fe640 10 client.25294 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:32.106 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.102+0000 7f7fbb7fe640 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a3bdeb5c70 0x55a3bdeb80f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8141e0bc134419a server_cookie=bcd16d363f6f4420 in_seq=0 out_seq=0 2024-03-19T01:14:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.103+0000 7f7f9b7fe640 1 -- 172.21.15.116:0/2156274036 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a3bdde7120 con 0x55a3bdeb5c70 2024-03-19T01:14:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.103+0000 7f7f9b7fe640 10 client.25294 handle_command_reply: tid=0 2024-03-19T01:14:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.103+0000 7f7fc51af740 1 client.25294 shutdown 2024-03-19T01:14:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.104+0000 7f7fc51af740 1 -- 172.21.15.116:0/2156274036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7d910 msgr2=0x55a3bde87910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.104+0000 7f7fc51af740 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7d910 0x55a3bde87910 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7fac011c10 tx=0x7f7fac0046c0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.104+0000 7f7fc51af740 1 -- 172.21.15.116:0/2156274036 shutdown_connections 2024-03-19T01:14:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.104+0000 7f7fc51af740 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde83470 0x55a3bde8ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.104+0000 7f7fc51af740 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7e300 0x55a3bde87e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.104+0000 7f7fc51af740 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a3bdeb5c70 0x55a3bdeb80f0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.104+0000 7f7fc51af740 1 --2- 172.21.15.116:0/2156274036 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7d910 0x55a3bde87910 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.104+0000 7f7f9b7fe640 0 client.25294 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.104+0000 7f7fc51af740 1 -- 172.21.15.116:0/2156274036 >> 172.21.15.116:0/2156274036 conn(0x55a3bde78e80 msgr2=0x55a3bde7a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.104+0000 7f7fc51af740 1 -- 172.21.15.116:0/2156274036 shutdown_connections 2024-03-19T01:14:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.104+0000 7f7fc51af740 1 -- 172.21.15.116:0/2156274036 wait complete. 2024-03-19T01:14:32.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.104+0000 7f7fc51af740 20 client.25294 trim_cache size 0 max 16384 2024-03-19T01:14:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.124+0000 7f7fc51af740 1 Processor -- start 2024-03-19T01:14:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.124+0000 7f7fc51af740 1 -- start start 2024-03-19T01:14:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.124+0000 7f7fc51af740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7d910 0x55a3bde87c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fc51af740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde7e300 0x55a3bde8ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fbbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7d910 0x55a3bde87c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fbbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7d910 0x55a3bde87c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:59490/0 (socket says 172.21.15.116:59490) 2024-03-19T01:14:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fbb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde7e300 0x55a3bde8ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fbbfff640 1 -- 172.21.15.116:0/860939363 learned_addr learned my addr 172.21.15.116:0/860939363 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fc51af740 1 --2- 172.21.15.116:0/860939363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde83470 0x55a3bde7ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fc51af740 1 -- 172.21.15.116:0/860939363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa800e5f0 con 0x55a3bde7e300 2024-03-19T01:14:32.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fc51af740 1 -- 172.21.15.116:0/860939363 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa8010280 con 0x55a3bde7d910 2024-03-19T01:14:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fc51af740 1 -- 172.21.15.116:0/860939363 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa800ecc0 con 0x55a3bde83470 2024-03-19T01:14:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fc0a85640 1 --2- 172.21.15.116:0/860939363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde83470 0x55a3bde7ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fbb7fe640 1 -- 172.21.15.116:0/860939363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde83470 msgr2=0x55a3bde7ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fbb7fe640 1 --2- 172.21.15.116:0/860939363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde83470 0x55a3bde7ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fbb7fe640 1 -- 172.21.15.116:0/860939363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7d910 msgr2=0x55a3bde87c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fbb7fe640 1 --2- 172.21.15.116:0/860939363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7d910 0x55a3bde87c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.125+0000 7f7fbb7fe640 1 -- 172.21.15.116:0/860939363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a3bdbba620 con 0x55a3bde7e300 2024-03-19T01:14:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.126+0000 7f7fbb7fe640 1 --2- 172.21.15.116:0/860939363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde7e300 0x55a3bde8ee50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x55a3bde88df0 tx=0x7f7fb0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7107f01dbc2c7edc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:32.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.126+0000 7f7fa1ffb640 1 -- 172.21.15.116:0/860939363 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb000fa30 con 0x55a3bde7e300 2024-03-19T01:14:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.127+0000 7f7fa1ffb640 1 -- 172.21.15.116:0/860939363 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fb000fbd0 con 0x55a3bde7e300 2024-03-19T01:14:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.127+0000 7f7fa1ffb640 1 -- 172.21.15.116:0/860939363 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb0005570 con 0x55a3bde7e300 2024-03-19T01:14:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.127+0000 7f7fc51af740 1 -- 172.21.15.116:0/860939363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde7e300 msgr2=0x55a3bde8ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.127+0000 7f7fc51af740 1 --2- 172.21.15.116:0/860939363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde7e300 0x55a3bde8ee50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x55a3bde88df0 tx=0x7f7fb0005f90 comp rx=0 tx=0).stop 2024-03-19T01:14:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 -- 172.21.15.116:0/860939363 shutdown_connections 2024-03-19T01:14:32.132 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 --2- 172.21.15.116:0/860939363 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde83470 0x55a3bde7ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 --2- 172.21.15.116:0/860939363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde7e300 0x55a3bde8ee50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 --2- 172.21.15.116:0/860939363 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde7d910 0x55a3bde87c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 -- 172.21.15.116:0/860939363 >> 172.21.15.116:0/860939363 conn(0x55a3bde78e80 msgr2=0x55a3bde87520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 -- 172.21.15.116:0/860939363 shutdown_connections 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 -- 172.21.15.116:0/860939363 wait complete. 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 -- start start 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 Processor -- start 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde7d910 0x55a3bde8c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7e300 0x55a3bde8c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde83470 0x55a3bde8d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa800faf0 con 0x55a3bde7d910 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa800f3f0 con 0x55a3bde83470 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa800c5d0 con 0x55a3bde7e300 2024-03-19T01:14:32.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.128+0000 7f7fc51af740 10 client.0 _ll_register_callbacks cb 0x7f7fac009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fc0a85640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde83470 0x55a3bde8d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fbbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde7d910 0x55a3bde8c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fc0a85640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde83470 0x55a3bde8d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:59504/0 (socket says 172.21.15.116:59504) 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fc0a85640 1 -- 172.21.15.116:0/30915932 learned_addr learned my addr 172.21.15.116:0/30915932 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fbb7fe640 1 --2- 172.21.15.116:0/30915932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7e300 0x55a3bde8c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fc0a85640 1 -- 172.21.15.116:0/30915932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7e300 msgr2=0x55a3bde8c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fc0a85640 1 --2- 172.21.15.116:0/30915932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7e300 0x55a3bde8c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fc0a85640 1 -- 172.21.15.116:0/30915932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde7d910 msgr2=0x55a3bde8c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fc0a85640 1 --2- 172.21.15.116:0/30915932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde7d910 0x55a3bde8c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fc0a85640 1 -- 172.21.15.116:0/30915932 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a3bde8dc80 con 0x55a3bde83470 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fc0a85640 1 --2- 172.21.15.116:0/30915932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde83470 0x55a3bde8d690 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40bd030 tx=0x7f7fb40bca00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a7268de50bd9f9a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7f9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7f9b7fe640 1 -- 172.21.15.116:0/30915932 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb40bd500 con 0x55a3bde83470 2024-03-19T01:14:32.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7f9b7fe640 1 -- 172.21.15.116:0/30915932 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7fb40c53b0 con 0x55a3bde83470 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fc51af740 1 -- 172.21.15.116:0/30915932 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fb000fbd0 con 0x55a3bde83470 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7f9b7fe640 1 -- 172.21.15.116:0/30915932 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7fb40bd7a0 con 0x55a3bde83470 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fc51af740 10 client.25325 fetch_fsmap learned FSMap version 30 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.129+0000 7f7fc51af740 1 -- 172.21.15.116:0/30915932 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a3bde8f000 con 0x55a3bde83470 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.130+0000 7f7f9b7fe640 1 -- 172.21.15.116:0/30915932 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fb000fbd0 con 0x55a3bde83470 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.130+0000 7f7f9b7fe640 1 -- 172.21.15.116:0/30915932 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f7fb403c030 con 0x55a3bde83470 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.130+0000 7f7fc51af740 10 client.25325 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.130+0000 7f7fc51af740 10 client.25325 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.130+0000 7f7fc51af740 20 client.25325 populate_metadata read hostname 'smithi116' 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.130+0000 7f7fc51af740 1 --2- 172.21.15.116:0/30915932 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a3bdf1c750 0x55a3bdf1eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.130+0000 7f7fc51af740 4 client.25325 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.130+0000 7f7fc51af740 1 -- 172.21.15.116:0/30915932 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a3bdde1470 con 0x55a3bdf1c750 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.130+0000 7f7fbbfff640 1 --2- 172.21.15.116:0/30915932 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a3bdf1c750 0x55a3bdf1eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:32.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.130+0000 7f7f9b7fe640 10 client.25325 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:32.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.130+0000 7f7fbbfff640 1 --2- 172.21.15.116:0/30915932 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a3bdf1c750 0x55a3bdf1eb70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bed90dedbce1a1c0 server_cookie=166668e661489ec2 in_seq=0 out_seq=0 2024-03-19T01:14:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.142+0000 7f7f9b7fe640 1 -- 172.21.15.116:0/30915932 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a3bdde1470 con 0x55a3bdf1c750 2024-03-19T01:14:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.142+0000 7f7f9b7fe640 10 client.25325 handle_command_reply: tid=0 2024-03-19T01:14:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.143+0000 7f7fc51af740 1 client.25325 shutdown 2024-03-19T01:14:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.143+0000 7f7fc51af740 1 -- 172.21.15.116:0/30915932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde83470 msgr2=0x55a3bde8d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.143+0000 7f7fc51af740 1 --2- 172.21.15.116:0/30915932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde83470 0x55a3bde8d690 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7fb40bd030 tx=0x7f7fb40bca00 comp rx=0 tx=0).stop 2024-03-19T01:14:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.143+0000 7f7fc51af740 1 -- 172.21.15.116:0/30915932 shutdown_connections 2024-03-19T01:14:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.143+0000 7f7fc51af740 1 --2- 172.21.15.116:0/30915932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a3bde83470 0x55a3bde8d690 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.143+0000 7f7fc51af740 1 --2- 172.21.15.116:0/30915932 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a3bdf1c750 0x55a3bdf1eb70 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.143+0000 7f7fc51af740 1 --2- 172.21.15.116:0/30915932 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a3bde7e300 0x55a3bde8c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.143+0000 7f7fc51af740 1 --2- 172.21.15.116:0/30915932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a3bde7d910 0x55a3bde8c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.144 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.143+0000 7f7fc51af740 1 -- 172.21.15.116:0/30915932 >> 172.21.15.116:0/30915932 conn(0x55a3bde78e80 msgr2=0x55a3bde7c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.143+0000 7f7f9b7fe640 0 client.25325 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.143+0000 7f7fc51af740 1 -- 172.21.15.116:0/30915932 shutdown_connections 2024-03-19T01:14:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.143+0000 7f7fc51af740 1 -- 172.21.15.116:0/30915932 wait complete. 2024-03-19T01:14:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.143+0000 7f7fc51af740 20 client.25325 trim_cache size 0 max 16384 2024-03-19T01:14:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:14:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:14:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:14:32.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:14:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.144+0000 7f7fc24e8640 1 -- 172.21.15.116:0/980431644 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7fa40a29f0 msgr2=0x7f7fa40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.144+0000 7f7fc24e8640 1 --2- 172.21.15.116:0/980431644 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7fa40a29f0 0x7f7fa40a4eb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x55a3bde7e700 tx=0x7f7fac005f90 comp rx=0 tx=0).stop 2024-03-19T01:14:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.144+0000 7f7fc24e8640 1 -- 172.21.15.116:0/980431644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fbc1075c0 msgr2=0x7f7fbc0727c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.144+0000 7f7fc24e8640 1 --2- 172.21.15.116:0/980431644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fbc1075c0 0x7f7fbc0727c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009bf0 tx=0x7f7fb4032ea0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.144+0000 7f7fc24e8640 1 -- 172.21.15.116:0/980431644 shutdown_connections 2024-03-19T01:14:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.144+0000 7f7fc24e8640 1 --2- 172.21.15.116:0/980431644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fbc1075c0 0x7f7fbc0727c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.144+0000 7f7fc24e8640 1 --2- 172.21.15.116:0/980431644 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7fa40a29f0 0x7f7fa40a4eb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.144+0000 7f7fc24e8640 1 --2- 172.21.15.116:0/980431644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fbc104c90 0x7f7fbc0717d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.144+0000 7f7fc24e8640 1 --2- 172.21.15.116:0/980431644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fbc102360 0x7f7fbc071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.144+0000 7f7fc24e8640 1 -- 172.21.15.116:0/980431644 >> 172.21.15.116:0/980431644 conn(0x7f7fbc0fbcf0 msgr2=0x7f7fbc102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.144+0000 7f7fc24e8640 1 -- 172.21.15.116:0/980431644 shutdown_connections 2024-03-19T01:14:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:32.144+0000 7f7fc24e8640 1 -- 172.21.15.116:0/980431644 wait complete. 2024-03-19T01:14:32.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:32 smithi156 ceph-mon[66799]: pgmap v409: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 782 op/s 2024-03-19T01:14:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:32 smithi162 ceph-mon[67178]: pgmap v409: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 782 op/s 2024-03-19T01:14:32.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:32 smithi116 ceph-mon[62701]: pgmap v409: 97 pgs: 97 active+clean; 1.1 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.9 MiB/s wr, 782 op/s 2024-03-19T01:14:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:34 smithi156 ceph-mon[66799]: pgmap v410: 97 pgs: 97 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 748 op/s 2024-03-19T01:14:34.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:34 smithi162 ceph-mon[67178]: pgmap v410: 97 pgs: 97 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 748 op/s 2024-03-19T01:14:34.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:34 smithi116 ceph-mon[62701]: pgmap v410: 97 pgs: 97 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 748 op/s 2024-03-19T01:14:36.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:35 smithi162 ceph-mon[67178]: pgmap v411: 97 pgs: 97 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 863 op/s 2024-03-19T01:14:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:35 smithi116 ceph-mon[62701]: pgmap v411: 97 pgs: 97 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 863 op/s 2024-03-19T01:14:36.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:35 smithi156 ceph-mon[66799]: pgmap v411: 97 pgs: 97 active+clean; 1.1 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 11 MiB/s wr, 863 op/s 2024-03-19T01:14:36.544 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 110.0 s, 3148.6 tps, lat 31.620 ms stddev 36.239 2024-03-19T01:14:38.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:38 smithi156 ceph-mon[66799]: pgmap v412: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 809 op/s 2024-03-19T01:14:38.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:38 smithi162 ceph-mon[67178]: pgmap v412: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 809 op/s 2024-03-19T01:14:38.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:38 smithi116 ceph-mon[62701]: pgmap v412: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 MiB/s wr, 809 op/s 2024-03-19T01:14:40.451 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:40 smithi162 ceph-mon[67178]: pgmap v413: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.8 MiB/s wr, 763 op/s 2024-03-19T01:14:40.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:40 smithi156 ceph-mon[66799]: pgmap v413: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.8 MiB/s wr, 763 op/s 2024-03-19T01:14:40.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:40 smithi116 ceph-mon[62701]: pgmap v413: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 8.8 MiB/s wr, 763 op/s 2024-03-19T01:14:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 115.0 s, 3448.4 tps, lat 29.086 ms stddev 31.433 2024-03-19T01:14:42.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:42 smithi156 ceph-mon[66799]: pgmap v414: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.1 MiB/s wr, 702 op/s 2024-03-19T01:14:42.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:42 smithi162 ceph-mon[67178]: pgmap v414: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.1 MiB/s wr, 702 op/s 2024-03-19T01:14:42.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:42 smithi116 ceph-mon[62701]: pgmap v414: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 8.1 MiB/s wr, 702 op/s 2024-03-19T01:14:44.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:44 smithi156 ceph-mon[66799]: pgmap v415: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.2 MiB/s wr, 709 op/s 2024-03-19T01:14:44.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:44 smithi162 ceph-mon[67178]: pgmap v415: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.2 MiB/s wr, 709 op/s 2024-03-19T01:14:44.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:44 smithi116 ceph-mon[62701]: pgmap v415: 97 pgs: 97 active+clean; 1.1 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 9.2 MiB/s wr, 709 op/s 2024-03-19T01:14:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.275+0000 7f8dffed1640 1 -- 172.21.15.116:0/2504644805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8df8102360 msgr2=0x7f8df8104750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.275+0000 7f8dffed1640 1 --2- 172.21.15.116:0/2504644805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8df8102360 0x7f8df8104750 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f8df4009900 tx=0x7f8df402bc30 comp rx=0 tx=0).stop 2024-03-19T01:14:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.276+0000 7f8dffed1640 1 -- 172.21.15.116:0/2504644805 shutdown_connections 2024-03-19T01:14:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.276+0000 7f8dffed1640 1 --2- 172.21.15.116:0/2504644805 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8df81075c0 0x7f8df81099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.276+0000 7f8dffed1640 1 --2- 172.21.15.116:0/2504644805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8df8104c90 0x7f8df8107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.276+0000 7f8dffed1640 1 --2- 172.21.15.116:0/2504644805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8df8102360 0x7f8df8104750 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.276+0000 7f8dffed1640 1 -- 172.21.15.116:0/2504644805 >> 172.21.15.116:0/2504644805 conn(0x7f8df80fbcf0 msgr2=0x7f8df80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.276+0000 7f8dffed1640 1 -- 172.21.15.116:0/2504644805 shutdown_connections 2024-03-19T01:14:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.276+0000 7f8dffed1640 1 -- 172.21.15.116:0/2504644805 wait complete. 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.276+0000 7f8dffed1640 1 Processor -- start 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.276+0000 7f8dffed1640 1 -- start start 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.276+0000 7f8dffed1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8df8102360 0x7f8df8197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.276+0000 7f8dffed1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8df8104c90 0x7f8df8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dffed1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8df81075c0 0x7f8df819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dffed1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810d520 con 0x7f8df8104c90 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dffed1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810d3a0 con 0x7f8df8102360 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dffed1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8df810d6a0 con 0x7f8df81075c0 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dfd46c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8df8104c90 0x7f8df8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dfdc6d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8df8102360 0x7f8df8197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dfd46c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8df8104c90 0x7f8df8198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40570/0 (socket says 172.21.15.116:40570) 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dfd46c640 1 -- 172.21.15.116:0/1238891732 learned_addr learned my addr 172.21.15.116:0/1238891732 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8df81075c0 0x7f8df819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dfd46c640 1 -- 172.21.15.116:0/1238891732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8df81075c0 msgr2=0x7f8df819f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dfd46c640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8df81075c0 0x7f8df819f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dfd46c640 1 -- 172.21.15.116:0/1238891732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8df8102360 msgr2=0x7f8df8197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dfd46c640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8df8102360 0x7f8df8197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dfd46c640 1 -- 172.21.15.116:0/1238891732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df819f730 con 0x7f8df8104c90 2024-03-19T01:14:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8df81075c0 0x7f8df819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:14:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dfdc6d640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8df8102360 0x7f8df8197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:14:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dfd46c640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8df8104c90 0x7f8df8198050 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8de800e9c0 tx=0x7f8de800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8deeffd640 1 -- 172.21.15.116:0/1238891732 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de800cd80 con 0x7f8df8104c90 2024-03-19T01:14:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8dffed1640 1 -- 172.21.15.116:0/1238891732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df819fa20 con 0x7f8df8104c90 2024-03-19T01:14:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8deeffd640 1 -- 172.21.15.116:0/1238891732 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de8005500 con 0x7f8df8104c90 2024-03-19T01:14:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.277+0000 7f8deeffd640 1 -- 172.21.15.116:0/1238891732 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8010690 con 0x7f8df8104c90 2024-03-19T01:14:45.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.278+0000 7f8dffed1640 1 -- 172.21.15.116:0/1238891732 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df8100150 con 0x7f8df8104c90 2024-03-19T01:14:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.280+0000 7f8deeffd640 1 -- 172.21.15.116:0/1238891732 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f8de8010830 con 0x7f8df8104c90 2024-03-19T01:14:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.280+0000 7f8e02b98740 1 Processor -- start 2024-03-19T01:14:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.280+0000 7f8deeffd640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8de40a6d30 0x7f8de40a91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.281+0000 7f8deeffd640 1 -- 172.21.15.116:0/1238891732 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8de80cee70 con 0x7f8df8104c90 2024-03-19T01:14:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.281+0000 7f8e02b98740 1 -- start start 2024-03-19T01:14:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.281+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8de40a6d30 0x7f8de40a91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.281+0000 7f8e02b98740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6100 0x55cddd3c6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.281+0000 7f8e02b98740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6af0 0x55cddd3cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.281+0000 7f8e02b98740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3cbc60 0x55cddd3ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.281+0000 7f8e02b98740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cddd32abe0 con 0x55cddd3cbc60 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.281+0000 7f8e02b98740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55cddd2d3960 con 0x55cddd3c6100 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.281+0000 7f8e02b98740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cddd326c80 con 0x55cddd3c6af0 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.282+0000 7f8dfd46c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6af0 0x55cddd3cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.282+0000 7f8dfd46c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6af0 0x55cddd3cb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42010/0 (socket says 172.21.15.116:42010) 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.282+0000 7f8dfd46c640 1 -- 172.21.15.116:0/1568184934 learned_addr learned my addr 172.21.15.116:0/1568184934 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.282+0000 7f8dfdc6d640 1 --2- 172.21.15.116:0/1568184934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6100 0x55cddd3c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.282+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/1568184934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3cbc60 0x55cddd3ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.282+0000 7f8dfd46c640 1 -- 172.21.15.116:0/1568184934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6100 msgr2=0x55cddd3c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.282+0000 7f8dfd46c640 1 --2- 172.21.15.116:0/1568184934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6100 0x55cddd3c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.282+0000 7f8dfd46c640 1 -- 172.21.15.116:0/1568184934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3cbc60 msgr2=0x55cddd3ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.282+0000 7f8dfd46c640 1 --2- 172.21.15.116:0/1568184934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3cbc60 0x55cddd3ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.282+0000 7f8dfd46c640 1 -- 172.21.15.116:0/1568184934 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cddd2cad20 con 0x55cddd3c6af0 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.282+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8de40a6d30 0x7f8de40a91f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8df0005fd0 tx=0x7f8df000f3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:45.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.282+0000 7f8dfd46c640 1 --2- 172.21.15.116:0/1568184934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6af0 0x55cddd3cb5f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8de809b720 tx=0x7f8de809b2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=909baba975db455 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.283+0000 7f8de3fff640 1 -- 172.21.15.116:0/1568184934 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de809b790 con 0x55cddd3c6af0 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.283+0000 7f8de3fff640 1 -- 172.21.15.116:0/1568184934 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de809f8e0 con 0x55cddd3c6af0 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.283+0000 7f8de3fff640 1 -- 172.21.15.116:0/1568184934 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de80a3aa0 con 0x55cddd3c6af0 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.283+0000 7f8e02b98740 1 -- 172.21.15.116:0/1568184934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6af0 msgr2=0x55cddd3cb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.283+0000 7f8e02b98740 1 --2- 172.21.15.116:0/1568184934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6af0 0x55cddd3cb5f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8de809b720 tx=0x7f8de809b2d0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.283+0000 7f8e02b98740 1 -- 172.21.15.116:0/1568184934 shutdown_connections 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.283+0000 7f8e02b98740 1 --2- 172.21.15.116:0/1568184934 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3cbc60 0x55cddd3ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.283+0000 7f8e02b98740 1 --2- 172.21.15.116:0/1568184934 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6af0 0x55cddd3cb5f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.283+0000 7f8e02b98740 1 --2- 172.21.15.116:0/1568184934 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6100 0x55cddd3c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.283+0000 7f8e02b98740 1 -- 172.21.15.116:0/1568184934 >> 172.21.15.116:0/1568184934 conn(0x55cddd3c1670 msgr2=0x55cddd3c1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.284+0000 7f8e02b98740 1 -- 172.21.15.116:0/1568184934 shutdown_connections 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.284+0000 7f8e02b98740 1 -- 172.21.15.116:0/1568184934 wait complete. 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.284+0000 7f8e02b98740 1 -- start start 2024-03-19T01:14:45.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.284+0000 7f8e02b98740 1 Processor -- start 2024-03-19T01:14:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.284+0000 7f8e02b98740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6100 0x55cddd3d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.284+0000 7f8e02b98740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6af0 0x55cddd3d0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.284+0000 7f8e02b98740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3cbc60 0x55cddd3d75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.284+0000 7f8e02b98740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55cddd3ceb50 con 0x55cddd3cbc60 2024-03-19T01:14:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.284+0000 7f8e02b98740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55cddd2e6db0 con 0x55cddd3c6af0 2024-03-19T01:14:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.284+0000 7f8e02b98740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55cddd3cecd0 con 0x55cddd3c6100 2024-03-19T01:14:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.285+0000 7f8e02b98740 10 client.0 _ll_register_callbacks cb 0x55cddd2ebe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:14:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.285+0000 7f8dfe46e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3cbc60 0x55cddd3d75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.285+0000 7f8dfe46e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3cbc60 0x55cddd3d75c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40590/0 (socket says 172.21.15.116:40590) 2024-03-19T01:14:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.285+0000 7f8dfe46e640 1 -- 172.21.15.116:0/4237050624 learned_addr learned my addr 172.21.15.116:0/4237050624 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.285+0000 7f8dfd46c640 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6af0 0x55cddd3d0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.289 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.285+0000 7f8dfdc6d640 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6100 0x55cddd3d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.285+0000 7f8dfe46e640 1 -- 172.21.15.116:0/4237050624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6100 msgr2=0x55cddd3d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.285+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6100 0x55cddd3d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.285+0000 7f8dfe46e640 1 -- 172.21.15.116:0/4237050624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6af0 msgr2=0x55cddd3d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.285+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6af0 0x55cddd3d0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.285+0000 7f8dfe46e640 1 -- 172.21.15.116:0/4237050624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cddd2e5c60 con 0x55cddd3cbc60 2024-03-19T01:14:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.285+0000 7f8dfd46c640 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6af0 0x55cddd3d0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:14:45.293 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.286+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3cbc60 0x55cddd3d75c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8df000fd30 tx=0x7f8df0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ef2e2e39c79dc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.286+0000 7f8de17fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:14:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.286+0000 7f8de17fa640 1 -- 172.21.15.116:0/4237050624 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df0024070 con 0x55cddd3cbc60 2024-03-19T01:14:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.286+0000 7f8de17fa640 1 -- 172.21.15.116:0/4237050624 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8df0003b70 con 0x55cddd3cbc60 2024-03-19T01:14:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.286+0000 7f8e02b98740 1 -- 172.21.15.116:0/4237050624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8de809f8e0 con 0x55cddd3cbc60 2024-03-19T01:14:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.286+0000 7f8de17fa640 1 -- 172.21.15.116:0/4237050624 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8df002e070 con 0x55cddd3cbc60 2024-03-19T01:14:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.286+0000 7f8e02b98740 10 client.15504 fetch_fsmap learned FSMap version 30 2024-03-19T01:14:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.286+0000 7f8e02b98740 1 -- 172.21.15.116:0/4237050624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cddd03f210 con 0x55cddd3cbc60 2024-03-19T01:14:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.287+0000 7f8de17fa640 1 -- 172.21.15.116:0/4237050624 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de809f8e0 con 0x55cddd3cbc60 2024-03-19T01:14:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.287+0000 7f8de17fa640 1 -- 172.21.15.116:0/4237050624 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f8df0020450 con 0x55cddd3cbc60 2024-03-19T01:14:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.287+0000 7f8e02b98740 10 client.15504 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:14:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.287+0000 7f8e02b98740 10 client.15504 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:14:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.287+0000 7f8e02b98740 20 client.15504 populate_metadata read hostname 'smithi116' 2024-03-19T01:14:45.294 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.287+0000 7f8e02b98740 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55cddd3e4790 0x55cddd3e6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.287+0000 7f8e02b98740 4 client.15504 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.287+0000 7f8e02b98740 1 -- 172.21.15.116:0/4237050624 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cddd338120 con 0x55cddd3e4790 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.289+0000 7f8dfdc6d640 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55cddd3e4790 0x55cddd3e6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.290+0000 7f8dfdc6d640 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55cddd3e4790 0x55cddd3e6bc0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d839166a36fc6891 server_cookie=f958131e2541f438 in_seq=0 out_seq=0 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.290+0000 7f8de17fa640 10 client.15504 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.291+0000 7f8de17fa640 1 -- 172.21.15.116:0/4237050624 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cddd338120 con 0x55cddd3e4790 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.291+0000 7f8de17fa640 10 client.15504 handle_command_reply: tid=0 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.291+0000 7f8e02b98740 1 client.15504 shutdown 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.291+0000 7f8e02b98740 1 -- 172.21.15.116:0/4237050624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3cbc60 msgr2=0x55cddd3d75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.291+0000 7f8e02b98740 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3cbc60 0x55cddd3d75c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8df000fd30 tx=0x7f8df0004ab0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.292+0000 7f8e02b98740 1 -- 172.21.15.116:0/4237050624 shutdown_connections 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.292+0000 7f8e02b98740 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3cbc60 0x55cddd3d75c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.292+0000 7f8e02b98740 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6af0 0x55cddd3d0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.295 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.292+0000 7f8e02b98740 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55cddd3e4790 0x55cddd3e6bc0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.292+0000 7f8de17fa640 0 client.15504 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.292+0000 7f8e02b98740 1 --2- 172.21.15.116:0/4237050624 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6100 0x55cddd3d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.292+0000 7f8e02b98740 1 -- 172.21.15.116:0/4237050624 >> 172.21.15.116:0/4237050624 conn(0x55cddd3c1670 msgr2=0x55cddd3c28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.292+0000 7f8e02b98740 1 -- 172.21.15.116:0/4237050624 shutdown_connections 2024-03-19T01:14:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.292+0000 7f8e02b98740 1 -- 172.21.15.116:0/4237050624 wait complete. 2024-03-19T01:14:45.296 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.292+0000 7f8e02b98740 20 client.15504 trim_cache size 0 max 16384 2024-03-19T01:14:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.316+0000 7f8e02b98740 1 Processor -- start 2024-03-19T01:14:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.317+0000 7f8e02b98740 1 -- start start 2024-03-19T01:14:45.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.317+0000 7f8e02b98740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3c6100 0x55cddd3c0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.317+0000 7f8e02b98740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6af0 0x55cddd3d7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.317+0000 7f8dfdc6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3c6100 0x55cddd3c0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.317+0000 7f8dfdc6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3c6100 0x55cddd3c0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40604/0 (socket says 172.21.15.116:40604) 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.317+0000 7f8dfdc6d640 1 -- 172.21.15.116:0/3992910861 learned_addr learned my addr 172.21.15.116:0/3992910861 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.318+0000 7f8dfd46c640 1 --2- 172.21.15.116:0/3992910861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6af0 0x55cddd3d7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.318+0000 7f8e02b98740 1 --2- 172.21.15.116:0/3992910861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3cbc60 0x55cddd3c7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.318+0000 7f8e02b98740 1 -- 172.21.15.116:0/3992910861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8df81a6b90 con 0x55cddd3c6100 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.318+0000 7f8e02b98740 1 -- 172.21.15.116:0/3992910861 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8df81a8820 con 0x55cddd3cbc60 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.318+0000 7f8e02b98740 1 -- 172.21.15.116:0/3992910861 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8df81a7260 con 0x55cddd3c6af0 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.318+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/3992910861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3cbc60 0x55cddd3c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.318+0000 7f8dfe46e640 1 -- 172.21.15.116:0/3992910861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6af0 msgr2=0x55cddd3d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.318+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/3992910861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6af0 0x55cddd3d7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.318+0000 7f8dfe46e640 1 -- 172.21.15.116:0/3992910861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3c6100 msgr2=0x55cddd3c0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.318+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/3992910861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3c6100 0x55cddd3c0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.318+0000 7f8dfe46e640 1 -- 172.21.15.116:0/3992910861 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cddd10b620 con 0x55cddd3cbc60 2024-03-19T01:14:45.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.319+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/3992910861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3cbc60 0x55cddd3c7590 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8df0011fd0 tx=0x7f8df0009630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=575893bc93358a2e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:45.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.319+0000 7f8de37fe640 1 -- 172.21.15.116:0/3992910861 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df0037070 con 0x55cddd3cbc60 2024-03-19T01:14:45.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.319+0000 7f8de37fe640 1 -- 172.21.15.116:0/3992910861 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8df0020d30 con 0x55cddd3cbc60 2024-03-19T01:14:45.321 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.319+0000 7f8de37fe640 1 -- 172.21.15.116:0/3992910861 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df0024070 con 0x55cddd3cbc60 2024-03-19T01:14:45.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.320+0000 7f8e02b98740 1 -- 172.21.15.116:0/3992910861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3cbc60 msgr2=0x55cddd3c7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:45.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.320+0000 7f8e02b98740 1 --2- 172.21.15.116:0/3992910861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3cbc60 0x55cddd3c7590 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8df0011fd0 tx=0x7f8df0009630 comp rx=0 tx=0).stop 2024-03-19T01:14:45.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.320+0000 7f8e02b98740 1 -- 172.21.15.116:0/3992910861 shutdown_connections 2024-03-19T01:14:45.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.320+0000 7f8e02b98740 1 --2- 172.21.15.116:0/3992910861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3cbc60 0x55cddd3c7590 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.320+0000 7f8e02b98740 1 --2- 172.21.15.116:0/3992910861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3c6af0 0x55cddd3d7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.320+0000 7f8e02b98740 1 --2- 172.21.15.116:0/3992910861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3c6100 0x55cddd3c0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.320+0000 7f8e02b98740 1 -- 172.21.15.116:0/3992910861 >> 172.21.15.116:0/3992910861 conn(0x55cddd3c1670 msgr2=0x55cddd3c0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:45.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.320+0000 7f8e02b98740 1 -- 172.21.15.116:0/3992910861 shutdown_connections 2024-03-19T01:14:45.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.320+0000 7f8e02b98740 1 -- 172.21.15.116:0/3992910861 wait complete. 2024-03-19T01:14:45.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.321+0000 7f8e02b98740 1 -- start start 2024-03-19T01:14:45.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.321+0000 7f8e02b98740 1 Processor -- start 2024-03-19T01:14:45.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.322+0000 7f8e02b98740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6100 0x55cddd3d4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.322+0000 7f8e02b98740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3c6af0 0x55cddd3d4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.322+0000 7f8e02b98740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3cbc60 0x55cddd3d5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.322+0000 7f8e02b98740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8df81a8090 con 0x55cddd3c6af0 2024-03-19T01:14:45.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.322+0000 7f8dfdc6d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6100 0x55cddd3d4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.322+0000 7f8e02b98740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8df81a7990 con 0x55cddd3c6100 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.322+0000 7f8dfe46e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3cbc60 0x55cddd3d5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.322+0000 7f8dfe46e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3cbc60 0x55cddd3d5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42032/0 (socket says 172.21.15.116:42032) 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.322+0000 7f8dfe46e640 1 -- 172.21.15.116:0/1691307448 learned_addr learned my addr 172.21.15.116:0/1691307448 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.322+0000 7f8dfd46c640 1 --2- 172.21.15.116:0/1691307448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3c6af0 0x55cddd3d4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.322+0000 7f8e02b98740 1 -- 172.21.15.116:0/1691307448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8df81a4b70 con 0x55cddd3cbc60 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.322+0000 7f8e02b98740 10 client.0 _ll_register_callbacks cb 0x7f8df000fc40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8dfe46e640 1 -- 172.21.15.116:0/1691307448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6100 msgr2=0x55cddd3d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/1691307448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6100 0x55cddd3d4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8dfe46e640 1 -- 172.21.15.116:0/1691307448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3c6af0 msgr2=0x55cddd3d4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/1691307448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3c6af0 0x55cddd3d4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8dfe46e640 1 -- 172.21.15.116:0/1691307448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cddd3d6310 con 0x55cddd3cbc60 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8dfdc6d640 1 --2- 172.21.15.116:0/1691307448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6100 0x55cddd3d4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:14:45.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8dfe46e640 1 --2- 172.21.15.116:0/1691307448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3cbc60 0x55cddd3d5d20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8df0020840 tx=0x7f8df00119e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eac8980a03ccd3db server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8de17fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8de17fa640 1 -- 172.21.15.116:0/1691307448 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df003f070 con 0x55cddd3cbc60 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8de17fa640 1 -- 172.21.15.116:0/1691307448 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8df0004c90 con 0x55cddd3cbc60 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8e02b98740 1 -- 172.21.15.116:0/1691307448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8df0020d30 con 0x55cddd3cbc60 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8de17fa640 1 -- 172.21.15.116:0/1691307448 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8df004b050 con 0x55cddd3cbc60 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8e02b98740 10 client.25318 fetch_fsmap learned FSMap version 30 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8e02b98740 1 -- 172.21.15.116:0/1691307448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55cddd3d7600 con 0x55cddd3cbc60 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.323+0000 7f8de17fa640 1 -- 172.21.15.116:0/1691307448 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8df0020d30 con 0x55cddd3cbc60 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.324+0000 7f8de17fa640 1 -- 172.21.15.116:0/1691307448 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f8df0004650 con 0x55cddd3cbc60 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.324+0000 7f8e02b98740 10 client.25318 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.324+0000 7f8e02b98740 10 client.25318 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.324+0000 7f8e02b98740 20 client.25318 populate_metadata read hostname 'smithi116' 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.324+0000 7f8e02b98740 1 --2- 172.21.15.116:0/1691307448 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55cddd44b030 0x55cddd44d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:45.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.324+0000 7f8e02b98740 4 client.25318 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:14:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.324+0000 7f8e02b98740 1 -- 172.21.15.116:0/1691307448 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cddd332470 con 0x55cddd44b030 2024-03-19T01:14:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.324+0000 7f8dfdc6d640 1 --2- 172.21.15.116:0/1691307448 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55cddd44b030 0x55cddd44d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.324+0000 7f8de17fa640 10 client.25318 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:45.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.324+0000 7f8dfdc6d640 1 --2- 172.21.15.116:0/1691307448 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55cddd44b030 0x55cddd44d450 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3f4f0313bba3d20 server_cookie=73a8447d89b1c3a8 in_seq=0 out_seq=0 2024-03-19T01:14:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8de17fa640 1 -- 172.21.15.116:0/1691307448 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cddd332470 con 0x55cddd44b030 2024-03-19T01:14:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8de17fa640 10 client.25318 handle_command_reply: tid=0 2024-03-19T01:14:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8e02b98740 1 client.25318 shutdown 2024-03-19T01:14:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8e02b98740 1 -- 172.21.15.116:0/1691307448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3cbc60 msgr2=0x55cddd3d5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8e02b98740 1 --2- 172.21.15.116:0/1691307448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3cbc60 0x55cddd3d5d20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8df0020840 tx=0x7f8df00119e0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8e02b98740 1 -- 172.21.15.116:0/1691307448 shutdown_connections 2024-03-19T01:14:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8e02b98740 1 --2- 172.21.15.116:0/1691307448 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55cddd44b030 0x55cddd44d450 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.334 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8e02b98740 1 --2- 172.21.15.116:0/1691307448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55cddd3cbc60 0x55cddd3d5d20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8de17fa640 0 client.25318 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8e02b98740 1 --2- 172.21.15.116:0/1691307448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55cddd3c6af0 0x55cddd3d4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8e02b98740 1 --2- 172.21.15.116:0/1691307448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55cddd3c6100 0x55cddd3d4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8e02b98740 1 -- 172.21.15.116:0/1691307448 >> 172.21.15.116:0/1691307448 conn(0x55cddd3c1670 msgr2=0x55cddd3c0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8e02b98740 1 -- 172.21.15.116:0/1691307448 shutdown_connections 2024-03-19T01:14:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8e02b98740 1 -- 172.21.15.116:0/1691307448 wait complete. 2024-03-19T01:14:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.333+0000 7f8e02b98740 20 client.25318 trim_cache size 0 max 16384 2024-03-19T01:14:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:14:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:14:45.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:14:45.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:14:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.334+0000 7f8dffed1640 1 -- 172.21.15.116:0/1238891732 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8de40a6d30 msgr2=0x7f8de40a91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.334+0000 7f8dffed1640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8de40a6d30 0x7f8de40a91f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8df0005fd0 tx=0x7f8df000f3c0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.334+0000 7f8dffed1640 1 -- 172.21.15.116:0/1238891732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8df8104c90 msgr2=0x7f8df8198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.334+0000 7f8dffed1640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8df8104c90 0x7f8df8198050 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8de800e9c0 tx=0x7f8de800ee80 comp rx=0 tx=0).stop 2024-03-19T01:14:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.334+0000 7f8dffed1640 1 -- 172.21.15.116:0/1238891732 shutdown_connections 2024-03-19T01:14:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.334+0000 7f8dffed1640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8df81075c0 0x7f8df819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.334+0000 7f8dffed1640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8df8104c90 0x7f8df8198050 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.334+0000 7f8dffed1640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8de40a6d30 0x7f8de40a91f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.334+0000 7f8dffed1640 1 --2- 172.21.15.116:0/1238891732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8df8102360 0x7f8df8197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.334+0000 7f8dffed1640 1 -- 172.21.15.116:0/1238891732 >> 172.21.15.116:0/1238891732 conn(0x7f8df80fbcf0 msgr2=0x7f8df8102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:45.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.334+0000 7f8dffed1640 1 -- 172.21.15.116:0/1238891732 shutdown_connections 2024-03-19T01:14:45.338 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:45.334+0000 7f8dffed1640 1 -- 172.21.15.116:0/1238891732 wait complete. 2024-03-19T01:14:46.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:46 smithi156 ceph-mon[66799]: pgmap v416: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 11 MiB/s wr, 817 op/s 2024-03-19T01:14:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 120.0 s, 3147.0 tps, lat 31.770 ms stddev 33.689 2024-03-19T01:14:46.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:46 smithi162 ceph-mon[67178]: pgmap v416: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 11 MiB/s wr, 817 op/s 2024-03-19T01:14:46.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:46 smithi116 ceph-mon[62701]: pgmap v416: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 11 MiB/s wr, 817 op/s 2024-03-19T01:14:48.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:48 smithi156 ceph-mon[66799]: pgmap v417: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 9.1 MiB/s wr, 699 op/s 2024-03-19T01:14:48.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:48 smithi162 ceph-mon[67178]: pgmap v417: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 9.1 MiB/s wr, 699 op/s 2024-03-19T01:14:48.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:48 smithi116 ceph-mon[62701]: pgmap v417: 97 pgs: 97 active+clean; 1.1 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 9.1 MiB/s wr, 699 op/s 2024-03-19T01:14:49.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:49 smithi156 ceph-mon[66799]: pgmap v418: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 8.9 MiB/s wr, 748 op/s 2024-03-19T01:14:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:49 smithi162 ceph-mon[67178]: pgmap v418: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 8.9 MiB/s wr, 748 op/s 2024-03-19T01:14:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:49 smithi116 ceph-mon[62701]: pgmap v418: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 8.9 MiB/s wr, 748 op/s 2024-03-19T01:14:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 125.0 s, 3350.8 tps, lat 29.768 ms stddev 36.059 2024-03-19T01:14:52.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:52 smithi156 ceph-mon[66799]: pgmap v419: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.3 MiB/s wr, 736 op/s 2024-03-19T01:14:52.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:52 smithi162 ceph-mon[67178]: pgmap v419: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.3 MiB/s wr, 736 op/s 2024-03-19T01:14:52.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:52 smithi116 ceph-mon[62701]: pgmap v419: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.3 MiB/s wr, 736 op/s 2024-03-19T01:14:54.456 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:54 smithi116 ceph-mon[62701]: pgmap v420: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 9.9 MiB/s wr, 745 op/s 2024-03-19T01:14:54.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:54 smithi156 ceph-mon[66799]: pgmap v420: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 9.9 MiB/s wr, 745 op/s 2024-03-19T01:14:54.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:54 smithi162 ceph-mon[67178]: pgmap v420: 97 pgs: 97 active+clean; 1.1 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 9.9 MiB/s wr, 745 op/s 2024-03-19T01:14:56.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:56 smithi156 ceph-mon[66799]: pgmap v421: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 11 MiB/s wr, 825 op/s 2024-03-19T01:14:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 130.0 s, 3527.4 tps, lat 28.423 ms stddev 29.347 2024-03-19T01:14:56.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:56 smithi162 ceph-mon[67178]: pgmap v421: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 11 MiB/s wr, 825 op/s 2024-03-19T01:14:56.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:56 smithi116 ceph-mon[62701]: pgmap v421: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 11 MiB/s wr, 825 op/s 2024-03-19T01:14:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:57 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:14:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:57 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:14:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:57 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:14:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.471+0000 7f0fc5360640 1 -- 172.21.15.116:0/1857520678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0103430 msgr2=0x7f0fc0109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.471+0000 7f0fc5360640 1 --2- 172.21.15.116:0/1857520678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0103430 0x7f0fc0109cc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8009ca0 tx=0x7f0fb802c0f0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.471+0000 7f0fc5360640 1 -- 172.21.15.116:0/1857520678 shutdown_connections 2024-03-19T01:14:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.471+0000 7f0fc5360640 1 --2- 172.21.15.116:0/1857520678 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0103430 0x7f0fc0109cc0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.471+0000 7f0fc5360640 1 --2- 172.21.15.116:0/1857520678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fc0102a70 0x7f0fc0102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.471+0000 7f0fc5360640 1 --2- 172.21.15.116:0/1857520678 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fc0101880 0x7f0fc0101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.471+0000 7f0fc5360640 1 -- 172.21.15.116:0/1857520678 >> 172.21.15.116:0/1857520678 conn(0x7f0fc00fcdf0 msgr2=0x7f0fc00ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.471+0000 7f0fc5360640 1 -- 172.21.15.116:0/1857520678 shutdown_connections 2024-03-19T01:14:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.471+0000 7f0fc5360640 1 -- 172.21.15.116:0/1857520678 wait complete. 2024-03-19T01:14:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.472+0000 7f0fc5360640 1 Processor -- start 2024-03-19T01:14:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.472+0000 7f0fc5360640 1 -- start start 2024-03-19T01:14:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.472+0000 7f0fc5360640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0101880 0x7f0fc0197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.472+0000 7f0fc5360640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fc0102a70 0x7f0fc0197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.472+0000 7f0fc5360640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fc0103430 0x7f0fc019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.472+0000 7f0fc5360640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc010d4e0 con 0x7f0fc0101880 2024-03-19T01:14:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.472+0000 7f0fc5360640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc010d360 con 0x7f0fc0103430 2024-03-19T01:14:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.472+0000 7f0fbeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0101880 0x7f0fc0197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.472+0000 7f0fbeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0101880 0x7f0fc0197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50968/0 (socket says 172.21.15.116:50968) 2024-03-19T01:14:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.472+0000 7f0fbeffd640 1 -- 172.21.15.116:0/251922605 learned_addr learned my addr 172.21.15.116:0/251922605 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.472+0000 7f0fbf7fe640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fc0103430 0x7f0fc019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.472+0000 7f0fbe7fc640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fc0102a70 0x7f0fc0197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.472+0000 7f0fc5360640 1 -- 172.21.15.116:0/251922605 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc010d660 con 0x7f0fc0102a70 2024-03-19T01:14:58.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.473+0000 7f0fbeffd640 1 -- 172.21.15.116:0/251922605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fc0102a70 msgr2=0x7f0fc0197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:58.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.473+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fc0102a70 0x7f0fc0197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.473+0000 7f0fbeffd640 1 -- 172.21.15.116:0/251922605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fc0103430 msgr2=0x7f0fc019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:58.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.473+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fc0103430 0x7f0fc019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.473+0000 7f0fbeffd640 1 -- 172.21.15.116:0/251922605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc019f680 con 0x7f0fc0101880 2024-03-19T01:14:58.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.473+0000 7f0fbf7fe640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fc0103430 0x7f0fc019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:14:58.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.473+0000 7f0fbe7fc640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fc0102a70 0x7f0fc0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:14:58.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.473+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0101880 0x7f0fc0197a60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000e9c0 tx=0x7f0fb000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:58.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.473+0000 7f0fa7fff640 1 -- 172.21.15.116:0/251922605 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb000cd80 con 0x7f0fc0101880 2024-03-19T01:14:58.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.474+0000 7f0fa7fff640 1 -- 172.21.15.116:0/251922605 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fb0010910 con 0x7f0fc0101880 2024-03-19T01:14:58.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.474+0000 7f0fc5360640 1 -- 172.21.15.116:0/251922605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc019f970 con 0x7f0fc0101880 2024-03-19T01:14:58.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.474+0000 7f0fc5360640 1 -- 172.21.15.116:0/251922605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc019ff10 con 0x7f0fc0101880 2024-03-19T01:14:58.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.474+0000 7f0fa7fff640 1 -- 172.21.15.116:0/251922605 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb00188c0 con 0x7f0fc0101880 2024-03-19T01:14:58.478 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:58 smithi156 ceph-mon[66799]: pgmap v422: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 753 op/s 2024-03-19T01:14:58.478 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:14:58.478 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:14:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.476+0000 7f0fa7fff640 1 -- 172.21.15.116:0/251922605 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f0fb001f020 con 0x7f0fc0101880 2024-03-19T01:14:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.477+0000 7f0fa7fff640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0fa80a2920 0x7f0fa80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.477+0000 7f0fbe7fc640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0fa80a2920 0x7f0fa80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.477+0000 7f0fc8020740 1 Processor -- start 2024-03-19T01:14:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.477+0000 7f0fc8020740 1 -- start start 2024-03-19T01:14:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.477+0000 7f0fa7fff640 1 -- 172.21.15.116:0/251922605 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0fb00ceba0 con 0x7f0fc0101880 2024-03-19T01:14:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.477+0000 7f0fc8020740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac38930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.477+0000 7f0fc8020740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac38f00 0x55a50ac3da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.477+0000 7f0fc8020740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac3e070 0x55a50ac40500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.477+0000 7f0fc8020740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a50ab98be0 con 0x55a50ac38f00 2024-03-19T01:14:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.477+0000 7f0fc8020740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a50ab41960 con 0x55a50ac3e070 2024-03-19T01:14:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.477+0000 7f0fc8020740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a50ab94c80 con 0x55a50ac38510 2024-03-19T01:14:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbe7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac38f00 0x55a50ac3da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbf7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac3e070 0x55a50ac40500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbe7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac38f00 0x55a50ac3da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50976/0 (socket says 172.21.15.116:50976) 2024-03-19T01:14:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbe7fc640 1 -- 172.21.15.116:0/4157409203 learned_addr learned my addr 172.21.15.116:0/4157409203 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/4157409203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac38930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbe7fc640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0fa80a2920 0x7f0fa80a4de0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0198f70 tx=0x7f0fb4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbf7fe640 1 -- 172.21.15.116:0/4157409203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 msgr2=0x55a50ac38930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbf7fe640 1 --2- 172.21.15.116:0/4157409203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac38930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbf7fe640 1 -- 172.21.15.116:0/4157409203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac38f00 msgr2=0x55a50ac3da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbf7fe640 1 --2- 172.21.15.116:0/4157409203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac38f00 0x55a50ac3da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbf7fe640 1 -- 172.21.15.116:0/4157409203 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a50ab38d20 con 0x55a50ac3e070 2024-03-19T01:14:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbe7fc640 1 --2- 172.21.15.116:0/4157409203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac38f00 0x55a50ac3da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:14:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/4157409203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac38930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:14:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fbf7fe640 1 --2- 172.21.15.116:0/4157409203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac3e070 0x55a50ac40500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8037040 tx=0x7f0fb8005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1a3081dc64a7613 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fa57fa640 1 -- 172.21.15.116:0/4157409203 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8043070 con 0x55a50ac3e070 2024-03-19T01:14:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fa57fa640 1 -- 172.21.15.116:0/4157409203 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fb8032c90 con 0x55a50ac3e070 2024-03-19T01:14:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.478+0000 7f0fa57fa640 1 -- 172.21.15.116:0/4157409203 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8039040 con 0x55a50ac3e070 2024-03-19T01:14:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.479+0000 7f0fc8020740 1 -- 172.21.15.116:0/4157409203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac3e070 msgr2=0x55a50ac40500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.479+0000 7f0fc8020740 1 --2- 172.21.15.116:0/4157409203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac3e070 0x55a50ac40500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8037040 tx=0x7f0fb8005ef0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.479+0000 7f0fc8020740 1 -- 172.21.15.116:0/4157409203 shutdown_connections 2024-03-19T01:14:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.479+0000 7f0fc8020740 1 --2- 172.21.15.116:0/4157409203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac3e070 0x55a50ac40500 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.479+0000 7f0fc8020740 1 --2- 172.21.15.116:0/4157409203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac38f00 0x55a50ac3da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.479+0000 7f0fc8020740 1 --2- 172.21.15.116:0/4157409203 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac38930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.479+0000 7f0fc8020740 1 -- 172.21.15.116:0/4157409203 >> 172.21.15.116:0/4157409203 conn(0x55a50ac33a80 msgr2=0x55a50ac33e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.479+0000 7f0fc8020740 1 -- 172.21.15.116:0/4157409203 shutdown_connections 2024-03-19T01:14:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.479+0000 7f0fc8020740 1 -- 172.21.15.116:0/4157409203 wait complete. 2024-03-19T01:14:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.480+0000 7f0fc8020740 1 -- start start 2024-03-19T01:14:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.480+0000 7f0fc8020740 1 Processor -- start 2024-03-19T01:14:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.480+0000 7f0fc8020740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.480+0000 7f0fc8020740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38f00 0x55a50ac429f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.480+0000 7f0fc8020740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac3e070 0x55a50ac49a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.480+0000 7f0fc8020740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a50ac40fc0 con 0x55a50ac3e070 2024-03-19T01:14:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.480+0000 7f0fc8020740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a50ab54db0 con 0x55a50ac38f00 2024-03-19T01:14:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.480+0000 7f0fc8020740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a50ac41140 con 0x55a50ac38510 2024-03-19T01:14:58.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.480+0000 7f0fc8020740 10 client.0 _ll_register_callbacks cb 0x55a50ab59e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:14:58.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.481+0000 7f0fbe7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38f00 0x55a50ac429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.481+0000 7f0fbe7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38f00 0x55a50ac429f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:42868/0 (socket says 172.21.15.116:42868) 2024-03-19T01:14:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.481+0000 7f0fbe7fc640 1 -- 172.21.15.116:0/86273113 learned_addr learned my addr 172.21.15.116:0/86273113 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.481+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/86273113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.481+0000 7f0fbe7fc640 1 -- 172.21.15.116:0/86273113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 msgr2=0x55a50ac424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.482+0000 7f0fbe7fc640 1 --2- 172.21.15.116:0/86273113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac424b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.482+0000 7f0fbe7fc640 1 -- 172.21.15.116:0/86273113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac3e070 msgr2=0x55a50ac49a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.482+0000 7f0fbe7fc640 1 --2- 172.21.15.116:0/86273113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac3e070 0x55a50ac49a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.482+0000 7f0fbe7fc640 1 -- 172.21.15.116:0/86273113 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a50ab53c60 con 0x55a50ac38f00 2024-03-19T01:14:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.482+0000 7f0fbe7fc640 1 --2- 172.21.15.116:0/86273113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38f00 0x55a50ac429f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4003870 tx=0x7f0fb4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d26282c43967c298 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.482+0000 7f0fa2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:14:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.482+0000 7f0fa2ff5640 1 -- 172.21.15.116:0/86273113 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb400f7d0 con 0x55a50ac38f00 2024-03-19T01:14:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.482+0000 7f0fa2ff5640 1 -- 172.21.15.116:0/86273113 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0fb400f970 con 0x55a50ac38f00 2024-03-19T01:14:58.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.482+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/86273113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.482+0000 7f0fc8020740 1 -- 172.21.15.116:0/86273113 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fb8032c90 con 0x55a50ac38f00 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.483+0000 7f0fa2ff5640 1 -- 172.21.15.116:0/86273113 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fb400fb70 con 0x55a50ac38f00 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.483+0000 7f0fc8020740 10 client.25355 fetch_fsmap learned FSMap version 30 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.483+0000 7f0fc8020740 1 -- 172.21.15.116:0/86273113 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a50a8ad210 con 0x55a50ac38f00 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.483+0000 7f0fa2ff5640 1 -- 172.21.15.116:0/86273113 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fb8032c90 con 0x55a50ac38f00 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.483+0000 7f0fa2ff5640 1 -- 172.21.15.116:0/86273113 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f0fb401a070 con 0x55a50ac38f00 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.483+0000 7f0fc8020740 10 client.25355 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.483+0000 7f0fc8020740 10 client.25355 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.483+0000 7f0fc8020740 20 client.25355 populate_metadata read hostname 'smithi116' 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.483+0000 7f0fc8020740 1 --2- 172.21.15.116:0/86273113 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a50ac5f3f0 0x55a50ac61890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.483+0000 7f0fc8020740 4 client.25355 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.483+0000 7f0fc8020740 1 -- 172.21.15.116:0/86273113 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a50aba6120 con 0x55a50ac5f3f0 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.483+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/86273113 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a50ac5f3f0 0x55a50ac61890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.484+0000 7f0fa2ff5640 10 client.25355 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.484+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/86273113 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a50ac5f3f0 0x55a50ac61890 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a529fcc23a7d0caf server_cookie=668c3a9d66893b76 in_seq=0 out_seq=0 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.484+0000 7f0fa2ff5640 1 -- 172.21.15.116:0/86273113 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a50aba6120 con 0x55a50ac5f3f0 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.484+0000 7f0fa2ff5640 10 client.25355 handle_command_reply: tid=0 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.484+0000 7f0fc8020740 1 client.25355 shutdown 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.484+0000 7f0fc8020740 1 -- 172.21.15.116:0/86273113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38f00 msgr2=0x55a50ac429f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.484+0000 7f0fc8020740 1 --2- 172.21.15.116:0/86273113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38f00 0x55a50ac429f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4003870 tx=0x7f0fb4004730 comp rx=0 tx=0).stop 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.485+0000 7f0fc8020740 1 -- 172.21.15.116:0/86273113 shutdown_connections 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.485+0000 7f0fc8020740 1 --2- 172.21.15.116:0/86273113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac3e070 0x55a50ac49a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.485+0000 7f0fc8020740 1 --2- 172.21.15.116:0/86273113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38f00 0x55a50ac429f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.485+0000 7f0fc8020740 1 --2- 172.21.15.116:0/86273113 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a50ac5f3f0 0x55a50ac61890 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.485+0000 7f0fc8020740 1 --2- 172.21.15.116:0/86273113 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.485+0000 7f0fa2ff5640 0 client.25355 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.485+0000 7f0fc8020740 1 -- 172.21.15.116:0/86273113 >> 172.21.15.116:0/86273113 conn(0x55a50ac33a80 msgr2=0x55a50ac34cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.485+0000 7f0fc8020740 1 -- 172.21.15.116:0/86273113 shutdown_connections 2024-03-19T01:14:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.485+0000 7f0fc8020740 1 -- 172.21.15.116:0/86273113 wait complete. 2024-03-19T01:14:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.485+0000 7f0fc8020740 20 client.25355 trim_cache size 0 max 16384 2024-03-19T01:14:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.505+0000 7f0fc8020740 1 Processor -- start 2024-03-19T01:14:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.505+0000 7f0fc8020740 1 -- start start 2024-03-19T01:14:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.505+0000 7f0fc8020740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38510 0x55a50ac424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.507 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.505+0000 7f0fc8020740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38f00 0x55a50ac499f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fc8020740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac3e070 0x55a50ac429f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fc8020740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc0073f30 con 0x55a50ac3e070 2024-03-19T01:14:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fc8020740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc01a0170 con 0x55a50ac38510 2024-03-19T01:14:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fc8020740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc0074600 con 0x55a50ac38f00 2024-03-19T01:14:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fbeffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38510 0x55a50ac424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fbeffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38510 0x55a50ac424b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:42884/0 (socket says 172.21.15.116:42884) 2024-03-19T01:14:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fbeffd640 1 -- 172.21.15.116:0/3842020676 learned_addr learned my addr 172.21.15.116:0/3842020676 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fbe7fc640 1 --2- 172.21.15.116:0/3842020676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38f00 0x55a50ac499f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fbf7fe640 1 --2- 172.21.15.116:0/3842020676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac3e070 0x55a50ac429f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fbeffd640 1 -- 172.21.15.116:0/3842020676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38f00 msgr2=0x55a50ac499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/3842020676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38f00 0x55a50ac499f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fbeffd640 1 -- 172.21.15.116:0/3842020676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac3e070 msgr2=0x55a50ac429f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:58.508 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/3842020676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac3e070 0x55a50ac429f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fbeffd640 1 -- 172.21.15.116:0/3842020676 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a50a979620 con 0x55a50ac38510 2024-03-19T01:14:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/3842020676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38510 0x55a50ac424b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000c6a0 tx=0x7f0fb00ce040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ac9824b7becfb83 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fa4ff9640 1 -- 172.21.15.116:0/3842020676 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb00995c0 con 0x55a50ac38510 2024-03-19T01:14:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fa4ff9640 1 -- 172.21.15.116:0/3842020676 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55a50aba6120 con 0x55a50ac38510 2024-03-19T01:14:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.506+0000 7f0fa4ff9640 1 -- 172.21.15.116:0/3842020676 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb00a3a90 con 0x55a50ac38510 2024-03-19T01:14:58.509 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.507+0000 7f0fc8020740 1 -- 172.21.15.116:0/3842020676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38510 msgr2=0x55a50ac424b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.507+0000 7f0fc8020740 1 --2- 172.21.15.116:0/3842020676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38510 0x55a50ac424b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000c6a0 tx=0x7f0fb00ce040 comp rx=0 tx=0).stop 2024-03-19T01:14:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.507+0000 7f0fc8020740 1 -- 172.21.15.116:0/3842020676 shutdown_connections 2024-03-19T01:14:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.507+0000 7f0fc8020740 1 --2- 172.21.15.116:0/3842020676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac3e070 0x55a50ac429f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.507+0000 7f0fc8020740 1 --2- 172.21.15.116:0/3842020676 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38f00 0x55a50ac499f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.507+0000 7f0fc8020740 1 --2- 172.21.15.116:0/3842020676 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac38510 0x55a50ac424b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.507+0000 7f0fc8020740 1 -- 172.21.15.116:0/3842020676 >> 172.21.15.116:0/3842020676 conn(0x55a50ac33a80 msgr2=0x55a50ac33e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.508+0000 7f0fc8020740 1 -- 172.21.15.116:0/3842020676 shutdown_connections 2024-03-19T01:14:58.510 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.509+0000 7f0fc8020740 1 -- 172.21.15.116:0/3842020676 wait complete. 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.509+0000 7f0fc8020740 1 -- start start 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.509+0000 7f0fc8020740 1 Processor -- start 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.510+0000 7f0fc8020740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac459d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.510+0000 7f0fc8020740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac38f00 0x55a50ac47f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.510+0000 7f0fc8020740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac3e070 0x55a50ac48460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.510+0000 7f0fc8020740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc0075430 con 0x55a50ac38f00 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.510+0000 7f0fc8020740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc0074d30 con 0x55a50ac3e070 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.510+0000 7f0fc8020740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc0071f10 con 0x55a50ac38510 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.510+0000 7f0fbf7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac3e070 0x55a50ac48460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.510+0000 7f0fbf7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac3e070 0x55a50ac48460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:42892/0 (socket says 172.21.15.116:42892) 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.510+0000 7f0fbf7fe640 1 -- 172.21.15.116:0/3081380341 learned_addr learned my addr 172.21.15.116:0/3081380341 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.510+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/3081380341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.510+0000 7f0fbe7fc640 1 --2- 172.21.15.116:0/3081380341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac38f00 0x55a50ac47f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.510+0000 7f0fc8020740 10 client.0 _ll_register_callbacks cb 0x7f0fb4004190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:14:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fbf7fe640 1 -- 172.21.15.116:0/3081380341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 msgr2=0x55a50ac459d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fbf7fe640 1 --2- 172.21.15.116:0/3081380341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac459d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fbf7fe640 1 -- 172.21.15.116:0/3081380341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac38f00 msgr2=0x55a50ac47f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:14:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fbf7fe640 1 --2- 172.21.15.116:0/3081380341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac38f00 0x55a50ac47f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fbf7fe640 1 -- 172.21.15.116:0/3081380341 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a50ac48a20 con 0x55a50ac3e070 2024-03-19T01:14:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fbf7fe640 1 --2- 172.21.15.116:0/3081380341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac3e070 0x55a50ac48460 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x55a50ac349f0 tx=0x7f0fb8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=beacc91188d925fb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:14:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fa2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:14:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fa2ff5640 1 -- 172.21.15.116:0/3081380341 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8039050 con 0x55a50ac3e070 2024-03-19T01:14:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fa2ff5640 1 -- 172.21.15.116:0/3081380341 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0fb8032690 con 0x55a50ac3e070 2024-03-19T01:14:58.513 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fc8020740 1 -- 172.21.15.116:0/3081380341 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a50aba6120 con 0x55a50ac3e070 2024-03-19T01:14:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fa2ff5640 1 -- 172.21.15.116:0/3081380341 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0fb807a050 con 0x55a50ac3e070 2024-03-19T01:14:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fc8020740 10 client.25361 fetch_fsmap learned FSMap version 30 2024-03-19T01:14:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fc8020740 1 -- 172.21.15.116:0/3081380341 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a50ac49d60 con 0x55a50ac3e070 2024-03-19T01:14:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.511+0000 7f0fa2ff5640 1 -- 172.21.15.116:0/3081380341 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fb8003e50 con 0x55a50ac3e070 2024-03-19T01:14:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.512+0000 7f0fa2ff5640 1 -- 172.21.15.116:0/3081380341 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 30) ==== 8382+0+0 (secure 0 0 0) 0x7f0fb8034bc0 con 0x55a50ac3e070 2024-03-19T01:14:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.512+0000 7f0fc8020740 10 client.25361 fetch_fsmap finished waiting for FSMap version 30 2024-03-19T01:14:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.512+0000 7f0fc8020740 10 client.25361 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:14:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.512+0000 7f0fc8020740 20 client.25361 populate_metadata read hostname 'smithi116' 2024-03-19T01:14:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.512+0000 7f0fc8020740 1 --2- 172.21.15.116:0/3081380341 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a50acc1b70 0x55a50acc3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:14:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.512+0000 7f0fc8020740 4 client.25361 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:14:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.512+0000 7f0fc8020740 1 -- 172.21.15.116:0/3081380341 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a50aba6120 con 0x55a50acc1b70 2024-03-19T01:14:58.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.513+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/3081380341 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a50acc1b70 0x55a50acc3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:14:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.513+0000 7f0fa2ff5640 10 client.25361 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:58.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.513+0000 7f0fbeffd640 1 --2- 172.21.15.116:0/3081380341 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a50acc1b70 0x55a50acc3f90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb8c4fd3836b1b34 server_cookie=9a1d8489a75d5785 in_seq=0 out_seq=0 2024-03-19T01:14:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.521+0000 7f0fa2ff5640 1 -- 172.21.15.116:0/3081380341 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a50aba6120 con 0x55a50acc1b70 2024-03-19T01:14:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.521+0000 7f0fa2ff5640 10 client.25361 handle_command_reply: tid=0 2024-03-19T01:14:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.521+0000 7f0fc8020740 1 client.25361 shutdown 2024-03-19T01:14:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.521+0000 7f0fc8020740 1 -- 172.21.15.116:0/3081380341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac3e070 msgr2=0x55a50ac48460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:58.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.521+0000 7f0fc8020740 1 --2- 172.21.15.116:0/3081380341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac3e070 0x55a50ac48460 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x55a50ac349f0 tx=0x7f0fb8004a40 comp rx=0 tx=0).stop 2024-03-19T01:14:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.521+0000 7f0fc8020740 1 -- 172.21.15.116:0/3081380341 shutdown_connections 2024-03-19T01:14:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.521+0000 7f0fc8020740 1 --2- 172.21.15.116:0/3081380341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a50ac3e070 0x55a50ac48460 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.521+0000 7f0fc8020740 1 --2- 172.21.15.116:0/3081380341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a50ac38f00 0x55a50ac47f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.521+0000 7f0fc8020740 1 --2- 172.21.15.116:0/3081380341 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a50acc1b70 0x55a50acc3f90 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.521+0000 7f0fc8020740 1 --2- 172.21.15.116:0/3081380341 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a50ac38510 0x55a50ac459d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.522+0000 7f0fa2ff5640 0 client.25361 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:14:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.522+0000 7f0fc8020740 1 -- 172.21.15.116:0/3081380341 >> 172.21.15.116:0/3081380341 conn(0x55a50ac33a80 msgr2=0x55a50ac2e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.523+0000 7f0fc8020740 1 -- 172.21.15.116:0/3081380341 shutdown_connections 2024-03-19T01:14:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.523+0000 7f0fc8020740 1 -- 172.21.15.116:0/3081380341 wait complete. 2024-03-19T01:14:58.524 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.523+0000 7f0fc8020740 20 client.25361 trim_cache size 0 max 16384 2024-03-19T01:14:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:14:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:14:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:14:58.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:14:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.524+0000 7f0fc5360640 1 -- 172.21.15.116:0/251922605 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0fa80a2920 msgr2=0x7f0fa80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.524+0000 7f0fc5360640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0fa80a2920 0x7f0fa80a4de0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0198f70 tx=0x7f0fb4005f90 comp rx=0 tx=0).stop 2024-03-19T01:14:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.524+0000 7f0fc5360640 1 -- 172.21.15.116:0/251922605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0101880 msgr2=0x7f0fc0197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:14:58.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.524+0000 7f0fc5360640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0101880 0x7f0fc0197a60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000e9c0 tx=0x7f0fb000ee80 comp rx=0 tx=0).stop 2024-03-19T01:14:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.525+0000 7f0fc5360640 1 -- 172.21.15.116:0/251922605 shutdown_connections 2024-03-19T01:14:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.525+0000 7f0fc5360640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0fa80a2920 0x7f0fa80a4de0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.525+0000 7f0fc5360640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0fc0103430 0x7f0fc019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.525+0000 7f0fc5360640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0fc0102a70 0x7f0fc0197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.525+0000 7f0fc5360640 1 --2- 172.21.15.116:0/251922605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0fc0101880 0x7f0fc0197a60 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:14:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.525+0000 7f0fc5360640 1 -- 172.21.15.116:0/251922605 >> 172.21.15.116:0/251922605 conn(0x7f0fc00fcdf0 msgr2=0x7f0fc0105ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:14:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.525+0000 7f0fc5360640 1 -- 172.21.15.116:0/251922605 shutdown_connections 2024-03-19T01:14:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:14:58.525+0000 7f0fc5360640 1 -- 172.21.15.116:0/251922605 wait complete. 2024-03-19T01:14:58.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:58 smithi162 ceph-mon[67178]: pgmap v422: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 753 op/s 2024-03-19T01:14:58.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:14:58.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:14:58.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:58 smithi116 ceph-mon[62701]: pgmap v422: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 10 MiB/s wr, 753 op/s 2024-03-19T01:14:58.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:14:58.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:14:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:14:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:14:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:14:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:14:59 smithi156 ceph-mon[66799]: pgmap v423: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 MiB/s wr, 851 op/s 2024-03-19T01:14:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:14:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:14:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:14:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:14:59 smithi162 ceph-mon[67178]: pgmap v423: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 MiB/s wr, 851 op/s 2024-03-19T01:14:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:14:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:14:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:14:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:14:59 smithi116 ceph-mon[62701]: pgmap v423: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 13 MiB/s wr, 851 op/s 2024-03-19T01:15:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 135.0 s, 2680.2 tps, lat 37.354 ms stddev 44.490 2024-03-19T01:15:02.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:02 smithi156 ceph-mon[66799]: pgmap v424: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 MiB/s wr, 786 op/s 2024-03-19T01:15:02.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:02 smithi162 ceph-mon[67178]: pgmap v424: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 MiB/s wr, 786 op/s 2024-03-19T01:15:02.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:02 smithi116 ceph-mon[62701]: pgmap v424: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 MiB/s wr, 786 op/s 2024-03-19T01:15:04.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:04 smithi156 ceph-mon[66799]: pgmap v425: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 859 op/s 2024-03-19T01:15:04.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:04 smithi162 ceph-mon[67178]: pgmap v425: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 859 op/s 2024-03-19T01:15:04.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:04 smithi116 ceph-mon[62701]: pgmap v425: 97 pgs: 97 active+clean; 1.1 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 859 op/s 2024-03-19T01:15:06.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:06 smithi156 ceph-mon[66799]: pgmap v426: 97 pgs: 97 active+clean; 1.1 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 MiB/s wr, 937 op/s 2024-03-19T01:15:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 140.0 s, 3234.4 tps, lat 30.832 ms stddev 31.674 2024-03-19T01:15:06.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:06 smithi162 ceph-mon[67178]: pgmap v426: 97 pgs: 97 active+clean; 1.1 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 MiB/s wr, 937 op/s 2024-03-19T01:15:06.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:06 smithi116 ceph-mon[62701]: pgmap v426: 97 pgs: 97 active+clean; 1.1 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 MiB/s wr, 937 op/s 2024-03-19T01:15:07.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:07 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:07.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:07 smithi156 ceph-mon[66799]: pgmap v427: 97 pgs: 97 active+clean; 1.1 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 MiB/s wr, 771 op/s 2024-03-19T01:15:07.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:07 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:07.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:07 smithi162 ceph-mon[67178]: pgmap v427: 97 pgs: 97 active+clean; 1.1 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 MiB/s wr, 771 op/s 2024-03-19T01:15:07.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:07 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:07.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:07 smithi116 ceph-mon[62701]: pgmap v427: 97 pgs: 97 active+clean; 1.1 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 MiB/s wr, 771 op/s 2024-03-19T01:15:10.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:10 smithi156 ceph-mon[66799]: pgmap v428: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 MiB/s wr, 841 op/s 2024-03-19T01:15:10.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:10 smithi162 ceph-mon[67178]: pgmap v428: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 MiB/s wr, 841 op/s 2024-03-19T01:15:10.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:10 smithi116 ceph-mon[62701]: pgmap v428: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 MiB/s wr, 841 op/s 2024-03-19T01:15:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 145.0 s, 3460.6 tps, lat 28.885 ms stddev 33.502 2024-03-19T01:15:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.650+0000 7f6578e61640 1 -- 172.21.15.116:0/4159052250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65741013f0 msgr2=0x7f6574108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.650+0000 7f6578e61640 1 --2- 172.21.15.116:0/4159052250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65741013f0 0x7f6574108c90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f656c009cd0 tx=0x7f656c02c0f0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.650+0000 7f6578e61640 1 -- 172.21.15.116:0/4159052250 shutdown_connections 2024-03-19T01:15:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.650+0000 7f6578e61640 1 --2- 172.21.15.116:0/4159052250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65741013f0 0x7f6574108c90 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.659 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.650+0000 7f6578e61640 1 --2- 172.21.15.116:0/4159052250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6574100a30 0x7f6574100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.650+0000 7f6578e61640 1 --2- 172.21.15.116:0/4159052250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6574101d90 0x7f65741021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.650+0000 7f6578e61640 1 -- 172.21.15.116:0/4159052250 >> 172.21.15.116:0/4159052250 conn(0x7f65740fb480 msgr2=0x7f65740fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.650+0000 7f6578e61640 1 -- 172.21.15.116:0/4159052250 shutdown_connections 2024-03-19T01:15:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.650+0000 7f6578e61640 1 -- 172.21.15.116:0/4159052250 wait complete. 2024-03-19T01:15:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.650+0000 7f6578e61640 1 Processor -- start 2024-03-19T01:15:11.660 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f6578e61640 1 -- start start 2024-03-19T01:15:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f6578e61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6574100a30 0x7f657419e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f6578e61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65741013f0 0x7f657419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f6578e61640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6574101d90 0x7f65741a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f6578e61640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f657410a920 con 0x7f6574100a30 2024-03-19T01:15:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f6578e61640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f657410a7a0 con 0x7f6574101d90 2024-03-19T01:15:11.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f6578e61640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f657410aaa0 con 0x7f65741013f0 2024-03-19T01:15:11.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f657259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6574100a30 0x7f657419e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f657259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6574100a30 0x7f657419e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36466/0 (socket says 172.21.15.116:36466) 2024-03-19T01:15:11.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f657259c640 1 -- 172.21.15.116:0/2345395156 learned_addr learned my addr 172.21.15.116:0/2345395156 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:11.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f656bfff640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65741013f0 0x7f657419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f6572d9d640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6574101d90 0x7f65741a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f657259c640 1 -- 172.21.15.116:0/2345395156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65741013f0 msgr2=0x7f657419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:11.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f657259c640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65741013f0 0x7f657419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f657259c640 1 -- 172.21.15.116:0/2345395156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6574101d90 msgr2=0x7f65741a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:11.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f657259c640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6574101d90 0x7f65741a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f657259c640 1 -- 172.21.15.116:0/2345395156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65741a5d20 con 0x7f6574100a30 2024-03-19T01:15:11.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f657259c640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6574100a30 0x7f657419e100 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f656400e9c0 tx=0x7f656400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:11.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.651+0000 7f656b7fe640 1 -- 172.21.15.116:0/2345395156 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656400cd00 con 0x7f6574100a30 2024-03-19T01:15:11.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.652+0000 7f6578e61640 1 -- 172.21.15.116:0/2345395156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65741a5fb0 con 0x7f6574100a30 2024-03-19T01:15:11.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.652+0000 7f656b7fe640 1 -- 172.21.15.116:0/2345395156 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65640035f0 con 0x7f6574100a30 2024-03-19T01:15:11.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.652+0000 7f656b7fe640 1 -- 172.21.15.116:0/2345395156 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564010690 con 0x7f6574100a30 2024-03-19T01:15:11.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.652+0000 7f6578e61640 1 -- 172.21.15.116:0/2345395156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65741a6470 con 0x7f6574100a30 2024-03-19T01:15:11.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.654+0000 7f656b7fe640 1 -- 172.21.15.116:0/2345395156 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f65640037a0 con 0x7f6574100a30 2024-03-19T01:15:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.654+0000 7f656b7fe640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f65600a28f0 0x7f65600a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.655+0000 7f656b7fe640 1 -- 172.21.15.116:0/2345395156 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f65640cedb0 con 0x7f6574100a30 2024-03-19T01:15:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.655+0000 7f656bfff640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f65600a28f0 0x7f65600a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.655+0000 7f657ba54740 1 Processor -- start 2024-03-19T01:15:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.655+0000 7f656bfff640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f65600a28f0 0x7f65600a4db0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f657419f610 tx=0x7f655c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.655+0000 7f657ba54740 1 -- start start 2024-03-19T01:15:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.655+0000 7f657ba54740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4100 0x561f375c4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.655+0000 7f657ba54740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c4af0 0x561f375c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.655+0000 7f657ba54740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375c9c60 0x561f375cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.655+0000 7f657ba54740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561f37528be0 con 0x561f375c4100 2024-03-19T01:15:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.656+0000 7f657ba54740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561f374d1960 con 0x561f375c9c60 2024-03-19T01:15:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.656+0000 7f657ba54740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561f37524c80 con 0x561f375c4af0 2024-03-19T01:15:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.656+0000 7f657259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4100 0x561f375c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.656+0000 7f657259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4100 0x561f375c4520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36474/0 (socket says 172.21.15.116:36474) 2024-03-19T01:15:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.656+0000 7f657259c640 1 -- 172.21.15.116:0/520217816 learned_addr learned my addr 172.21.15.116:0/520217816 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.656+0000 7f656bfff640 1 --2- 172.21.15.116:0/520217816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c4af0 0x561f375c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.656+0000 7f6572d9d640 1 --2- 172.21.15.116:0/520217816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375c9c60 0x561f375cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.656+0000 7f657259c640 1 -- 172.21.15.116:0/520217816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c4af0 msgr2=0x561f375c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.656+0000 7f657259c640 1 --2- 172.21.15.116:0/520217816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c4af0 0x561f375c95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.656+0000 7f657259c640 1 -- 172.21.15.116:0/520217816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375c9c60 msgr2=0x561f375cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.656+0000 7f657259c640 1 --2- 172.21.15.116:0/520217816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375c9c60 0x561f375cc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.656+0000 7f657259c640 1 -- 172.21.15.116:0/520217816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f374c8d20 con 0x561f375c4100 2024-03-19T01:15:11.666 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.656+0000 7f657259c640 1 --2- 172.21.15.116:0/520217816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4100 0x561f375c4520 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f656409b6c0 tx=0x7f656409b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4297540cafff460 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.657+0000 7f6568ff9640 1 -- 172.21.15.116:0/520217816 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65640b4070 con 0x561f375c4100 2024-03-19T01:15:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.657+0000 7f6568ff9640 1 -- 172.21.15.116:0/520217816 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f656409f880 con 0x561f375c4100 2024-03-19T01:15:11.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.657+0000 7f6568ff9640 1 -- 172.21.15.116:0/520217816 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564003c40 con 0x561f375c4100 2024-03-19T01:15:11.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.659+0000 7f657ba54740 1 -- 172.21.15.116:0/520217816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4100 msgr2=0x561f375c4520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:11.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.659+0000 7f657ba54740 1 --2- 172.21.15.116:0/520217816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4100 0x561f375c4520 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f656409b6c0 tx=0x7f656409b090 comp rx=0 tx=0).stop 2024-03-19T01:15:11.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.659+0000 7f657ba54740 1 -- 172.21.15.116:0/520217816 shutdown_connections 2024-03-19T01:15:11.673 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.659+0000 7f657ba54740 1 --2- 172.21.15.116:0/520217816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375c9c60 0x561f375cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.659+0000 7f657ba54740 1 --2- 172.21.15.116:0/520217816 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c4af0 0x561f375c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.659+0000 7f657ba54740 1 --2- 172.21.15.116:0/520217816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4100 0x561f375c4520 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.659+0000 7f657ba54740 1 -- 172.21.15.116:0/520217816 >> 172.21.15.116:0/520217816 conn(0x561f375bf670 msgr2=0x561f375bfa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.659+0000 7f657ba54740 1 -- 172.21.15.116:0/520217816 shutdown_connections 2024-03-19T01:15:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.659+0000 7f657ba54740 1 -- 172.21.15.116:0/520217816 wait complete. 2024-03-19T01:15:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.659+0000 7f657ba54740 1 -- start start 2024-03-19T01:15:11.674 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657ba54740 1 Processor -- start 2024-03-19T01:15:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657ba54740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4af0 0x561f375ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657ba54740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c9c60 0x561f375ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657ba54740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375d3880 0x561f375d5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657ba54740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561f375ccc10 con 0x561f375c4af0 2024-03-19T01:15:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657ba54740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561f374e4db0 con 0x561f375d3880 2024-03-19T01:15:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4af0 0x561f375ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657ba54740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561f375ccd90 con 0x561f375c9c60 2024-03-19T01:15:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4af0 0x561f375ce310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36482/0 (socket says 172.21.15.116:36482) 2024-03-19T01:15:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657259c640 1 -- 172.21.15.116:0/1915130227 learned_addr learned my addr 172.21.15.116:0/1915130227 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f656bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c9c60 0x561f375ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.675 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657ba54740 10 client.0 _ll_register_callbacks cb 0x561f374e9e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:15:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f6572d9d640 1 --2- 172.21.15.116:0/1915130227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375d3880 0x561f375d5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657259c640 1 -- 172.21.15.116:0/1915130227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c9c60 msgr2=0x561f375ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657259c640 1 --2- 172.21.15.116:0/1915130227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c9c60 0x561f375ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657259c640 1 -- 172.21.15.116:0/1915130227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375d3880 msgr2=0x561f375d5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657259c640 1 --2- 172.21.15.116:0/1915130227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375d3880 0x561f375d5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f657259c640 1 -- 172.21.15.116:0/1915130227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f374e3c60 con 0x561f375c4af0 2024-03-19T01:15:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.660+0000 7f6572d9d640 1 --2- 172.21.15.116:0/1915130227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375d3880 0x561f375d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:15:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f657259c640 1 --2- 172.21.15.116:0/1915130227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4af0 0x561f375ce310 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f65640ccc00 tx=0x7f65640973a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=795eccfe184f0029 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f6559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:15:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f6559ffb640 1 -- 172.21.15.116:0/1915130227 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65640b4040 con 0x561f375c4af0 2024-03-19T01:15:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f6559ffb640 1 -- 172.21.15.116:0/1915130227 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f656409ab80 con 0x561f375c4af0 2024-03-19T01:15:11.676 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f657ba54740 1 -- 172.21.15.116:0/1915130227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f656409f880 con 0x561f375c4af0 2024-03-19T01:15:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f6559ffb640 1 -- 172.21.15.116:0/1915130227 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f656409ad80 con 0x561f375c4af0 2024-03-19T01:15:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f657ba54740 10 client.15537 fetch_fsmap learned FSMap version 32 2024-03-19T01:15:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f657ba54740 1 -- 172.21.15.116:0/1915130227 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561f3723d210 con 0x561f375c4af0 2024-03-19T01:15:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f6559ffb640 1 -- 172.21.15.116:0/1915130227 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f656409f880 con 0x561f375c4af0 2024-03-19T01:15:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f6559ffb640 1 -- 172.21.15.116:0/1915130227 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 32) ==== 8390+0+0 (secure 0 0 0) 0x7f65640a07a0 con 0x561f375c4af0 2024-03-19T01:15:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f657ba54740 10 client.15537 fetch_fsmap finished waiting for FSMap version 32 2024-03-19T01:15:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f657ba54740 10 client.15537 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:15:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f657ba54740 20 client.15537 populate_metadata read hostname 'smithi116' 2024-03-19T01:15:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f657ba54740 1 --2- 172.21.15.116:0/1915130227 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561f375da650 0x561f375dcad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f657ba54740 4 client.15537 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:15:11.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.661+0000 7f657ba54740 1 -- 172.21.15.116:0/1915130227 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f37536120 con 0x561f375da650 2024-03-19T01:15:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.662+0000 7f656bfff640 1 --2- 172.21.15.116:0/1915130227 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561f375da650 0x561f375dcad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.662+0000 7f6559ffb640 10 client.15537 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.662+0000 7f656bfff640 1 --2- 172.21.15.116:0/1915130227 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561f375da650 0x561f375dcad0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fc4c60560367054 server_cookie=3dae37ab7ad99ca4 in_seq=0 out_seq=0 2024-03-19T01:15:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.662+0000 7f6559ffb640 1 -- 172.21.15.116:0/1915130227 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561f37536120 con 0x561f375da650 2024-03-19T01:15:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.662+0000 7f6559ffb640 10 client.15537 handle_command_reply: tid=0 2024-03-19T01:15:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.663+0000 7f657ba54740 1 client.15537 shutdown 2024-03-19T01:15:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.663+0000 7f657ba54740 1 -- 172.21.15.116:0/1915130227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4af0 msgr2=0x561f375ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.663+0000 7f657ba54740 1 --2- 172.21.15.116:0/1915130227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4af0 0x561f375ce310 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f65640ccc00 tx=0x7f65640973a0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.663+0000 7f657ba54740 1 -- 172.21.15.116:0/1915130227 shutdown_connections 2024-03-19T01:15:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.663+0000 7f657ba54740 1 --2- 172.21.15.116:0/1915130227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375d3880 0x561f375d5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.663+0000 7f657ba54740 1 --2- 172.21.15.116:0/1915130227 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561f375da650 0x561f375dcad0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.663+0000 7f657ba54740 1 --2- 172.21.15.116:0/1915130227 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c9c60 0x561f375ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.663+0000 7f657ba54740 1 --2- 172.21.15.116:0/1915130227 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c4af0 0x561f375ce310 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.663+0000 7f657ba54740 1 -- 172.21.15.116:0/1915130227 >> 172.21.15.116:0/1915130227 conn(0x561f375bf670 msgr2=0x561f375c08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.663+0000 7f6559ffb640 0 client.15537 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.663+0000 7f657ba54740 1 -- 172.21.15.116:0/1915130227 shutdown_connections 2024-03-19T01:15:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.663+0000 7f657ba54740 1 -- 172.21.15.116:0/1915130227 wait complete. 2024-03-19T01:15:11.679 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.663+0000 7f657ba54740 20 client.15537 trim_cache size 0 max 16384 2024-03-19T01:15:11.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.688+0000 7f657ba54740 1 Processor -- start 2024-03-19T01:15:11.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f657ba54740 1 -- start start 2024-03-19T01:15:11.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f657ba54740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c4af0 0x561f375beab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f657ba54740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c9c60 0x561f375cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f657ba54740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375d3880 0x561f375ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f657ba54740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f65741af470 con 0x561f375c9c60 2024-03-19T01:15:11.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f657ba54740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f65741b1160 con 0x561f375d3880 2024-03-19T01:15:11.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f657ba54740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f65741afb40 con 0x561f375c4af0 2024-03-19T01:15:11.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f6572d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375d3880 0x561f375ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f656bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c9c60 0x561f375cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f6572d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375d3880 0x561f375ce460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:41624/0 (socket says 172.21.15.116:41624) 2024-03-19T01:15:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f6572d9d640 1 -- 172.21.15.116:0/3520386299 learned_addr learned my addr 172.21.15.116:0/3520386299 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f657259c640 1 --2- 172.21.15.116:0/3520386299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c4af0 0x561f375beab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f6572d9d640 1 -- 172.21.15.116:0/3520386299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c4af0 msgr2=0x561f375beab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f6572d9d640 1 --2- 172.21.15.116:0/3520386299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c4af0 0x561f375beab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f6572d9d640 1 -- 172.21.15.116:0/3520386299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c9c60 msgr2=0x561f375cdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f6572d9d640 1 --2- 172.21.15.116:0/3520386299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c9c60 0x561f375cdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f6572d9d640 1 -- 172.21.15.116:0/3520386299 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f37309620 con 0x561f375d3880 2024-03-19T01:15:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.689+0000 7f6572d9d640 1 --2- 172.21.15.116:0/3520386299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375d3880 0x561f375ce460 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f656c009cd0 tx=0x7f656c005c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aee73f66644c87ba server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:11.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.690+0000 7f655bfff640 1 -- 172.21.15.116:0/3520386299 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c039040 con 0x561f375d3880 2024-03-19T01:15:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.690+0000 7f655bfff640 1 -- 172.21.15.116:0/3520386299 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f656c032c90 con 0x561f375d3880 2024-03-19T01:15:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.690+0000 7f655bfff640 1 -- 172.21.15.116:0/3520386299 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f656c004280 con 0x561f375d3880 2024-03-19T01:15:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.691+0000 7f657ba54740 1 -- 172.21.15.116:0/3520386299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375d3880 msgr2=0x561f375ce460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.691+0000 7f657ba54740 1 --2- 172.21.15.116:0/3520386299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375d3880 0x561f375ce460 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f656c009cd0 tx=0x7f656c005c30 comp rx=0 tx=0).stop 2024-03-19T01:15:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.691+0000 7f657ba54740 1 -- 172.21.15.116:0/3520386299 shutdown_connections 2024-03-19T01:15:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.691+0000 7f657ba54740 1 --2- 172.21.15.116:0/3520386299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375d3880 0x561f375ce460 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.691+0000 7f657ba54740 1 --2- 172.21.15.116:0/3520386299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375c9c60 0x561f375cdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.691+0000 7f657ba54740 1 --2- 172.21.15.116:0/3520386299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c4af0 0x561f375beab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.691+0000 7f657ba54740 1 -- 172.21.15.116:0/3520386299 >> 172.21.15.116:0/3520386299 conn(0x561f375bf670 msgr2=0x561f375bfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.691+0000 7f657ba54740 1 -- 172.21.15.116:0/3520386299 shutdown_connections 2024-03-19T01:15:11.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.691+0000 7f657ba54740 1 -- 172.21.15.116:0/3520386299 wait complete. 2024-03-19T01:15:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.691+0000 7f657ba54740 1 -- start start 2024-03-19T01:15:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.691+0000 7f657ba54740 1 Processor -- start 2024-03-19T01:15:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.692+0000 7f657ba54740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375c4af0 0x561f375eaf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.692+0000 7f657ba54740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c9c60 0x561f375eb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.692+0000 7f657259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375c4af0 0x561f375eaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.692+0000 7f657259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375c4af0 0x561f375eaf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:41634/0 (socket says 172.21.15.116:41634) 2024-03-19T01:15:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.692+0000 7f657259c640 1 -- 172.21.15.116:0/323888550 learned_addr learned my addr 172.21.15.116:0/323888550 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.692+0000 7f656bfff640 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c9c60 0x561f375eb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.692+0000 7f657ba54740 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375d3880 0x561f375ec470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.692+0000 7f657ba54740 1 -- 172.21.15.116:0/323888550 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f65741b09d0 con 0x561f375d3880 2024-03-19T01:15:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.692+0000 7f657ba54740 1 -- 172.21.15.116:0/323888550 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f65741b02d0 con 0x561f375c4af0 2024-03-19T01:15:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.692+0000 7f657ba54740 1 -- 172.21.15.116:0/323888550 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f65741ad450 con 0x561f375c9c60 2024-03-19T01:15:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.693+0000 7f657ba54740 10 client.0 _ll_register_callbacks cb 0x7f655c009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:15:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.693+0000 7f6572d9d640 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375d3880 0x561f375ec470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.693+0000 7f656bfff640 1 -- 172.21.15.116:0/323888550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375c4af0 msgr2=0x561f375eaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.693+0000 7f656bfff640 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375c4af0 0x561f375eaf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.693+0000 7f656bfff640 1 -- 172.21.15.116:0/323888550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375d3880 msgr2=0x561f375ec470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.693+0000 7f656bfff640 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375d3880 0x561f375ec470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.693+0000 7f656bfff640 1 -- 172.21.15.116:0/323888550 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f375eca60 con 0x561f375c9c60 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.693+0000 7f657259c640 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375c4af0 0x561f375eaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.693+0000 7f6572d9d640 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375d3880 0x561f375ec470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.693+0000 7f656bfff640 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c9c60 0x561f375eb460 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x561f375dd380 tx=0x7f655c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2217de2c11ca297 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.694+0000 7f6559ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.694+0000 7f6559ffb640 1 -- 172.21.15.116:0/323888550 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c02a380 con 0x561f375c9c60 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.694+0000 7f6559ffb640 1 -- 172.21.15.116:0/323888550 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f655c02a520 con 0x561f375c9c60 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.694+0000 7f657ba54740 1 -- 172.21.15.116:0/323888550 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f656c032c90 con 0x561f375c9c60 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.694+0000 7f6559ffb640 1 -- 172.21.15.116:0/323888550 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f655c02a720 con 0x561f375c9c60 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.694+0000 7f657ba54740 10 client.25354 fetch_fsmap learned FSMap version 32 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.694+0000 7f657ba54740 1 -- 172.21.15.116:0/323888550 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561f375ede00 con 0x561f375c9c60 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.694+0000 7f6559ffb640 1 -- 172.21.15.116:0/323888550 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f656c032c90 con 0x561f375c9c60 2024-03-19T01:15:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.695+0000 7f6559ffb640 1 -- 172.21.15.116:0/323888550 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 32) ==== 8390+0+0 (secure 0 0 0) 0x7f655c01e4b0 con 0x561f375c9c60 2024-03-19T01:15:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.695+0000 7f657ba54740 10 client.25354 fetch_fsmap finished waiting for FSMap version 32 2024-03-19T01:15:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.695+0000 7f657ba54740 10 client.25354 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:15:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.695+0000 7f657ba54740 20 client.25354 populate_metadata read hostname 'smithi116' 2024-03-19T01:15:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.695+0000 7f657ba54740 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561f37640a20 0x561f37642e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.695+0000 7f657ba54740 4 client.25354 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:15:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.695+0000 7f657ba54740 1 -- 172.21.15.116:0/323888550 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561f37530470 con 0x561f37640a20 2024-03-19T01:15:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.695+0000 7f657259c640 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561f37640a20 0x561f37642e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.695+0000 7f657259c640 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561f37640a20 0x561f37642e40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=805e8723fa7c89fa server_cookie=80ffc3542ae4c225 in_seq=0 out_seq=0 2024-03-19T01:15:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.696+0000 7f6559ffb640 10 client.25354 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.703+0000 7f6559ffb640 1 -- 172.21.15.116:0/323888550 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561f37530470 con 0x561f37640a20 2024-03-19T01:15:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.703+0000 7f6559ffb640 10 client.25354 handle_command_reply: tid=0 2024-03-19T01:15:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.703+0000 7f657ba54740 1 client.25354 shutdown 2024-03-19T01:15:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.704+0000 7f657ba54740 1 -- 172.21.15.116:0/323888550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c9c60 msgr2=0x561f375eb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.704+0000 7f657ba54740 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c9c60 0x561f375eb460 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x561f375dd380 tx=0x7f655c004730 comp rx=0 tx=0).stop 2024-03-19T01:15:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.704+0000 7f657ba54740 1 -- 172.21.15.116:0/323888550 shutdown_connections 2024-03-19T01:15:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.704+0000 7f657ba54740 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561f375d3880 0x561f375ec470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.704+0000 7f657ba54740 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561f37640a20 0x561f37642e40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.704+0000 7f657ba54740 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561f375c9c60 0x561f375eb460 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.704+0000 7f6559ffb640 0 client.25354 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.704+0000 7f657ba54740 1 --2- 172.21.15.116:0/323888550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561f375c4af0 0x561f375eaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.704+0000 7f657ba54740 1 -- 172.21.15.116:0/323888550 >> 172.21.15.116:0/323888550 conn(0x561f375bf670 msgr2=0x561f375cdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.704+0000 7f657ba54740 1 -- 172.21.15.116:0/323888550 shutdown_connections 2024-03-19T01:15:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.704+0000 7f657ba54740 1 -- 172.21.15.116:0/323888550 wait complete. 2024-03-19T01:15:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:15:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:15:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:15:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:15:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.704+0000 7f657ba54740 20 client.25354 trim_cache size 0 max 16384 2024-03-19T01:15:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.705+0000 7f6578e61640 1 -- 172.21.15.116:0/2345395156 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f65600a28f0 msgr2=0x7f65600a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.705+0000 7f6578e61640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f65600a28f0 0x7f65600a4db0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f657419f610 tx=0x7f655c005f90 comp rx=0 tx=0).stop 2024-03-19T01:15:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.705+0000 7f6578e61640 1 -- 172.21.15.116:0/2345395156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6574100a30 msgr2=0x7f657419e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.705+0000 7f6578e61640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6574100a30 0x7f657419e100 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f656400e9c0 tx=0x7f656400ee80 comp rx=0 tx=0).stop 2024-03-19T01:15:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.706+0000 7f6578e61640 1 -- 172.21.15.116:0/2345395156 shutdown_connections 2024-03-19T01:15:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.706+0000 7f6578e61640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f65600a28f0 0x7f65600a4db0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.706+0000 7f6578e61640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6574101d90 0x7f65741a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.706+0000 7f6578e61640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f65741013f0 0x7f657419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.706+0000 7f6578e61640 1 --2- 172.21.15.116:0/2345395156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6574100a30 0x7f657419e100 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.706+0000 7f6578e61640 1 -- 172.21.15.116:0/2345395156 >> 172.21.15.116:0/2345395156 conn(0x7f65740fb480 msgr2=0x7f657410d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.706+0000 7f6578e61640 1 -- 172.21.15.116:0/2345395156 shutdown_connections 2024-03-19T01:15:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:11.706+0000 7f6578e61640 1 -- 172.21.15.116:0/2345395156 wait complete. 2024-03-19T01:15:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:12 smithi162 ceph-mon[67178]: pgmap v429: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 803 op/s 2024-03-19T01:15:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:12 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:12.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:12 smithi156 ceph-mon[66799]: pgmap v429: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 803 op/s 2024-03-19T01:15:12.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:12 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:12.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:12 smithi116 ceph-mon[62701]: pgmap v429: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 MiB/s wr, 803 op/s 2024-03-19T01:15:12.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:12 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:14.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:14 smithi156 ceph-mon[66799]: pgmap v430: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 808 op/s 2024-03-19T01:15:14.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:14 smithi162 ceph-mon[67178]: pgmap v430: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 808 op/s 2024-03-19T01:15:14.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:14 smithi116 ceph-mon[62701]: pgmap v430: 97 pgs: 97 active+clean; 1.2 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 808 op/s 2024-03-19T01:15:16.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:16 smithi156 ceph-mon[66799]: pgmap v431: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.5 MiB/s wr, 751 op/s 2024-03-19T01:15:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 150.0 s, 3384.2 tps, lat 29.582 ms stddev 31.152 2024-03-19T01:15:16.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:16 smithi162 ceph-mon[67178]: pgmap v431: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.5 MiB/s wr, 751 op/s 2024-03-19T01:15:16.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:16 smithi116 ceph-mon[62701]: pgmap v431: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.5 MiB/s wr, 751 op/s 2024-03-19T01:15:17.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:17 smithi156 ceph-mon[66799]: pgmap v432: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 647 op/s 2024-03-19T01:15:17.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:17 smithi162 ceph-mon[67178]: pgmap v432: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 647 op/s 2024-03-19T01:15:17.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:17 smithi116 ceph-mon[62701]: pgmap v432: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 MiB/s wr, 647 op/s 2024-03-19T01:15:20.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:20 smithi156 ceph-mon[66799]: pgmap v433: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.5 MiB/s wr, 735 op/s 2024-03-19T01:15:20.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:20 smithi162 ceph-mon[67178]: pgmap v433: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.5 MiB/s wr, 735 op/s 2024-03-19T01:15:20.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:20 smithi116 ceph-mon[62701]: pgmap v433: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.5 MiB/s wr, 735 op/s 2024-03-19T01:15:21.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:21 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:21.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:21 smithi156 ceph-mon[66799]: pgmap v434: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.6 MiB/s wr, 721 op/s 2024-03-19T01:15:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 155.0 s, 3461.8 tps, lat 28.907 ms stddev 30.685 2024-03-19T01:15:21.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:21 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:21.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:21 smithi162 ceph-mon[67178]: pgmap v434: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.6 MiB/s wr, 721 op/s 2024-03-19T01:15:21.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:21 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:21.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:21 smithi116 ceph-mon[62701]: pgmap v434: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.6 MiB/s wr, 721 op/s 2024-03-19T01:15:24.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:24 smithi156 ceph-mon[66799]: pgmap v435: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.8 MiB/s wr, 710 op/s 2024-03-19T01:15:24.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:24 smithi162 ceph-mon[67178]: pgmap v435: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.8 MiB/s wr, 710 op/s 2024-03-19T01:15:24.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:24 smithi116 ceph-mon[62701]: pgmap v435: 97 pgs: 97 active+clean; 1.2 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 8.8 MiB/s wr, 710 op/s 2024-03-19T01:15:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.823+0000 7f3b6d24b640 1 -- 172.21.15.116:0/610879509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b681000a0 msgr2=0x7f3b681004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:24.825 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.823+0000 7f3b6d24b640 1 --2- 172.21.15.116:0/610879509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b681000a0 0x7f3b681004c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3b58009890 tx=0x7f3b5802bb70 comp rx=0 tx=0).stop 2024-03-19T01:15:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.823+0000 7f3b6d24b640 1 -- 172.21.15.116:0/610879509 shutdown_connections 2024-03-19T01:15:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.823+0000 7f3b6d24b640 1 --2- 172.21.15.116:0/610879509 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b681019b0 0x7f3b681082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.823+0000 7f3b6d24b640 1 --2- 172.21.15.116:0/610879509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b68100ff0 0x7f3b68101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.823+0000 7f3b6d24b640 1 --2- 172.21.15.116:0/610879509 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b681000a0 0x7f3b681004c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.823+0000 7f3b6d24b640 1 -- 172.21.15.116:0/610879509 >> 172.21.15.116:0/610879509 conn(0x7f3b680fb000 msgr2=0x7f3b680fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.823+0000 7f3b6d24b640 1 -- 172.21.15.116:0/610879509 shutdown_connections 2024-03-19T01:15:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.823+0000 7f3b6d24b640 1 -- 172.21.15.116:0/610879509 wait complete. 2024-03-19T01:15:24.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.824+0000 7f3b6d24b640 1 Processor -- start 2024-03-19T01:15:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.824+0000 7f3b6d24b640 1 -- start start 2024-03-19T01:15:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.824+0000 7f3b6d24b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b681000a0 0x7f3b68103e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.824+0000 7f3b6d24b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b68100ff0 0x7f3b68102460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.825+0000 7f3b6d24b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b681019b0 0x7f3b681029a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.825+0000 7f3b6d24b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6810b2d0 con 0x7f3b681019b0 2024-03-19T01:15:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.825+0000 7f3b6d24b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6810b150 con 0x7f3b68100ff0 2024-03-19T01:15:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.825+0000 7f3b6d24b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3b6810b450 con 0x7f3b681000a0 2024-03-19T01:15:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.825+0000 7f3b6759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b681019b0 0x7f3b681029a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.825+0000 7f3b6659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b68100ff0 0x7f3b68102460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.825+0000 7f3b6759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b681019b0 0x7f3b681029a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58570/0 (socket says 172.21.15.116:58570) 2024-03-19T01:15:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.825+0000 7f3b6759e640 1 -- 172.21.15.116:0/1426434841 learned_addr learned my addr 172.21.15.116:0/1426434841 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.825+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b681000a0 0x7f3b68103e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.826+0000 7f3b6759e640 1 -- 172.21.15.116:0/1426434841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b681000a0 msgr2=0x7f3b68103e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.826+0000 7f3b6759e640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b681000a0 0x7f3b68103e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.826+0000 7f3b6759e640 1 -- 172.21.15.116:0/1426434841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b68100ff0 msgr2=0x7f3b68102460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.826+0000 7f3b6759e640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b68100ff0 0x7f3b68102460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.826+0000 7f3b6759e640 1 -- 172.21.15.116:0/1426434841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b68103230 con 0x7f3b681019b0 2024-03-19T01:15:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.826+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b681000a0 0x7f3b68103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:15:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.826+0000 7f3b6659c640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b68100ff0 0x7f3b68102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:15:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.826+0000 7f3b6759e640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b681019b0 0x7f3b681029a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3b60005c40 tx=0x7f3b600095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.827+0000 7f3b4ffff640 1 -- 172.21.15.116:0/1426434841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b60003970 con 0x7f3b681019b0 2024-03-19T01:15:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.827+0000 7f3b4ffff640 1 -- 172.21.15.116:0/1426434841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b60004020 con 0x7f3b681019b0 2024-03-19T01:15:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.827+0000 7f3b4ffff640 1 -- 172.21.15.116:0/1426434841 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b60015670 con 0x7f3b681019b0 2024-03-19T01:15:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.827+0000 7f3b6d24b640 1 -- 172.21.15.116:0/1426434841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b681977e0 con 0x7f3b681019b0 2024-03-19T01:15:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.827+0000 7f3b6d24b640 1 -- 172.21.15.116:0/1426434841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b68197ce0 con 0x7f3b681019b0 2024-03-19T01:15:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.831+0000 7f3b6fe39740 1 Processor -- start 2024-03-19T01:15:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.831+0000 7f3b4ffff640 1 -- 172.21.15.116:0/1426434841 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f3b60003b10 con 0x7f3b681019b0 2024-03-19T01:15:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.831+0000 7f3b6fe39740 1 -- start start 2024-03-19T01:15:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.831+0000 7f3b6fe39740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75100 0x559e9fe75520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.831+0000 7f3b6fe39740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75af0 0x559e9fe7a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.831+0000 7f3b6fe39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 0x559e9fe7d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.831+0000 7f3b6fe39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559e9fdd9be0 con 0x559e9fe7ac60 2024-03-19T01:15:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.831+0000 7f3b6fe39740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559e9fd82960 con 0x559e9fe75100 2024-03-19T01:15:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.831+0000 7f3b6fe39740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559e9fdd5c80 con 0x559e9fe75af0 2024-03-19T01:15:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.831+0000 7f3b6659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75100 0x559e9fe75520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.831+0000 7f3b6659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75100 0x559e9fe75520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:50636/0 (socket says 172.21.15.116:50636) 2024-03-19T01:15:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.832+0000 7f3b6659c640 1 -- 172.21.15.116:0/2749843668 learned_addr learned my addr 172.21.15.116:0/2749843668 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.832+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/2749843668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75af0 0x559e9fe7a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.832+0000 7f3b6659c640 1 --2- 172.21.15.116:0/2749843668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 0x559e9fe7d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.832+0000 7f3b6659c640 1 -- 172.21.15.116:0/2749843668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75af0 msgr2=0x559e9fe7a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.832+0000 7f3b6659c640 1 --2- 172.21.15.116:0/2749843668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75af0 0x559e9fe7a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.832+0000 7f3b6659c640 1 -- 172.21.15.116:0/2749843668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 msgr2=0x559e9fe7d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.832+0000 7f3b6659c640 1 --2- 172.21.15.116:0/2749843668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 0x559e9fe7d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.835 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.832+0000 7f3b6659c640 1 -- 172.21.15.116:0/2749843668 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e9fd79d20 con 0x559e9fe75100 2024-03-19T01:15:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.832+0000 7f3b4ffff640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3b500a2a90 0x7f3b500a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.832+0000 7f3b6659c640 1 --2- 172.21.15.116:0/2749843668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75100 0x559e9fe75520 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c0126c0 tx=0x7f3b5c009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26df8ab0a09713b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.833+0000 7f3b4ffff640 1 -- 172.21.15.116:0/1426434841 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f3b600ce1c0 con 0x7f3b681019b0 2024-03-19T01:15:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.833+0000 7f3b4d7fa640 1 -- 172.21.15.116:0/2749843668 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c010ad0 con 0x559e9fe75100 2024-03-19T01:15:24.837 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.833+0000 7f3b4d7fa640 1 -- 172.21.15.116:0/2749843668 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b5c010c70 con 0x559e9fe75100 2024-03-19T01:15:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.833+0000 7f3b4d7fa640 1 -- 172.21.15.116:0/2749843668 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c01c5a0 con 0x559e9fe75100 2024-03-19T01:15:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.833+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3b500a2a90 0x7f3b500a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.834+0000 7f3b6fe39740 1 -- 172.21.15.116:0/2749843668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75100 msgr2=0x559e9fe75520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.834+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/2749843668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75100 0x559e9fe75520 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c0126c0 tx=0x7f3b5c009880 comp rx=0 tx=0).stop 2024-03-19T01:15:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.834+0000 7f3b6fe39740 1 -- 172.21.15.116:0/2749843668 shutdown_connections 2024-03-19T01:15:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.834+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/2749843668 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 0x559e9fe7d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.834+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/2749843668 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75af0 0x559e9fe7a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.838 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.834+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/2749843668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75100 0x559e9fe75520 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.834+0000 7f3b6fe39740 1 -- 172.21.15.116:0/2749843668 >> 172.21.15.116:0/2749843668 conn(0x559e9fe70670 msgr2=0x559e9fe70a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.834+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3b500a2a90 0x7f3b500a4f50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3b580097b0 tx=0x7f3b58005a90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.834+0000 7f3b6fe39740 1 -- 172.21.15.116:0/2749843668 shutdown_connections 2024-03-19T01:15:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.834+0000 7f3b6fe39740 1 -- 172.21.15.116:0/2749843668 wait complete. 2024-03-19T01:15:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.835+0000 7f3b6fe39740 1 -- start start 2024-03-19T01:15:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.835+0000 7f3b6fe39740 1 Processor -- start 2024-03-19T01:15:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.835+0000 7f3b6fe39740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75100 0x559e9fe7f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.835+0000 7f3b6fe39740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75af0 0x559e9fe7f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.835+0000 7f3b6fe39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 0x559e9fe86770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.835+0000 7f3b6759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 0x559e9fe86770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.835+0000 7f3b6759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 0x559e9fe86770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58590/0 (socket says 172.21.15.116:58590) 2024-03-19T01:15:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.835+0000 7f3b6759e640 1 -- 172.21.15.116:0/3030168477 learned_addr learned my addr 172.21.15.116:0/3030168477 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:24.839 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.835+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3030168477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559e9fe7dc10 con 0x559e9fe7ac60 2024-03-19T01:15:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.835+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3030168477 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559e9fd95db0 con 0x559e9fe75af0 2024-03-19T01:15:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.835+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3030168477 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559e9fe7dd90 con 0x559e9fe75100 2024-03-19T01:15:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.835+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75100 0x559e9fe7f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.835+0000 7f3b6659c640 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75af0 0x559e9fe7f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.836+0000 7f3b6fe39740 10 client.0 _ll_register_callbacks cb 0x559e9fd9ae70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:15:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.836+0000 7f3b6759e640 1 -- 172.21.15.116:0/3030168477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75100 msgr2=0x559e9fe7f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.836+0000 7f3b6759e640 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75100 0x559e9fe7f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.836+0000 7f3b6759e640 1 -- 172.21.15.116:0/3030168477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75af0 msgr2=0x559e9fe7f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.836+0000 7f3b6759e640 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75af0 0x559e9fe7f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.836+0000 7f3b6759e640 1 -- 172.21.15.116:0/3030168477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e9fd94c60 con 0x559e9fe7ac60 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.836+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75100 0x559e9fe7f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.836+0000 7f3b6659c640 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75af0 0x559e9fe7f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.836+0000 7f3b6759e640 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 0x559e9fe86770 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3b6000a380 tx=0x7f3b6009b320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c524cd10d1aef2e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.836+0000 7f3b4aff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.837+0000 7f3b4aff5640 1 -- 172.21.15.116:0/3030168477 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b600b3060 con 0x559e9fe7ac60 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.837+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3030168477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b5c010c70 con 0x559e9fe7ac60 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.837+0000 7f3b4aff5640 1 -- 172.21.15.116:0/3030168477 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f3b60099f30 con 0x559e9fe7ac60 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.837+0000 7f3b4aff5640 1 -- 172.21.15.116:0/3030168477 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3b600a3280 con 0x559e9fe7ac60 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.837+0000 7f3b6fe39740 10 client.15552 fetch_fsmap learned FSMap version 33 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.837+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3030168477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559e9faee210 con 0x559e9fe7ac60 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.837+0000 7f3b4aff5640 1 -- 172.21.15.116:0/3030168477 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b5c010c70 con 0x559e9fe7ac60 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.837+0000 7f3b4aff5640 1 -- 172.21.15.116:0/3030168477 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8386+0+0 (secure 0 0 0) 0x7f3b6009fb50 con 0x559e9fe7ac60 2024-03-19T01:15:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.838+0000 7f3b6fe39740 10 client.15552 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.838+0000 7f3b6fe39740 10 client.15552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.838+0000 7f3b6fe39740 20 client.15552 populate_metadata read hostname 'smithi116' 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.838+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e9fe8b670 0x559e9fe8dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.838+0000 7f3b6fe39740 4 client.15552 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.838+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3030168477 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e9fde7120 con 0x559e9fe8b670 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.838+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e9fe8b670 0x559e9fe8dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.839+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e9fe8b670 0x559e9fe8dad0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25fa8468ac2c69f6 server_cookie=7cfe398bb8f10eed in_seq=0 out_seq=0 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.839+0000 7f3b4aff5640 10 client.15552 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.840+0000 7f3b4aff5640 1 -- 172.21.15.116:0/3030168477 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559e9fde7120 con 0x559e9fe8b670 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.840+0000 7f3b4aff5640 10 client.15552 handle_command_reply: tid=0 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.840+0000 7f3b6fe39740 1 client.15552 shutdown 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.840+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3030168477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 msgr2=0x559e9fe86770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.840+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 0x559e9fe86770 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f3b6000a380 tx=0x7f3b6009b320 comp rx=0 tx=0).stop 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.840+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3030168477 shutdown_connections 2024-03-19T01:15:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.840+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 0x559e9fe86770 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.840+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75af0 0x559e9fe7f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.840+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e9fe8b670 0x559e9fe8dad0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.840+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3030168477 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75100 0x559e9fe7f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.840+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3030168477 >> 172.21.15.116:0/3030168477 conn(0x559e9fe70670 msgr2=0x559e9fe718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.841+0000 7f3b4aff5640 0 client.15552 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.841+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3030168477 shutdown_connections 2024-03-19T01:15:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.841+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3030168477 wait complete. 2024-03-19T01:15:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.841+0000 7f3b6fe39740 20 client.15552 trim_cache size 0 max 16384 2024-03-19T01:15:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.863+0000 7f3b6fe39740 1 Processor -- start 2024-03-19T01:15:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.863+0000 7f3b6fe39740 1 -- start start 2024-03-19T01:15:24.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.863+0000 7f3b6fe39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe75100 0x559e9fe6f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6fe39740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75af0 0x559e9fe86730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6fe39740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe7ac60 0x559e9fe6f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6fe39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3b681af380 con 0x559e9fe75100 2024-03-19T01:15:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6fe39740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3b681b1040 con 0x559e9fe7ac60 2024-03-19T01:15:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6fe39740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3b681afa50 con 0x559e9fe75af0 2024-03-19T01:15:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b66d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe75100 0x559e9fe6f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75af0 0x559e9fe86730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b66d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe75100 0x559e9fe6f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58606/0 (socket says 172.21.15.116:58606) 2024-03-19T01:15:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b66d9d640 1 -- 172.21.15.116:0/3844733655 learned_addr learned my addr 172.21.15.116:0/3844733655 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6759e640 1 --2- 172.21.15.116:0/3844733655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe7ac60 0x559e9fe6f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6759e640 1 -- 172.21.15.116:0/3844733655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75af0 msgr2=0x559e9fe86730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6759e640 1 --2- 172.21.15.116:0/3844733655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75af0 0x559e9fe86730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6759e640 1 -- 172.21.15.116:0/3844733655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe75100 msgr2=0x559e9fe6f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6759e640 1 --2- 172.21.15.116:0/3844733655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe75100 0x559e9fe6f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6759e640 1 -- 172.21.15.116:0/3844733655 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e9fbba620 con 0x559e9fe7ac60 2024-03-19T01:15:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/3844733655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe75100 0x559e9fe6f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:15:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6659c640 1 --2- 172.21.15.116:0/3844733655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75af0 0x559e9fe86730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:15:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.864+0000 7f3b6759e640 1 --2- 172.21.15.116:0/3844733655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe7ac60 0x559e9fe6f760 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3b6009aa70 tx=0x7f3b6000a310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfe5c95e8e7aa47e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.865+0000 7f3b4cff9640 1 -- 172.21.15.116:0/3844733655 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6009a8b0 con 0x559e9fe7ac60 2024-03-19T01:15:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.865+0000 7f3b4cff9640 1 -- 172.21.15.116:0/3844733655 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b60099d40 con 0x559e9fe7ac60 2024-03-19T01:15:24.868 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.865+0000 7f3b4cff9640 1 -- 172.21.15.116:0/3844733655 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b600b3ba0 con 0x559e9fe7ac60 2024-03-19T01:15:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3844733655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe7ac60 msgr2=0x559e9fe6f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:24.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3844733655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe7ac60 0x559e9fe6f760 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3b6009aa70 tx=0x7f3b6000a310 comp rx=0 tx=0).stop 2024-03-19T01:15:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3844733655 shutdown_connections 2024-03-19T01:15:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3844733655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe7ac60 0x559e9fe6f760 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3844733655 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75af0 0x559e9fe86730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3844733655 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe75100 0x559e9fe6f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3844733655 >> 172.21.15.116:0/3844733655 conn(0x559e9fe70670 msgr2=0x559e9fe7f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3844733655 shutdown_connections 2024-03-19T01:15:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3844733655 wait complete. 2024-03-19T01:15:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 -- start start 2024-03-19T01:15:24.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 Processor -- start 2024-03-19T01:15:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75100 0x559e9fe83350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75af0 0x559e9fe83890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 0x559e9feef9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3b681b08b0 con 0x559e9fe7ac60 2024-03-19T01:15:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3b681b01e0 con 0x559e9fe75af0 2024-03-19T01:15:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3b681ad360 con 0x559e9fe75100 2024-03-19T01:15:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.866+0000 7f3b6fe39740 10 client.0 _ll_register_callbacks cb 0x7f3b60096e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:15:24.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.867+0000 7f3b6659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75af0 0x559e9fe83890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.867+0000 7f3b6659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75af0 0x559e9fe83890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:50652/0 (socket says 172.21.15.116:50652) 2024-03-19T01:15:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.867+0000 7f3b6659c640 1 -- 172.21.15.116:0/3646006780 learned_addr learned my addr 172.21.15.116:0/3646006780 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.867+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/3646006780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75100 0x559e9fe83350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.867+0000 7f3b6659c640 1 -- 172.21.15.116:0/3646006780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75100 msgr2=0x559e9fe83350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.867+0000 7f3b6659c640 1 --2- 172.21.15.116:0/3646006780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75100 0x559e9fe83350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.867+0000 7f3b6659c640 1 -- 172.21.15.116:0/3646006780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 msgr2=0x559e9feef9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.867+0000 7f3b6659c640 1 --2- 172.21.15.116:0/3646006780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 0x559e9feef9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.867+0000 7f3b6659c640 1 -- 172.21.15.116:0/3646006780 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e9fe86960 con 0x559e9fe75af0 2024-03-19T01:15:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.867+0000 7f3b6659c640 1 --2- 172.21.15.116:0/3646006780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75af0 0x559e9fe83890 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x559e9fe76350 tx=0x7f3b5c009c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcf44ea422e0bea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:24.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.867+0000 7f3b4aff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:15:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b4aff5640 1 -- 172.21.15.116:0/3646006780 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c02d060 con 0x559e9fe75af0 2024-03-19T01:15:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b4aff5640 1 -- 172.21.15.116:0/3646006780 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f3b5c0054c0 con 0x559e9fe75af0 2024-03-19T01:15:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3646006780 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b60099d40 con 0x559e9fe75af0 2024-03-19T01:15:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b4aff5640 1 -- 172.21.15.116:0/3646006780 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f3b5c0035f0 con 0x559e9fe75af0 2024-03-19T01:15:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b6fe39740 10 client.25397 fetch_fsmap learned FSMap version 33 2024-03-19T01:15:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3646006780 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x559e9fef0f30 con 0x559e9fe75af0 2024-03-19T01:15:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b4aff5640 1 -- 172.21.15.116:0/3646006780 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b5c003c70 con 0x559e9fe75af0 2024-03-19T01:15:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b4aff5640 1 -- 172.21.15.116:0/3646006780 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 33) ==== 8386+0+0 (secure 0 0 0) 0x7f3b5c01c020 con 0x559e9fe75af0 2024-03-19T01:15:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b6fe39740 10 client.25397 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T01:15:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b6fe39740 10 client.25397 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:15:24.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b6fe39740 20 client.25397 populate_metadata read hostname 'smithi116' 2024-03-19T01:15:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3646006780 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e9fef16f0 0x559e9fef3b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b6fe39740 4 client.25397 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:15:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.868+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3646006780 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f3b60099d40 con 0x559e9fef16f0 2024-03-19T01:15:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.869+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/3646006780 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e9fef16f0 0x559e9fef3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.870+0000 7f3b66d9d640 1 --2- 172.21.15.116:0/3646006780 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e9fef16f0 0x559e9fef3b10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d431f07ee49c8a2 server_cookie=b57a0f306b10fcdc in_seq=0 out_seq=0 2024-03-19T01:15:24.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.870+0000 7f3b4aff5640 10 client.25397 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.877+0000 7f3b4aff5640 1 -- 172.21.15.116:0/3646006780 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f3b60099d40 con 0x559e9fef16f0 2024-03-19T01:15:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.877+0000 7f3b4aff5640 10 client.25397 handle_command_reply: tid=0 2024-03-19T01:15:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.877+0000 7f3b6fe39740 1 client.25397 shutdown 2024-03-19T01:15:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.878+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3646006780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75af0 msgr2=0x559e9fe83890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.878+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3646006780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75af0 0x559e9fe83890 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x559e9fe76350 tx=0x7f3b5c009c60 comp rx=0 tx=0).stop 2024-03-19T01:15:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.878+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3646006780 shutdown_connections 2024-03-19T01:15:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.878+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3646006780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e9fe7ac60 0x559e9feef9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.878+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3646006780 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e9fe75af0 0x559e9fe83890 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.878+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3646006780 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e9fef16f0 0x559e9fef3b10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.878+0000 7f3b6fe39740 1 --2- 172.21.15.116:0/3646006780 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e9fe75100 0x559e9fe83350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.878+0000 7f3b4aff5640 0 client.25397 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.878+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3646006780 >> 172.21.15.116:0/3646006780 conn(0x559e9fe70670 msgr2=0x559e9fe6f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.878+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3646006780 shutdown_connections 2024-03-19T01:15:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.878+0000 7f3b6fe39740 1 -- 172.21.15.116:0/3646006780 wait complete. 2024-03-19T01:15:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.878+0000 7f3b6fe39740 20 client.25397 trim_cache size 0 max 16384 2024-03-19T01:15:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:15:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:15:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:15:24.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:15:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.879+0000 7f3b6d24b640 1 -- 172.21.15.116:0/1426434841 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3b500a2a90 msgr2=0x7f3b500a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.879+0000 7f3b6d24b640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3b500a2a90 0x7f3b500a4f50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3b580097b0 tx=0x7f3b58005a90 comp rx=0 tx=0).stop 2024-03-19T01:15:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.879+0000 7f3b6d24b640 1 -- 172.21.15.116:0/1426434841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b681019b0 msgr2=0x7f3b681029a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.879+0000 7f3b6d24b640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b681019b0 0x7f3b681029a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3b60005c40 tx=0x7f3b600095b0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.879+0000 7f3b6d24b640 1 -- 172.21.15.116:0/1426434841 shutdown_connections 2024-03-19T01:15:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.879+0000 7f3b6d24b640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b681019b0 0x7f3b681029a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.879+0000 7f3b6d24b640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3b500a2a90 0x7f3b500a4f50 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.879+0000 7f3b6d24b640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3b68100ff0 0x7f3b68102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.879+0000 7f3b6d24b640 1 --2- 172.21.15.116:0/1426434841 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3b681000a0 0x7f3b68103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.879+0000 7f3b6d24b640 1 -- 172.21.15.116:0/1426434841 >> 172.21.15.116:0/1426434841 conn(0x7f3b680fb000 msgr2=0x7f3b681064c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.879+0000 7f3b6d24b640 1 -- 172.21.15.116:0/1426434841 shutdown_connections 2024-03-19T01:15:24.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:24.879+0000 7f3b6d24b640 1 -- 172.21.15.116:0/1426434841 wait complete. 2024-03-19T01:15:26.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:26 smithi156 ceph-mon[66799]: pgmap v436: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.4 MiB/s wr, 764 op/s 2024-03-19T01:15:26.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 160.0 s, 3779.6 tps, lat 26.448 ms stddev 29.299 2024-03-19T01:15:26.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:26 smithi162 ceph-mon[67178]: pgmap v436: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.4 MiB/s wr, 764 op/s 2024-03-19T01:15:26.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:26 smithi116 ceph-mon[62701]: pgmap v436: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.4 MiB/s wr, 764 op/s 2024-03-19T01:15:27.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:27 smithi156 ceph-mon[66799]: pgmap v437: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 10 MiB/s wr, 790 op/s 2024-03-19T01:15:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:27 smithi162 ceph-mon[67178]: pgmap v437: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 10 MiB/s wr, 790 op/s 2024-03-19T01:15:27.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:27 smithi116 ceph-mon[62701]: pgmap v437: 97 pgs: 97 active+clean; 1.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 10 MiB/s wr, 790 op/s 2024-03-19T01:15:30.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:30 smithi156 ceph-mon[66799]: pgmap v438: 97 pgs: 97 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 MiB/s wr, 920 op/s 2024-03-19T01:15:30.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:30 smithi162 ceph-mon[67178]: pgmap v438: 97 pgs: 97 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 MiB/s wr, 920 op/s 2024-03-19T01:15:30.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:30 smithi116 ceph-mon[62701]: pgmap v438: 97 pgs: 97 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 12 MiB/s wr, 920 op/s 2024-03-19T01:15:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 165.0 s, 3110.6 tps, lat 31.994 ms stddev 34.197 2024-03-19T01:15:32.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:32 smithi156 ceph-mon[66799]: pgmap v439: 97 pgs: 97 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 11 MiB/s wr, 853 op/s 2024-03-19T01:15:32.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:32 smithi162 ceph-mon[67178]: pgmap v439: 97 pgs: 97 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 11 MiB/s wr, 853 op/s 2024-03-19T01:15:32.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:32 smithi116 ceph-mon[62701]: pgmap v439: 97 pgs: 97 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 11 MiB/s wr, 853 op/s 2024-03-19T01:15:33.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:33 smithi156 ceph-mon[66799]: pgmap v440: 97 pgs: 97 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 10 MiB/s wr, 810 op/s 2024-03-19T01:15:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:33 smithi162 ceph-mon[67178]: pgmap v440: 97 pgs: 97 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 10 MiB/s wr, 810 op/s 2024-03-19T01:15:33.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:33 smithi116 ceph-mon[62701]: pgmap v440: 97 pgs: 97 active+clean; 1.2 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 10 MiB/s wr, 810 op/s 2024-03-19T01:15:36.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:36 smithi156 ceph-mon[66799]: pgmap v441: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 821 op/s 2024-03-19T01:15:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 170.0 s, 2529.4 tps, lat 39.591 ms stddev 41.845 2024-03-19T01:15:36.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:36 smithi162 ceph-mon[67178]: pgmap v441: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 821 op/s 2024-03-19T01:15:36.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:36 smithi116 ceph-mon[62701]: pgmap v441: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 MiB/s wr, 821 op/s 2024-03-19T01:15:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:37 smithi162 ceph-mon[67178]: pgmap v442: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.1 MiB/s wr, 758 op/s 2024-03-19T01:15:37.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:37 smithi116 ceph-mon[62701]: pgmap v442: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.1 MiB/s wr, 758 op/s 2024-03-19T01:15:37.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:37 smithi156 ceph-mon[66799]: pgmap v442: 97 pgs: 97 active+clean; 1.2 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 9.1 MiB/s wr, 758 op/s 2024-03-19T01:15:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.009+0000 7f47fe5d4640 1 -- 172.21.15.116:0/1244263086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f81022f0 msgr2=0x7f47f8108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:38.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.009+0000 7f47fe5d4640 1 --2- 172.21.15.116:0/1244263086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f81022f0 0x7f47f8108b80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009ca0 tx=0x7f47f002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.013+0000 7f47fe5d4640 1 -- 172.21.15.116:0/1244263086 shutdown_connections 2024-03-19T01:15:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.013+0000 7f47fe5d4640 1 --2- 172.21.15.116:0/1244263086 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f81022f0 0x7f47f8108b80 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.013+0000 7f47fe5d4640 1 --2- 172.21.15.116:0/1244263086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47f8101930 0x7f47f8101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.013+0000 7f47fe5d4640 1 --2- 172.21.15.116:0/1244263086 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47f8100740 0x7f47f8100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.013+0000 7f47fe5d4640 1 -- 172.21.15.116:0/1244263086 >> 172.21.15.116:0/1244263086 conn(0x7f47f80fbcb0 msgr2=0x7f47f80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:38.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.017+0000 7f47fe5d4640 1 -- 172.21.15.116:0/1244263086 shutdown_connections 2024-03-19T01:15:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.019+0000 7f47fe5d4640 1 -- 172.21.15.116:0/1244263086 wait complete. 2024-03-19T01:15:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.020+0000 7f47fe5d4640 1 Processor -- start 2024-03-19T01:15:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.020+0000 7f47fe5d4640 1 -- start start 2024-03-19T01:15:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.020+0000 7f47fe5d4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47f8100740 0x7f47f81111f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.020+0000 7f47fe5d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47f8101930 0x7f47f8111730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.020+0000 7f47fe5d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f81022f0 0x7f47f810f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.021+0000 7f47fe5d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f47f810a1e0 con 0x7f47f81022f0 2024-03-19T01:15:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.021+0000 7f47fe5d4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f47f810a060 con 0x7f47f8100740 2024-03-19T01:15:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.021+0000 7f47fe5d4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47f810a360 con 0x7f47f8101930 2024-03-19T01:15:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.021+0000 7f47fcb71640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f81022f0 0x7f47f810f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.021+0000 7f47fcb71640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f81022f0 0x7f47f810f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60450/0 (socket says 172.21.15.116:60450) 2024-03-19T01:15:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.021+0000 7f47fcb71640 1 -- 172.21.15.116:0/3467086482 learned_addr learned my addr 172.21.15.116:0/3467086482 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.021+0000 7f47fcb71640 1 -- 172.21.15.116:0/3467086482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47f8101930 msgr2=0x7f47f8111730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.021+0000 7f47fcb71640 1 --2- 172.21.15.116:0/3467086482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47f8101930 0x7f47f8111730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.021+0000 7f47fcb71640 1 -- 172.21.15.116:0/3467086482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47f8100740 msgr2=0x7f47f81111f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.021+0000 7f47fcb71640 1 --2- 172.21.15.116:0/3467086482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47f8100740 0x7f47f81111f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.021+0000 7f47fcb71640 1 -- 172.21.15.116:0/3467086482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47f810ffa0 con 0x7f47f81022f0 2024-03-19T01:15:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.022+0000 7f47fcb71640 1 --2- 172.21.15.116:0/3467086482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f81022f0 0x7f47f810f870 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f47f00325f0 tx=0x7f47f0032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.024+0000 7f47f57fa640 1 -- 172.21.15.116:0/3467086482 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0041020 con 0x7f47f81022f0 2024-03-19T01:15:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.024+0000 7f47fe5d4640 1 -- 172.21.15.116:0/3467086482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47f8110230 con 0x7f47f81022f0 2024-03-19T01:15:38.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.024+0000 7f47fe5d4640 1 -- 172.21.15.116:0/3467086482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47f81190c0 con 0x7f47f81022f0 2024-03-19T01:15:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.024+0000 7f47f57fa640 1 -- 172.21.15.116:0/3467086482 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f0034cd0 con 0x7f47f81022f0 2024-03-19T01:15:38.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.024+0000 7f47f57fa640 1 -- 172.21.15.116:0/3467086482 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f0039040 con 0x7f47f81022f0 2024-03-19T01:15:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.028+0000 7f48011bf740 1 Processor -- start 2024-03-19T01:15:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.029+0000 7f47f57fa640 1 -- 172.21.15.116:0/3467086482 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f47f0044020 con 0x7f47f81022f0 2024-03-19T01:15:38.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.029+0000 7f48011bf740 1 -- start start 2024-03-19T01:15:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.034+0000 7f48011bf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8e5110 0x5626df8e5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.034+0000 7f48011bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8e5b00 0x5626df8ea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.034+0000 7f48011bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8eac70 0x5626df8ed100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.034+0000 7f48011bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5626df838be0 con 0x5626df8eac70 2024-03-19T01:15:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.034+0000 7f48011bf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5626df7e1960 con 0x5626df8e5b00 2024-03-19T01:15:38.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.034+0000 7f48011bf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5626df834c80 con 0x5626df8e5110 2024-03-19T01:15:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.035+0000 7f47f57fa640 1 --2- 172.21.15.116:0/3467086482 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47e00a2ae0 0x7f47e00a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.035+0000 7f47fcb71640 1 --2- 172.21.15.116:0/3467086482 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47e00a2ae0 0x7f47e00a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.036+0000 7f47f57fa640 1 -- 172.21.15.116:0/3467086482 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f47f00f1970 con 0x7f47f81022f0 2024-03-19T01:15:38.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.036+0000 7f47fcb71640 1 --2- 172.21.15.116:0/3467086482 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47e00a2ae0 0x7f47e00a4fa0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f47f0037040 tx=0x7f47f00b3770 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:38.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.036+0000 7f47f7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8e5b00 0x5626df8ea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.036+0000 7f47f7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8e5b00 0x5626df8ea600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:44308/0 (socket says 172.21.15.116:44308) 2024-03-19T01:15:38.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.036+0000 7f47f7fff640 1 -- 172.21.15.116:0/1061097894 learned_addr learned my addr 172.21.15.116:0/1061097894 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:38.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.037+0000 7f47f77fe640 1 --2- 172.21.15.116:0/1061097894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8eac70 0x5626df8ed100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.037+0000 7f47f77fe640 1 --2- 172.21.15.116:0/1061097894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8e5110 0x5626df8e5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.037+0000 7f47f7fff640 1 -- 172.21.15.116:0/1061097894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8e5110 msgr2=0x5626df8e5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.037+0000 7f47f7fff640 1 --2- 172.21.15.116:0/1061097894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8e5110 0x5626df8e5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.037+0000 7f47f7fff640 1 -- 172.21.15.116:0/1061097894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8eac70 msgr2=0x5626df8ed100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.037+0000 7f47f7fff640 1 --2- 172.21.15.116:0/1061097894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8eac70 0x5626df8ed100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.037+0000 7f47f7fff640 1 -- 172.21.15.116:0/1061097894 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626df7d8d20 con 0x5626df8e5b00 2024-03-19T01:15:38.038 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.037+0000 7f47f7fff640 1 --2- 172.21.15.116:0/1061097894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8e5b00 0x5626df8ea600 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f47e80058c0 tx=0x7f47e800d2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85aca02579dbd000 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.041+0000 7f47de7fc640 1 -- 172.21.15.116:0/1061097894 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8014070 con 0x5626df8e5b00 2024-03-19T01:15:38.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.041+0000 7f47de7fc640 1 -- 172.21.15.116:0/1061097894 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47e800dd10 con 0x5626df8e5b00 2024-03-19T01:15:38.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.042+0000 7f48011bf740 1 -- 172.21.15.116:0/1061097894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8e5b00 msgr2=0x5626df8ea600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:38.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.042+0000 7f48011bf740 1 --2- 172.21.15.116:0/1061097894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8e5b00 0x5626df8ea600 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f47e80058c0 tx=0x7f47e800d2e0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.042+0000 7f48011bf740 1 -- 172.21.15.116:0/1061097894 shutdown_connections 2024-03-19T01:15:38.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.042+0000 7f48011bf740 1 --2- 172.21.15.116:0/1061097894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8eac70 0x5626df8ed100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.042+0000 7f48011bf740 1 --2- 172.21.15.116:0/1061097894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8e5b00 0x5626df8ea600 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.042+0000 7f48011bf740 1 --2- 172.21.15.116:0/1061097894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8e5110 0x5626df8e5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.042+0000 7f48011bf740 1 -- 172.21.15.116:0/1061097894 >> 172.21.15.116:0/1061097894 conn(0x5626df8e0680 msgr2=0x5626df8e0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:38.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.042+0000 7f48011bf740 1 -- 172.21.15.116:0/1061097894 shutdown_connections 2024-03-19T01:15:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.043+0000 7f48011bf740 1 -- 172.21.15.116:0/1061097894 wait complete. 2024-03-19T01:15:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.044+0000 7f48011bf740 1 -- start start 2024-03-19T01:15:38.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.044+0000 7f48011bf740 1 Processor -- start 2024-03-19T01:15:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f48011bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8e5110 0x5626df8ef0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f48011bf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8eac70 0x5626df8ef600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f48011bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 0x5626df8f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f48011bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5626df8edc70 con 0x5626df8e5110 2024-03-19T01:15:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f48011bf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5626df7f4db0 con 0x5626df8f4630 2024-03-19T01:15:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f48011bf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5626df8eddf0 con 0x5626df8eac70 2024-03-19T01:15:38.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f48011bf740 10 client.0 _ll_register_callbacks cb 0x5626df7f9e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:15:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f47f77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8e5110 0x5626df8ef0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f47f77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8e5110 0x5626df8ef0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60480/0 (socket says 172.21.15.116:60480) 2024-03-19T01:15:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f47f77fe640 1 -- 172.21.15.116:0/2836258738 learned_addr learned my addr 172.21.15.116:0/2836258738 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f47f7fff640 1 --2- 172.21.15.116:0/2836258738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8eac70 0x5626df8ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f47f77fe640 1 --2- 172.21.15.116:0/2836258738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 0x5626df8f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f47f77fe640 1 -- 172.21.15.116:0/2836258738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8eac70 msgr2=0x5626df8ef600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f47f77fe640 1 --2- 172.21.15.116:0/2836258738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8eac70 0x5626df8ef600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.045+0000 7f47f77fe640 1 -- 172.21.15.116:0/2836258738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 msgr2=0x5626df8f6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.046+0000 7f47f77fe640 1 --2- 172.21.15.116:0/2836258738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 0x5626df8f6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.046+0000 7f47f77fe640 1 -- 172.21.15.116:0/2836258738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626df7f3c60 con 0x5626df8e5110 2024-03-19T01:15:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.046+0000 7f47f77fe640 1 --2- 172.21.15.116:0/2836258738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8e5110 0x5626df8ef0c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f47ec011e20 tx=0x7f47ec009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c91a622aa5938d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:38.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.046+0000 7f47d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:15:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.046+0000 7f47d77fe640 1 -- 172.21.15.116:0/2836258738 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47ec00edf0 con 0x5626df8e5110 2024-03-19T01:15:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.046+0000 7f47d77fe640 1 -- 172.21.15.116:0/2836258738 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f47ec004030 con 0x5626df8e5110 2024-03-19T01:15:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.046+0000 7f48011bf740 1 -- 172.21.15.116:0/2836258738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5626df846120 con 0x5626df8e5110 2024-03-19T01:15:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.046+0000 7f47d77fe640 1 -- 172.21.15.116:0/2836258738 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f47ec026050 con 0x5626df8e5110 2024-03-19T01:15:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.047+0000 7f48011bf740 10 client.15567 fetch_fsmap learned FSMap version 33 2024-03-19T01:15:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.047+0000 7f48011bf740 1 -- 172.21.15.116:0/2836258738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5626df54d210 con 0x5626df8e5110 2024-03-19T01:15:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.047+0000 7f47d77fe640 1 -- 172.21.15.116:0/2836258738 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5626df846120 con 0x5626df8e5110 2024-03-19T01:15:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.047+0000 7f47d77fe640 1 -- 172.21.15.116:0/2836258738 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8386+0+0 (secure 0 0 0) 0x7f47ec003ce0 con 0x5626df8e5110 2024-03-19T01:15:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.048+0000 7f48011bf740 10 client.15567 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T01:15:38.049 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.048+0000 7f48011bf740 10 client.15567 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:15:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.048+0000 7f48011bf740 20 client.15567 populate_metadata read hostname 'smithi116' 2024-03-19T01:15:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.048+0000 7f48011bf740 1 --2- 172.21.15.116:0/2836258738 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5626df8fb4f0 0x5626df8fd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.048+0000 7f48011bf740 4 client.15567 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:15:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.048+0000 7f48011bf740 1 -- 172.21.15.116:0/2836258738 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626df840470 con 0x5626df8fb4f0 2024-03-19T01:15:38.050 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.048+0000 7f47f7fff640 1 --2- 172.21.15.116:0/2836258738 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5626df8fb4f0 0x5626df8fd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.049+0000 7f47f7fff640 1 --2- 172.21.15.116:0/2836258738 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5626df8fb4f0 0x5626df8fd950 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8548689494aaea3 server_cookie=6e559e7734d86cd1 in_seq=0 out_seq=0 2024-03-19T01:15:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.049+0000 7f47d77fe640 10 client.15567 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.050+0000 7f47d77fe640 1 -- 172.21.15.116:0/2836258738 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5626df840470 con 0x5626df8fb4f0 2024-03-19T01:15:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.050+0000 7f47d77fe640 10 client.15567 handle_command_reply: tid=0 2024-03-19T01:15:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.050+0000 7f48011bf740 1 client.15567 shutdown 2024-03-19T01:15:38.051 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.050+0000 7f48011bf740 1 -- 172.21.15.116:0/2836258738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8e5110 msgr2=0x5626df8ef0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.050+0000 7f48011bf740 1 --2- 172.21.15.116:0/2836258738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8e5110 0x5626df8ef0c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f47ec011e20 tx=0x7f47ec009880 comp rx=0 tx=0).stop 2024-03-19T01:15:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.051+0000 7f48011bf740 1 -- 172.21.15.116:0/2836258738 shutdown_connections 2024-03-19T01:15:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.051+0000 7f48011bf740 1 --2- 172.21.15.116:0/2836258738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 0x5626df8f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.051+0000 7f48011bf740 1 --2- 172.21.15.116:0/2836258738 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5626df8fb4f0 0x5626df8fd950 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.051+0000 7f48011bf740 1 --2- 172.21.15.116:0/2836258738 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8eac70 0x5626df8ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.051+0000 7f48011bf740 1 --2- 172.21.15.116:0/2836258738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8e5110 0x5626df8ef0c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.051+0000 7f48011bf740 1 -- 172.21.15.116:0/2836258738 >> 172.21.15.116:0/2836258738 conn(0x5626df8e0680 msgr2=0x5626df8e18d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.051+0000 7f47d77fe640 0 client.15567 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.051+0000 7f48011bf740 1 -- 172.21.15.116:0/2836258738 shutdown_connections 2024-03-19T01:15:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.051+0000 7f48011bf740 1 -- 172.21.15.116:0/2836258738 wait complete. 2024-03-19T01:15:38.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.051+0000 7f48011bf740 20 client.15567 trim_cache size 0 max 16384 2024-03-19T01:15:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.076+0000 7f48011bf740 1 Processor -- start 2024-03-19T01:15:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.076+0000 7f48011bf740 1 -- start start 2024-03-19T01:15:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.076+0000 7f48011bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8e5110 0x5626df8ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.076+0000 7f48011bf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8eac70 0x5626df8eecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.076+0000 7f48011bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 0x5626df8ef210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.076+0000 7f48011bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f47e400e570 con 0x5626df8e5110 2024-03-19T01:15:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.076+0000 7f48011bf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f47e4010230 con 0x5626df8f4630 2024-03-19T01:15:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.076+0000 7f48011bf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47e400ec40 con 0x5626df8eac70 2024-03-19T01:15:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.076+0000 7f47f7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8eac70 0x5626df8eecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.076+0000 7f47f7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8eac70 0x5626df8eecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48404/0 (socket says 172.21.15.116:48404) 2024-03-19T01:15:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.076+0000 7f47f7fff640 1 -- 172.21.15.116:0/1041272829 learned_addr learned my addr 172.21.15.116:0/1041272829 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.076+0000 7f47f77fe640 1 --2- 172.21.15.116:0/1041272829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8e5110 0x5626df8ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.076+0000 7f47f77fe640 1 --2- 172.21.15.116:0/1041272829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 0x5626df8ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.077+0000 7f47f7fff640 1 -- 172.21.15.116:0/1041272829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 msgr2=0x5626df8ef210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.077+0000 7f47f7fff640 1 --2- 172.21.15.116:0/1041272829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 0x5626df8ef210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.077+0000 7f47f7fff640 1 -- 172.21.15.116:0/1041272829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8e5110 msgr2=0x5626df8ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.077+0000 7f47f7fff640 1 --2- 172.21.15.116:0/1041272829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8e5110 0x5626df8ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.077+0000 7f47f7fff640 1 -- 172.21.15.116:0/1041272829 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626df619620 con 0x5626df8eac70 2024-03-19T01:15:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.077+0000 7f47f7fff640 1 --2- 172.21.15.116:0/1041272829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8eac70 0x5626df8eecd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f47e80058c0 tx=0x7f47e80047a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69538b78b286ae9d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.078+0000 7f47ddffb640 1 -- 172.21.15.116:0/1041272829 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8014040 con 0x5626df8eac70 2024-03-19T01:15:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.078+0000 7f47ddffb640 1 -- 172.21.15.116:0/1041272829 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47e802c070 con 0x5626df8eac70 2024-03-19T01:15:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.078+0000 7f47ddffb640 1 -- 172.21.15.116:0/1041272829 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8003850 con 0x5626df8eac70 2024-03-19T01:15:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.079+0000 7f48011bf740 1 -- 172.21.15.116:0/1041272829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8eac70 msgr2=0x5626df8eecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.079+0000 7f48011bf740 1 --2- 172.21.15.116:0/1041272829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8eac70 0x5626df8eecd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f47e80058c0 tx=0x7f47e80047a0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.079+0000 7f48011bf740 1 -- 172.21.15.116:0/1041272829 shutdown_connections 2024-03-19T01:15:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.079+0000 7f48011bf740 1 --2- 172.21.15.116:0/1041272829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 0x5626df8ef210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.079+0000 7f48011bf740 1 --2- 172.21.15.116:0/1041272829 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8eac70 0x5626df8eecd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.079+0000 7f48011bf740 1 --2- 172.21.15.116:0/1041272829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8e5110 0x5626df8ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.079+0000 7f48011bf740 1 -- 172.21.15.116:0/1041272829 >> 172.21.15.116:0/1041272829 conn(0x5626df8e0680 msgr2=0x5626df8ce220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.079+0000 7f48011bf740 1 -- 172.21.15.116:0/1041272829 shutdown_connections 2024-03-19T01:15:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.079+0000 7f48011bf740 1 -- 172.21.15.116:0/1041272829 wait complete. 2024-03-19T01:15:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.079+0000 7f48011bf740 1 -- start start 2024-03-19T01:15:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.080+0000 7f48011bf740 1 Processor -- start 2024-03-19T01:15:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.080+0000 7f48011bf740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8e5110 0x5626df90bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.080+0000 7f48011bf740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8eac70 0x5626df90c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.080+0000 7f48011bf740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 0x5626df90d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.080+0000 7f48011bf740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f47e400faa0 con 0x5626df8eac70 2024-03-19T01:15:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.080+0000 7f48011bf740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f47e400f3d0 con 0x5626df8f4630 2024-03-19T01:15:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.080+0000 7f48011bf740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47e400c550 con 0x5626df8e5110 2024-03-19T01:15:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.080+0000 7f47f77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8e5110 0x5626df90bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.080+0000 7f48011bf740 10 client.0 _ll_register_callbacks cb 0x7f47ec009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:15:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.080+0000 7f47f77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8e5110 0x5626df90bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48406/0 (socket says 172.21.15.116:48406) 2024-03-19T01:15:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.080+0000 7f47f77fe640 1 -- 172.21.15.116:0/648713285 learned_addr learned my addr 172.21.15.116:0/648713285 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.080+0000 7f47f7fff640 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8eac70 0x5626df90c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.080+0000 7f47f77fe640 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 0x5626df90d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f47f7fff640 1 -- 172.21.15.116:0/648713285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8e5110 msgr2=0x5626df90bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f47f7fff640 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8e5110 0x5626df90bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f47f7fff640 1 -- 172.21.15.116:0/648713285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 msgr2=0x5626df90d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f47f7fff640 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 0x5626df90d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f47f7fff640 1 -- 172.21.15.116:0/648713285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626df90d630 con 0x5626df8eac70 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f47f77fe640 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 0x5626df90d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f47f7fff640 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8eac70 0x5626df90c140 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f47e800bb30 tx=0x7f47e8027950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dfda08ab4784c11 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f47d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f47d77fe640 1 -- 172.21.15.116:0/648713285 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e8014070 con 0x5626df8eac70 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f47d77fe640 1 -- 172.21.15.116:0/648713285 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f47e8004240 con 0x5626df8eac70 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f48011bf740 1 -- 172.21.15.116:0/648713285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47e802c070 con 0x5626df8eac70 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f47d77fe640 1 -- 172.21.15.116:0/648713285 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f47e8004440 con 0x5626df8eac70 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f48011bf740 10 client.15573 fetch_fsmap learned FSMap version 33 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.081+0000 7f48011bf740 1 -- 172.21.15.116:0/648713285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5626df90e940 con 0x5626df8eac70 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.082+0000 7f47d77fe640 1 -- 172.21.15.116:0/648713285 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47e802c070 con 0x5626df8eac70 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.082+0000 7f47d77fe640 1 -- 172.21.15.116:0/648713285 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 33) ==== 8386+0+0 (secure 0 0 0) 0x7f47e8010660 con 0x5626df8eac70 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.082+0000 7f48011bf740 10 client.15573 fetch_fsmap finished waiting for FSMap version 33 2024-03-19T01:15:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.082+0000 7f48011bf740 10 client.15573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:15:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.082+0000 7f48011bf740 20 client.15573 populate_metadata read hostname 'smithi116' 2024-03-19T01:15:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.082+0000 7f48011bf740 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5626df961410 0x5626df963830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.082+0000 7f48011bf740 4 client.15573 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:15:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.082+0000 7f48011bf740 1 -- 172.21.15.116:0/648713285 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5626df7be770 con 0x5626df961410 2024-03-19T01:15:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.082+0000 7f47f77fe640 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5626df961410 0x5626df963830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.082+0000 7f47f77fe640 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5626df961410 0x5626df963830 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=698dcbf4125c850c server_cookie=b946013a60e015a5 in_seq=0 out_seq=0 2024-03-19T01:15:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.082+0000 7f47d77fe640 10 client.15573 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f47d77fe640 1 -- 172.21.15.116:0/648713285 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5626df7be770 con 0x5626df961410 2024-03-19T01:15:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f47d77fe640 10 client.15573 handle_command_reply: tid=0 2024-03-19T01:15:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f48011bf740 1 client.15573 shutdown 2024-03-19T01:15:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f48011bf740 1 -- 172.21.15.116:0/648713285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8eac70 msgr2=0x5626df90c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f48011bf740 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8eac70 0x5626df90c140 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f47e800bb30 tx=0x7f47e8027950 comp rx=0 tx=0).stop 2024-03-19T01:15:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f48011bf740 1 -- 172.21.15.116:0/648713285 shutdown_connections 2024-03-19T01:15:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f48011bf740 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5626df8f4630 0x5626df90d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f48011bf740 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5626df8eac70 0x5626df90c140 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f48011bf740 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5626df961410 0x5626df963830 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f48011bf740 1 --2- 172.21.15.116:0/648713285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5626df8e5110 0x5626df90bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f48011bf740 1 -- 172.21.15.116:0/648713285 >> 172.21.15.116:0/648713285 conn(0x5626df8e0680 msgr2=0x5626df8ce930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f47d77fe640 0 client.15573 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f48011bf740 1 -- 172.21.15.116:0/648713285 shutdown_connections 2024-03-19T01:15:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f48011bf740 1 -- 172.21.15.116:0/648713285 wait complete. 2024-03-19T01:15:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.090+0000 7f48011bf740 20 client.15573 trim_cache size 0 max 16384 2024-03-19T01:15:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:15:38.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:15:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:15:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:15:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.091+0000 7f47fe5d4640 1 -- 172.21.15.116:0/3467086482 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47e00a2ae0 msgr2=0x7f47e00a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.091+0000 7f47fe5d4640 1 --2- 172.21.15.116:0/3467086482 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47e00a2ae0 0x7f47e00a4fa0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f47f0037040 tx=0x7f47f00b3770 comp rx=0 tx=0).stop 2024-03-19T01:15:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.091+0000 7f47fe5d4640 1 -- 172.21.15.116:0/3467086482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f81022f0 msgr2=0x7f47f810f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.091+0000 7f47fe5d4640 1 --2- 172.21.15.116:0/3467086482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f81022f0 0x7f47f810f870 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f47f00325f0 tx=0x7f47f0032ec0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.091+0000 7f47fe5d4640 1 -- 172.21.15.116:0/3467086482 shutdown_connections 2024-03-19T01:15:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.091+0000 7f47fe5d4640 1 --2- 172.21.15.116:0/3467086482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47f81022f0 0x7f47f810f870 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.091+0000 7f47fe5d4640 1 --2- 172.21.15.116:0/3467086482 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47f8101930 0x7f47f8111730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.091+0000 7f47fe5d4640 1 --2- 172.21.15.116:0/3467086482 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47e00a2ae0 0x7f47e00a4fa0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.091+0000 7f47fe5d4640 1 --2- 172.21.15.116:0/3467086482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47f8100740 0x7f47f81111f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.091+0000 7f47fe5d4640 1 -- 172.21.15.116:0/3467086482 >> 172.21.15.116:0/3467086482 conn(0x7f47f80fbcb0 msgr2=0x7f47f8104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.091+0000 7f47fe5d4640 1 -- 172.21.15.116:0/3467086482 shutdown_connections 2024-03-19T01:15:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:38.091+0000 7f47fe5d4640 1 -- 172.21.15.116:0/3467086482 wait complete. 2024-03-19T01:15:39.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:39 smithi162 ceph-mon[67178]: pgmap v443: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 778 op/s 2024-03-19T01:15:39.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:39 smithi116 ceph-mon[62701]: pgmap v443: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 778 op/s 2024-03-19T01:15:39.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:39 smithi156 ceph-mon[66799]: pgmap v443: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 MiB/s wr, 778 op/s 2024-03-19T01:15:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 175.0 s, 3040.8 tps, lat 33.027 ms stddev 39.235 2024-03-19T01:15:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:42 smithi156 ceph-mon[66799]: pgmap v444: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 737 op/s 2024-03-19T01:15:42.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:42 smithi162 ceph-mon[67178]: pgmap v444: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 737 op/s 2024-03-19T01:15:42.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:42 smithi116 ceph-mon[62701]: pgmap v444: 97 pgs: 97 active+clean; 1.2 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 737 op/s 2024-03-19T01:15:43.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:43 smithi162 ceph-mon[67178]: pgmap v445: 97 pgs: 97 active+clean; 1.3 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 21 MiB/s wr, 773 op/s 2024-03-19T01:15:43.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:43 smithi116 ceph-mon[62701]: pgmap v445: 97 pgs: 97 active+clean; 1.3 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 21 MiB/s wr, 773 op/s 2024-03-19T01:15:43.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:43 smithi156 ceph-mon[66799]: pgmap v445: 97 pgs: 97 active+clean; 1.3 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 21 MiB/s wr, 773 op/s 2024-03-19T01:15:45.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:45 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:45.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:45 smithi162 ceph-mon[67178]: pgmap v446: 97 pgs: 97 active+clean; 1.3 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 30 MiB/s wr, 850 op/s 2024-03-19T01:15:45.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:45 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:45.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:45 smithi116 ceph-mon[62701]: pgmap v446: 97 pgs: 97 active+clean; 1.3 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 30 MiB/s wr, 850 op/s 2024-03-19T01:15:45.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:45 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:45.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:45 smithi156 ceph-mon[66799]: pgmap v446: 97 pgs: 97 active+clean; 1.3 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 30 MiB/s wr, 850 op/s 2024-03-19T01:15:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 180.0 s, 3899.9 tps, lat 25.615 ms stddev 31.010 2024-03-19T01:15:48.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:48 smithi156 ceph-mon[66799]: pgmap v447: 97 pgs: 97 active+clean; 1.4 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 36 MiB/s wr, 833 op/s 2024-03-19T01:15:48.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:48 smithi162 ceph-mon[67178]: pgmap v447: 97 pgs: 97 active+clean; 1.4 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 36 MiB/s wr, 833 op/s 2024-03-19T01:15:48.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:48 smithi116 ceph-mon[62701]: pgmap v447: 97 pgs: 97 active+clean; 1.4 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 36 MiB/s wr, 833 op/s 2024-03-19T01:15:49.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:49 smithi156 ceph-mon[66799]: pgmap v448: 97 pgs: 97 active+clean; 1.4 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 41 MiB/s wr, 866 op/s 2024-03-19T01:15:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:49 smithi162 ceph-mon[67178]: pgmap v448: 97 pgs: 97 active+clean; 1.4 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 41 MiB/s wr, 866 op/s 2024-03-19T01:15:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:49 smithi116 ceph-mon[62701]: pgmap v448: 97 pgs: 97 active+clean; 1.4 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 41 MiB/s wr, 866 op/s 2024-03-19T01:15:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.226+0000 7fca1e268640 1 -- 172.21.15.116:0/799158333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca18109990 msgr2=0x7fca181102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.226+0000 7fca1e268640 1 --2- 172.21.15.116:0/799158333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca18109990 0x7fca181102b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fca10009ca0 tx=0x7fca1002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.226+0000 7fca1e268640 1 -- 172.21.15.116:0/799158333 shutdown_connections 2024-03-19T01:15:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.226+0000 7fca1e268640 1 --2- 172.21.15.116:0/799158333 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca18109990 0x7fca181102b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.226+0000 7fca1e268640 1 --2- 172.21.15.116:0/799158333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca18108fd0 0x7fca18109450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.226+0000 7fca1e268640 1 --2- 172.21.15.116:0/799158333 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca18107e80 0x7fca181082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.226+0000 7fca1e268640 1 -- 172.21.15.116:0/799158333 >> 172.21.15.116:0/799158333 conn(0x7fca18076e20 msgr2=0x7fca18079260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.226+0000 7fca1e268640 1 -- 172.21.15.116:0/799158333 shutdown_connections 2024-03-19T01:15:51.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.226+0000 7fca1e268640 1 -- 172.21.15.116:0/799158333 wait complete. 2024-03-19T01:15:51.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.227+0000 7fca1e268640 1 Processor -- start 2024-03-19T01:15:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.227+0000 7fca1e268640 1 -- start start 2024-03-19T01:15:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.227+0000 7fca1e268640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca18107e80 0x7fca1819e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.227+0000 7fca1e268640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca18108fd0 0x7fca1819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.227+0000 7fca1e268640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca18109990 0x7fca181a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.228+0000 7fca1e268640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fca18113b30 con 0x7fca18107e80 2024-03-19T01:15:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.228+0000 7fca1e268640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fca181139b0 con 0x7fca18109990 2024-03-19T01:15:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.228+0000 7fca1e268640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fca18113cb0 con 0x7fca18108fd0 2024-03-19T01:15:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.228+0000 7fca1c805640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca18109990 0x7fca181a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.228+0000 7fca1c805640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca18109990 0x7fca181a55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:59986/0 (socket says 172.21.15.116:59986) 2024-03-19T01:15:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.228+0000 7fca1c805640 1 -- 172.21.15.116:0/338612486 learned_addr learned my addr 172.21.15.116:0/338612486 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.228+0000 7fca17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca18107e80 0x7fca1819e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.228+0000 7fca177fe640 1 --2- 172.21.15.116:0/338612486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca18108fd0 0x7fca1819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.228+0000 7fca1c805640 1 -- 172.21.15.116:0/338612486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca18108fd0 msgr2=0x7fca1819e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.228+0000 7fca1c805640 1 --2- 172.21.15.116:0/338612486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca18108fd0 0x7fca1819e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.228+0000 7fca1c805640 1 -- 172.21.15.116:0/338612486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca18107e80 msgr2=0x7fca1819e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.228+0000 7fca1c805640 1 --2- 172.21.15.116:0/338612486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca18107e80 0x7fca1819e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.228+0000 7fca1c805640 1 -- 172.21.15.116:0/338612486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca181a5c50 con 0x7fca18109990 2024-03-19T01:15:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.229+0000 7fca1c805640 1 --2- 172.21.15.116:0/338612486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca18109990 0x7fca181a55e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fca1002c630 tx=0x7fca10032ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.229+0000 7fca157fa640 1 -- 172.21.15.116:0/338612486 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10041020 con 0x7fca18109990 2024-03-19T01:15:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.229+0000 7fca157fa640 1 -- 172.21.15.116:0/338612486 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca10034cd0 con 0x7fca18109990 2024-03-19T01:15:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.229+0000 7fca1e268640 1 -- 172.21.15.116:0/338612486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca181a5ee0 con 0x7fca18109990 2024-03-19T01:15:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.229+0000 7fca157fa640 1 -- 172.21.15.116:0/338612486 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca10039040 con 0x7fca18109990 2024-03-19T01:15:51.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.229+0000 7fca1e268640 1 -- 172.21.15.116:0/338612486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca181a63a0 con 0x7fca18109990 2024-03-19T01:15:51.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.232+0000 7fca157fa640 1 -- 172.21.15.116:0/338612486 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fca10044020 con 0x7fca18109990 2024-03-19T01:15:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.232+0000 7fca157fa640 1 --2- 172.21.15.116:0/338612486 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fca000a2920 0x7fca000a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.233+0000 7fca157fa640 1 -- 172.21.15.116:0/338612486 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fca100f0810 con 0x7fca18109990 2024-03-19T01:15:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.233+0000 7fca20e50740 1 Processor -- start 2024-03-19T01:15:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.233+0000 7fca17fff640 1 --2- 172.21.15.116:0/338612486 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fca000a2920 0x7fca000a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.233+0000 7fca20e50740 1 -- start start 2024-03-19T01:15:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.233+0000 7fca20e50740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948100 0x56087e948520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.233+0000 7fca20e50740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948af0 0x56087e94d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.233+0000 7fca20e50740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e9500f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.233+0000 7fca20e50740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56087e8acbe0 con 0x56087e94dc60 2024-03-19T01:15:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.233+0000 7fca20e50740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56087e855960 con 0x56087e948100 2024-03-19T01:15:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.233+0000 7fca20e50740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56087e8a8c80 con 0x56087e948af0 2024-03-19T01:15:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fca1c805640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e9500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fca1c805640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e9500f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40762/0 (socket says 172.21.15.116:40762) 2024-03-19T01:15:51.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fca1c805640 1 -- 172.21.15.116:0/3570312461 learned_addr learned my addr 172.21.15.116:0/3570312461 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fca177fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948af0 0x56087e94d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fca17fff640 1 --2- 172.21.15.116:0/3570312461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948100 0x56087e948520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fca1c805640 1 -- 172.21.15.116:0/3570312461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948af0 msgr2=0x56087e94d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fca1c805640 1 --2- 172.21.15.116:0/3570312461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948af0 0x56087e94d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fca1c805640 1 -- 172.21.15.116:0/3570312461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948100 msgr2=0x56087e948520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fca1c805640 1 --2- 172.21.15.116:0/3570312461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948100 0x56087e948520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fca1c805640 1 -- 172.21.15.116:0/3570312461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56087e84cd20 con 0x56087e94dc60 2024-03-19T01:15:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fca1c805640 1 --2- 172.21.15.116:0/3570312461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e9500f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fca100bd1b0 tx=0x7fca100bcd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ea7b42c8deb7ba server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fca177fe640 1 --2- 172.21.15.116:0/3570312461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948af0 0x56087e94d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:15:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fca17fff640 1 --2- 172.21.15.116:0/338612486 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fca000a2920 0x7fca000a4de0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fca08013930 tx=0x7fca08013460 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fc9fe7fc640 1 -- 172.21.15.116:0/3570312461 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca100c1170 con 0x56087e94dc60 2024-03-19T01:15:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.234+0000 7fc9fe7fc640 1 -- 172.21.15.116:0/3570312461 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca100098b0 con 0x56087e94dc60 2024-03-19T01:15:51.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.235+0000 7fc9fe7fc640 1 -- 172.21.15.116:0/3570312461 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca100b9890 con 0x56087e94dc60 2024-03-19T01:15:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.235+0000 7fca20e50740 1 -- 172.21.15.116:0/3570312461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 msgr2=0x56087e9500f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.235+0000 7fca20e50740 1 --2- 172.21.15.116:0/3570312461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e9500f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fca100bd1b0 tx=0x7fca100bcd60 comp rx=0 tx=0).stop 2024-03-19T01:15:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.235+0000 7fca20e50740 1 -- 172.21.15.116:0/3570312461 shutdown_connections 2024-03-19T01:15:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.235+0000 7fca20e50740 1 --2- 172.21.15.116:0/3570312461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e9500f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.235+0000 7fca20e50740 1 --2- 172.21.15.116:0/3570312461 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948af0 0x56087e94d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.235+0000 7fca20e50740 1 --2- 172.21.15.116:0/3570312461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948100 0x56087e948520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.235+0000 7fca20e50740 1 -- 172.21.15.116:0/3570312461 >> 172.21.15.116:0/3570312461 conn(0x56087e943670 msgr2=0x56087e943a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.236+0000 7fca20e50740 1 -- 172.21.15.116:0/3570312461 shutdown_connections 2024-03-19T01:15:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.236+0000 7fca20e50740 1 -- 172.21.15.116:0/3570312461 wait complete. 2024-03-19T01:15:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.237+0000 7fca20e50740 1 -- start start 2024-03-19T01:15:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.237+0000 7fca20e50740 1 Processor -- start 2024-03-19T01:15:51.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.237+0000 7fca20e50740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e948100 0x56087e951e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.237+0000 7fca20e50740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948af0 0x56087e952340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.237+0000 7fca20e50740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e94dc60 0x56087e959380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.237+0000 7fca20e50740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56087e9509d0 con 0x56087e948100 2024-03-19T01:15:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.237+0000 7fca20e50740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56087e868db0 con 0x56087e948af0 2024-03-19T01:15:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.237+0000 7fca20e50740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56087e950b50 con 0x56087e94dc60 2024-03-19T01:15:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.237+0000 7fca20e50740 10 client.0 _ll_register_callbacks cb 0x56087e86de70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:15:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fca1c805640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e94dc60 0x56087e959380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fca17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e948100 0x56087e951e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fca17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e948100 0x56087e951e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40764/0 (socket says 172.21.15.116:40764) 2024-03-19T01:15:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fca17fff640 1 -- 172.21.15.116:0/352253982 learned_addr learned my addr 172.21.15.116:0/352253982 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fca177fe640 1 --2- 172.21.15.116:0/352253982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948af0 0x56087e952340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fca1c805640 1 -- 172.21.15.116:0/352253982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948af0 msgr2=0x56087e952340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:51.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fca1c805640 1 --2- 172.21.15.116:0/352253982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948af0 0x56087e952340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fca1c805640 1 -- 172.21.15.116:0/352253982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e948100 msgr2=0x56087e951e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fca1c805640 1 --2- 172.21.15.116:0/352253982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e948100 0x56087e951e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fca1c805640 1 -- 172.21.15.116:0/352253982 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56087e867c60 con 0x56087e94dc60 2024-03-19T01:15:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fca17fff640 1 --2- 172.21.15.116:0/352253982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e948100 0x56087e951e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:15:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fca1c805640 1 --2- 172.21.15.116:0/352253982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e94dc60 0x56087e959380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fca100bd1b0 tx=0x7fca100c1260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b7f76338c56e005 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fc9f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:15:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fc9f77fe640 1 -- 172.21.15.116:0/352253982 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca100c5930 con 0x56087e94dc60 2024-03-19T01:15:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fc9f77fe640 1 -- 172.21.15.116:0/352253982 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fca100c54a0 con 0x56087e94dc60 2024-03-19T01:15:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.238+0000 7fca20e50740 1 -- 172.21.15.116:0/352253982 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca100098b0 con 0x56087e94dc60 2024-03-19T01:15:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.239+0000 7fc9f77fe640 1 -- 172.21.15.116:0/352253982 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fca100c56a0 con 0x56087e94dc60 2024-03-19T01:15:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.239+0000 7fca20e50740 10 client.25399 fetch_fsmap learned FSMap version 35 2024-03-19T01:15:51.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.239+0000 7fca20e50740 1 -- 172.21.15.116:0/352253982 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56087e5c1210 con 0x56087e94dc60 2024-03-19T01:15:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.240+0000 7fc9f77fe640 1 -- 172.21.15.116:0/352253982 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca100098b0 con 0x56087e94dc60 2024-03-19T01:15:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.240+0000 7fc9f77fe640 1 -- 172.21.15.116:0/352253982 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7fca100a2080 con 0x56087e94dc60 2024-03-19T01:15:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.240+0000 7fca20e50740 10 client.25399 fetch_fsmap finished waiting for FSMap version 35 2024-03-19T01:15:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.240+0000 7fca20e50740 10 client.25399 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:15:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.240+0000 7fca20e50740 20 client.25399 populate_metadata read hostname 'smithi116' 2024-03-19T01:15:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.240+0000 7fca20e50740 1 --2- 172.21.15.116:0/352253982 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56087e95db60 0x56087e95ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.240+0000 7fca20e50740 4 client.25399 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.240+0000 7fca20e50740 1 -- 172.21.15.116:0/352253982 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56087e8ba120 con 0x56087e95db60 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.240+0000 7fca17fff640 1 --2- 172.21.15.116:0/352253982 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56087e95db60 0x56087e95ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.241+0000 7fca17fff640 1 --2- 172.21.15.116:0/352253982 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56087e95db60 0x56087e95ffb0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fb0eb67b19cb369 server_cookie=74d8e92cc6072e1c in_seq=0 out_seq=0 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.241+0000 7fc9f77fe640 10 client.25399 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.242+0000 7fc9f77fe640 1 -- 172.21.15.116:0/352253982 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56087e8ba120 con 0x56087e95db60 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.242+0000 7fc9f77fe640 10 client.25399 handle_command_reply: tid=0 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.242+0000 7fca20e50740 1 client.25399 shutdown 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.243+0000 7fca20e50740 1 -- 172.21.15.116:0/352253982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e94dc60 msgr2=0x56087e959380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.243+0000 7fca20e50740 1 --2- 172.21.15.116:0/352253982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e94dc60 0x56087e959380 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fca100bd1b0 tx=0x7fca100c1260 comp rx=0 tx=0).stop 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.243+0000 7fca20e50740 1 -- 172.21.15.116:0/352253982 shutdown_connections 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.243+0000 7fca20e50740 1 --2- 172.21.15.116:0/352253982 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56087e95db60 0x56087e95ffb0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.243+0000 7fca20e50740 1 --2- 172.21.15.116:0/352253982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e94dc60 0x56087e959380 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.243+0000 7fca20e50740 1 --2- 172.21.15.116:0/352253982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948af0 0x56087e952340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.243+0000 7fca20e50740 1 --2- 172.21.15.116:0/352253982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e948100 0x56087e951e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.243+0000 7fca20e50740 1 -- 172.21.15.116:0/352253982 >> 172.21.15.116:0/352253982 conn(0x56087e943670 msgr2=0x56087e9448c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.243+0000 7fc9f77fe640 0 client.25399 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.243+0000 7fca20e50740 1 -- 172.21.15.116:0/352253982 shutdown_connections 2024-03-19T01:15:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.243+0000 7fca20e50740 1 -- 172.21.15.116:0/352253982 wait complete. 2024-03-19T01:15:51.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.243+0000 7fca20e50740 20 client.25399 trim_cache size 0 max 16384 2024-03-19T01:15:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.266+0000 7fca20e50740 1 Processor -- start 2024-03-19T01:15:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.266+0000 7fca20e50740 1 -- start start 2024-03-19T01:15:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca20e50740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948100 0x56087e951e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca20e50740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948af0 0x56087e959340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca20e50740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e952340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca20e50740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fca181af2e0 con 0x56087e94dc60 2024-03-19T01:15:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca20e50740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fca181b1060 con 0x56087e948af0 2024-03-19T01:15:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca20e50740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fca181af9b0 con 0x56087e948100 2024-03-19T01:15:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca17fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948100 0x56087e951e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca1c805640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e952340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca1c805640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e952340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40772/0 (socket says 172.21.15.116:40772) 2024-03-19T01:15:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca1c805640 1 -- 172.21.15.116:0/1590693213 learned_addr learned my addr 172.21.15.116:0/1590693213 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca177fe640 1 --2- 172.21.15.116:0/1590693213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948af0 0x56087e959340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca17fff640 1 -- 172.21.15.116:0/1590693213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948af0 msgr2=0x56087e959340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca17fff640 1 --2- 172.21.15.116:0/1590693213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948af0 0x56087e959340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca17fff640 1 -- 172.21.15.116:0/1590693213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 msgr2=0x56087e952340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca17fff640 1 --2- 172.21.15.116:0/1590693213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e952340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.267+0000 7fca17fff640 1 -- 172.21.15.116:0/1590693213 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56087e68d620 con 0x56087e948100 2024-03-19T01:15:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.268+0000 7fca17fff640 1 --2- 172.21.15.116:0/1590693213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948100 0x56087e951e00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x56087e960800 tx=0x7fca08021000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=142db7b8ba3b1cd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.268+0000 7fc9fdffb640 1 -- 172.21.15.116:0/1590693213 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08012190 con 0x56087e948100 2024-03-19T01:15:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.268+0000 7fc9fdffb640 1 -- 172.21.15.116:0/1590693213 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca0802e070 con 0x56087e948100 2024-03-19T01:15:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.268+0000 7fc9fdffb640 1 -- 172.21.15.116:0/1590693213 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08010880 con 0x56087e948100 2024-03-19T01:15:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.269+0000 7fca20e50740 1 -- 172.21.15.116:0/1590693213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948100 msgr2=0x56087e951e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.269+0000 7fca20e50740 1 --2- 172.21.15.116:0/1590693213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948100 0x56087e951e00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x56087e960800 tx=0x7fca08021000 comp rx=0 tx=0).stop 2024-03-19T01:15:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.269+0000 7fca20e50740 1 -- 172.21.15.116:0/1590693213 shutdown_connections 2024-03-19T01:15:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.269+0000 7fca20e50740 1 --2- 172.21.15.116:0/1590693213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e952340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.269+0000 7fca20e50740 1 --2- 172.21.15.116:0/1590693213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948af0 0x56087e959340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.269+0000 7fca20e50740 1 --2- 172.21.15.116:0/1590693213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948100 0x56087e951e00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.269+0000 7fca20e50740 1 -- 172.21.15.116:0/1590693213 >> 172.21.15.116:0/1590693213 conn(0x56087e943670 msgr2=0x56087e93e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.269+0000 7fca20e50740 1 -- 172.21.15.116:0/1590693213 shutdown_connections 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.269+0000 7fca20e50740 1 -- 172.21.15.116:0/1590693213 wait complete. 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.269+0000 7fca20e50740 1 -- start start 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.270+0000 7fca20e50740 1 Processor -- start 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.270+0000 7fca20e50740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948100 0x56087e9560f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.270+0000 7fca20e50740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948af0 0x56087e956630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.270+0000 7fca20e50740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e9c1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.270+0000 7fca20e50740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fca181b08d0 con 0x56087e94dc60 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.270+0000 7fca20e50740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fca181b0140 con 0x56087e948100 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.270+0000 7fca20e50740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fca181ad2c0 con 0x56087e948af0 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.270+0000 7fca20e50740 10 client.0 _ll_register_callbacks cb 0x7fca08009e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.270+0000 7fca17fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948100 0x56087e9560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.270+0000 7fca17fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948100 0x56087e9560f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:60018/0 (socket says 172.21.15.116:60018) 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.270+0000 7fca1c805640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e9c1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.270+0000 7fca17fff640 1 -- 172.21.15.116:0/3077516656 learned_addr learned my addr 172.21.15.116:0/3077516656 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.270+0000 7fca177fe640 1 --2- 172.21.15.116:0/3077516656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948af0 0x56087e956630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.271+0000 7fca1c805640 1 -- 172.21.15.116:0/3077516656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948af0 msgr2=0x56087e956630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.271+0000 7fca1c805640 1 --2- 172.21.15.116:0/3077516656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948af0 0x56087e956630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.271+0000 7fca1c805640 1 -- 172.21.15.116:0/3077516656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948100 msgr2=0x56087e9560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.271+0000 7fca1c805640 1 --2- 172.21.15.116:0/3077516656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948100 0x56087e9560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.271+0000 7fca1c805640 1 -- 172.21.15.116:0/3077516656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56087e956c70 con 0x56087e94dc60 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.271+0000 7fca17fff640 1 --2- 172.21.15.116:0/3077516656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948100 0x56087e9560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.271+0000 7fca1c805640 1 --2- 172.21.15.116:0/3077516656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e9c1ee0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fca100bcf20 tx=0x7fca100bce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2718441db7df746d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.271+0000 7fc9f77fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.271+0000 7fc9f77fe640 1 -- 172.21.15.116:0/3077516656 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca100098b0 con 0x56087e94dc60 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.271+0000 7fc9f77fe640 1 -- 172.21.15.116:0/3077516656 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fca10004d50 con 0x56087e94dc60 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.272+0000 7fca20e50740 1 -- 172.21.15.116:0/3077516656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca0802e070 con 0x56087e94dc60 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.272+0000 7fc9f77fe640 1 -- 172.21.15.116:0/3077516656 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fca100a6020 con 0x56087e94dc60 2024-03-19T01:15:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.272+0000 7fca20e50740 10 client.15591 fetch_fsmap learned FSMap version 35 2024-03-19T01:15:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.272+0000 7fca20e50740 1 -- 172.21.15.116:0/3077516656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56087e956c70 con 0x56087e94dc60 2024-03-19T01:15:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.273+0000 7fc9f77fe640 1 -- 172.21.15.116:0/3077516656 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca1009f050 con 0x56087e94dc60 2024-03-19T01:15:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.273+0000 7fc9f77fe640 1 -- 172.21.15.116:0/3077516656 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7fca100d8060 con 0x56087e94dc60 2024-03-19T01:15:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.273+0000 7fca20e50740 10 client.15591 fetch_fsmap finished waiting for FSMap version 35 2024-03-19T01:15:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.273+0000 7fca20e50740 10 client.15591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:15:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.273+0000 7fca20e50740 20 client.15591 populate_metadata read hostname 'smithi116' 2024-03-19T01:15:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.273+0000 7fca20e50740 1 --2- 172.21.15.116:0/3077516656 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56087e9c3b30 0x56087e9c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:15:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.273+0000 7fca20e50740 4 client.15591 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:15:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.273+0000 7fca20e50740 1 -- 172.21.15.116:0/3077516656 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fca0802e070 con 0x56087e9c3b30 2024-03-19T01:15:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.274+0000 7fca17fff640 1 --2- 172.21.15.116:0/3077516656 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56087e9c3b30 0x56087e9c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:15:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.274+0000 7fca17fff640 1 --2- 172.21.15.116:0/3077516656 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56087e9c3b30 0x56087e9c5f50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceef94b7ea5e89f1 server_cookie=a1c86e09d4415e62 in_seq=0 out_seq=0 2024-03-19T01:15:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.274+0000 7fc9f77fe640 10 client.15591 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.283+0000 7fc9f77fe640 1 -- 172.21.15.116:0/3077516656 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fca0802e070 con 0x56087e9c3b30 2024-03-19T01:15:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.283+0000 7fc9f77fe640 10 client.15591 handle_command_reply: tid=0 2024-03-19T01:15:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.283+0000 7fca20e50740 1 client.15591 shutdown 2024-03-19T01:15:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.284+0000 7fca20e50740 1 -- 172.21.15.116:0/3077516656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 msgr2=0x56087e9c1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.284+0000 7fca20e50740 1 --2- 172.21.15.116:0/3077516656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e9c1ee0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fca100bcf20 tx=0x7fca100bce90 comp rx=0 tx=0).stop 2024-03-19T01:15:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.284+0000 7fca20e50740 1 -- 172.21.15.116:0/3077516656 shutdown_connections 2024-03-19T01:15:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.284+0000 7fca20e50740 1 --2- 172.21.15.116:0/3077516656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56087e94dc60 0x56087e9c1ee0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.284+0000 7fca20e50740 1 --2- 172.21.15.116:0/3077516656 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56087e9c3b30 0x56087e9c5f50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.284+0000 7fca20e50740 1 --2- 172.21.15.116:0/3077516656 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56087e948af0 0x56087e956630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.284+0000 7fca20e50740 1 --2- 172.21.15.116:0/3077516656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56087e948100 0x56087e9560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.284+0000 7fca20e50740 1 -- 172.21.15.116:0/3077516656 >> 172.21.15.116:0/3077516656 conn(0x56087e943670 msgr2=0x56087e942220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.284+0000 7fc9f77fe640 0 client.15591 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:15:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.284+0000 7fca20e50740 1 -- 172.21.15.116:0/3077516656 shutdown_connections 2024-03-19T01:15:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.284+0000 7fca20e50740 1 -- 172.21.15.116:0/3077516656 wait complete. 2024-03-19T01:15:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.284+0000 7fca20e50740 20 client.15591 trim_cache size 0 max 16384 2024-03-19T01:15:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:15:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:15:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:15:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:15:51.287 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.285+0000 7fca1e268640 1 -- 172.21.15.116:0/338612486 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fca000a2920 msgr2=0x7fca000a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.285+0000 7fca1e268640 1 --2- 172.21.15.116:0/338612486 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fca000a2920 0x7fca000a4de0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fca08013930 tx=0x7fca08013460 comp rx=0 tx=0).stop 2024-03-19T01:15:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.286+0000 7fca1e268640 1 -- 172.21.15.116:0/338612486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca18109990 msgr2=0x7fca181a55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:15:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.286+0000 7fca1e268640 1 --2- 172.21.15.116:0/338612486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca18109990 0x7fca181a55e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fca1002c630 tx=0x7fca10032ea0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.286+0000 7fca1e268640 1 -- 172.21.15.116:0/338612486 shutdown_connections 2024-03-19T01:15:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.286+0000 7fca1e268640 1 --2- 172.21.15.116:0/338612486 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fca000a2920 0x7fca000a4de0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.286+0000 7fca1e268640 1 --2- 172.21.15.116:0/338612486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fca18109990 0x7fca181a55e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.286+0000 7fca1e268640 1 --2- 172.21.15.116:0/338612486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fca18108fd0 0x7fca1819e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.286+0000 7fca1e268640 1 --2- 172.21.15.116:0/338612486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca18107e80 0x7fca1819e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:15:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.286+0000 7fca1e268640 1 -- 172.21.15.116:0/338612486 >> 172.21.15.116:0/338612486 conn(0x7fca18076e20 msgr2=0x7fca1810c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:15:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.287+0000 7fca1e268640 1 -- 172.21.15.116:0/338612486 shutdown_connections 2024-03-19T01:15:51.288 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:15:51.287+0000 7fca1e268640 1 -- 172.21.15.116:0/338612486 wait complete. 2024-03-19T01:15:51.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:51 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:51.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:51 smithi156 ceph-mon[66799]: pgmap v449: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 48 MiB/s wr, 875 op/s 2024-03-19T01:15:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 185.0 s, 3995.7 tps, lat 25.027 ms stddev 29.035 2024-03-19T01:15:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:51 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:51.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:51 smithi162 ceph-mon[67178]: pgmap v449: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 48 MiB/s wr, 875 op/s 2024-03-19T01:15:51.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:51 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:15:51.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:51 smithi116 ceph-mon[62701]: pgmap v449: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 48 MiB/s wr, 875 op/s 2024-03-19T01:15:54.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:54 smithi156 ceph-mon[66799]: pgmap v450: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 44 MiB/s wr, 770 op/s 2024-03-19T01:15:54.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:54 smithi162 ceph-mon[67178]: pgmap v450: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 44 MiB/s wr, 770 op/s 2024-03-19T01:15:54.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:54 smithi116 ceph-mon[62701]: pgmap v450: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 44 MiB/s wr, 770 op/s 2024-03-19T01:15:55.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:55 smithi156 ceph-mon[66799]: pgmap v451: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 45 MiB/s wr, 904 op/s 2024-03-19T01:15:55.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:55 smithi162 ceph-mon[67178]: pgmap v451: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 45 MiB/s wr, 904 op/s 2024-03-19T01:15:55.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:55 smithi116 ceph-mon[62701]: pgmap v451: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 45 MiB/s wr, 904 op/s 2024-03-19T01:15:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 190.0 s, 3133.0 tps, lat 29.519 ms stddev 37.011 2024-03-19T01:15:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:57 smithi162 ceph-mon[67178]: pgmap v452: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 42 MiB/s wr, 913 op/s 2024-03-19T01:15:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:57 smithi116 ceph-mon[62701]: pgmap v452: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 42 MiB/s wr, 913 op/s 2024-03-19T01:15:57.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:57 smithi156 ceph-mon[66799]: pgmap v452: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 42 MiB/s wr, 913 op/s 2024-03-19T01:15:59.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:58 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:15:59.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:58 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:15:59.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:58 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:16:00.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:59 smithi156 ceph-mon[66799]: pgmap v453: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 54 MiB/s wr, 1.34k op/s 2024-03-19T01:16:00.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:16:00.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:16:00.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:15:59 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:16:00.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:59 smithi162 ceph-mon[67178]: pgmap v453: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 54 MiB/s wr, 1.34k op/s 2024-03-19T01:16:00.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:16:00.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:16:00.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:15:59 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:16:00.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:59 smithi116 ceph-mon[62701]: pgmap v453: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 54 MiB/s wr, 1.34k op/s 2024-03-19T01:16:00.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:16:00.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:16:00.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:15:59 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:16:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 195.0 s, 3441.6 tps, lat 31.220 ms stddev 50.911 2024-03-19T01:16:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:01 smithi162 ceph-mon[67178]: pgmap v454: 97 pgs: 97 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 62 MiB/s wr, 1.60k op/s 2024-03-19T01:16:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:01 smithi116 ceph-mon[62701]: pgmap v454: 97 pgs: 97 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 62 MiB/s wr, 1.60k op/s 2024-03-19T01:16:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:01 smithi156 ceph-mon[66799]: pgmap v454: 97 pgs: 97 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 62 MiB/s wr, 1.60k op/s 2024-03-19T01:16:03.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:03 smithi162 ceph-mon[67178]: pgmap v455: 97 pgs: 97 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 MiB/s wr, 1.52k op/s 2024-03-19T01:16:03.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:03 smithi116 ceph-mon[62701]: pgmap v455: 97 pgs: 97 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 MiB/s wr, 1.52k op/s 2024-03-19T01:16:03.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:03 smithi156 ceph-mon[66799]: pgmap v455: 97 pgs: 97 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 MiB/s wr, 1.52k op/s 2024-03-19T01:16:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.414+0000 7f36ec80c640 1 -- 172.21.15.116:0/2181973686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8109930 msgr2=0x7f36e810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.414+0000 7f36ec80c640 1 --2- 172.21.15.116:0/2181973686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8109930 0x7f36e810bd20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009860 tx=0x7f36d802bb70 comp rx=0 tx=0).stop 2024-03-19T01:16:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.414+0000 7f36ec80c640 1 -- 172.21.15.116:0/2181973686 shutdown_connections 2024-03-19T01:16:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.414+0000 7f36ec80c640 1 --2- 172.21.15.116:0/2181973686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36e810ebe0 0x7f36e8111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.414+0000 7f36ec80c640 1 --2- 172.21.15.116:0/2181973686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e810c260 0x7f36e810e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.414+0000 7f36ec80c640 1 --2- 172.21.15.116:0/2181973686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e8109930 0x7f36e810bd20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.414+0000 7f36ec80c640 1 -- 172.21.15.116:0/2181973686 >> 172.21.15.116:0/2181973686 conn(0x7f36e806e860 msgr2=0x7f36e8070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.414+0000 7f36ec80c640 1 -- 172.21.15.116:0/2181973686 shutdown_connections 2024-03-19T01:16:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.414+0000 7f36ec80c640 1 -- 172.21.15.116:0/2181973686 wait complete. 2024-03-19T01:16:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.415+0000 7f36ec80c640 1 Processor -- start 2024-03-19T01:16:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.415+0000 7f36ec80c640 1 -- start start 2024-03-19T01:16:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.415+0000 7f36ec80c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e8109930 0x7f36e81a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.415+0000 7f36ec80c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36e810c260 0x7f36e81a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.416+0000 7f36ec80c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e810ebe0 0x7f36e81a7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.416+0000 7f36ec80c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36e8114c10 con 0x7f36e8109930 2024-03-19T01:16:04.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.416+0000 7f36ec80c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f36e8114a90 con 0x7f36e810c260 2024-03-19T01:16:04.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.416+0000 7f36ec80c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36e8114d90 con 0x7f36e810ebe0 2024-03-19T01:16:04.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.416+0000 7f36dffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36e810c260 0x7f36e81a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.416+0000 7f36e659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e8109930 0x7f36e81a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.416+0000 7f36dffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36e810c260 0x7f36e81a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:39354/0 (socket says 172.21.15.116:39354) 2024-03-19T01:16:04.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.416+0000 7f36dffff640 1 -- 172.21.15.116:0/972036468 learned_addr learned my addr 172.21.15.116:0/972036468 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:04.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.416+0000 7f36dffff640 1 -- 172.21.15.116:0/972036468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e810ebe0 msgr2=0x7f36e81a7890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:16:04.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.416+0000 7f36dffff640 1 --2- 172.21.15.116:0/972036468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e810ebe0 0x7f36e81a7890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.416+0000 7f36dffff640 1 -- 172.21.15.116:0/972036468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e8109930 msgr2=0x7f36e81a02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.416+0000 7f36dffff640 1 --2- 172.21.15.116:0/972036468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e8109930 0x7f36e81a02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.416+0000 7f36dffff640 1 -- 172.21.15.116:0/972036468 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e81a7f00 con 0x7f36e810c260 2024-03-19T01:16:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.417+0000 7f36dffff640 1 --2- 172.21.15.116:0/972036468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36e810c260 0x7f36e81a0820 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f36d000e9a0 tx=0x7f36d000ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.417+0000 7f36df7fe640 1 -- 172.21.15.116:0/972036468 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d000cd80 con 0x7f36e810c260 2024-03-19T01:16:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.417+0000 7f36df7fe640 1 -- 172.21.15.116:0/972036468 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36d00035f0 con 0x7f36e810c260 2024-03-19T01:16:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.417+0000 7f36ec80c640 1 -- 172.21.15.116:0/972036468 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e810f080 con 0x7f36e810c260 2024-03-19T01:16:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.417+0000 7f36df7fe640 1 -- 172.21.15.116:0/972036468 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d0010670 con 0x7f36e810c260 2024-03-19T01:16:04.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.417+0000 7f36ec80c640 1 -- 172.21.15.116:0/972036468 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e810f620 con 0x7f36e810c260 2024-03-19T01:16:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.420+0000 7f36df7fe640 1 -- 172.21.15.116:0/972036468 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f36d00037a0 con 0x7f36e810c260 2024-03-19T01:16:04.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.421+0000 7f36df7fe640 1 --2- 172.21.15.116:0/972036468 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f36d40a2970 0x7f36d40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.421+0000 7f36df7fe640 1 -- 172.21.15.116:0/972036468 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f36d00ced80 con 0x7f36e810c260 2024-03-19T01:16:04.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.421+0000 7f36e659c640 1 --2- 172.21.15.116:0/972036468 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f36d40a2970 0x7f36d40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.422+0000 7f36ef3f1740 1 Processor -- start 2024-03-19T01:16:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.422+0000 7f36e659c640 1 --2- 172.21.15.116:0/972036468 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f36d40a2970 0x7f36d40a4e30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f36d802c180 tx=0x7f36d8034ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:04.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.422+0000 7f36ef3f1740 1 -- start start 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.422+0000 7f36ef3f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2f100 0x5631eeb2f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.422+0000 7f36ef3f1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2faf0 0x5631eeb345f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36ef3f1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 0x5631eeb370f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36ef3f1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5631eea93be0 con 0x5631eeb2f100 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36ef3f1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5631eea3c960 con 0x5631eeb34c60 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36ef3f1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5631eea8fc80 con 0x5631eeb2faf0 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36dffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2faf0 0x5631eeb345f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36dffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2faf0 0x5631eeb345f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43852/0 (socket says 172.21.15.116:43852) 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36dffff640 1 -- 172.21.15.116:0/487381651 learned_addr learned my addr 172.21.15.116:0/487381651 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36e659c640 1 --2- 172.21.15.116:0/487381651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2f100 0x5631eeb2f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36e6d9d640 1 --2- 172.21.15.116:0/487381651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 0x5631eeb370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36dffff640 1 -- 172.21.15.116:0/487381651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 msgr2=0x5631eeb370f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36dffff640 1 --2- 172.21.15.116:0/487381651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 0x5631eeb370f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36dffff640 1 -- 172.21.15.116:0/487381651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2f100 msgr2=0x5631eeb2f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:04.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36dffff640 1 --2- 172.21.15.116:0/487381651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2f100 0x5631eeb2f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.423+0000 7f36dffff640 1 -- 172.21.15.116:0/487381651 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631eea33d20 con 0x5631eeb2faf0 2024-03-19T01:16:04.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.424+0000 7f36dffff640 1 --2- 172.21.15.116:0/487381651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2faf0 0x5631eeb345f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f36d0097b00 tx=0x7f36d0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=374e4ea58ee2285e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:04.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.425+0000 7f36dcff9640 1 -- 172.21.15.116:0/487381651 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d00b4070 con 0x5631eeb2faf0 2024-03-19T01:16:04.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.425+0000 7f36dcff9640 1 -- 172.21.15.116:0/487381651 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36d00860c0 con 0x5631eeb2faf0 2024-03-19T01:16:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.426+0000 7f36dcff9640 1 -- 172.21.15.116:0/487381651 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d0097d10 con 0x5631eeb2faf0 2024-03-19T01:16:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.426+0000 7f36ef3f1740 1 -- 172.21.15.116:0/487381651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2faf0 msgr2=0x5631eeb345f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:04.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.426+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/487381651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2faf0 0x5631eeb345f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f36d0097b00 tx=0x7f36d0000bc0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.426+0000 7f36ef3f1740 1 -- 172.21.15.116:0/487381651 shutdown_connections 2024-03-19T01:16:04.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.426+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/487381651 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 0x5631eeb370f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.426+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/487381651 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2faf0 0x5631eeb345f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.426+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/487381651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2f100 0x5631eeb2f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.426+0000 7f36ef3f1740 1 -- 172.21.15.116:0/487381651 >> 172.21.15.116:0/487381651 conn(0x5631eeb2a670 msgr2=0x5631eeb2aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.427+0000 7f36ef3f1740 1 -- 172.21.15.116:0/487381651 shutdown_connections 2024-03-19T01:16:04.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.427+0000 7f36ef3f1740 1 -- 172.21.15.116:0/487381651 wait complete. 2024-03-19T01:16:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.427+0000 7f36ef3f1740 1 -- start start 2024-03-19T01:16:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.428+0000 7f36ef3f1740 1 Processor -- start 2024-03-19T01:16:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.428+0000 7f36ef3f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2f100 0x5631eeb391c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.428+0000 7f36ef3f1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2faf0 0x5631eeb39700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.428+0000 7f36e659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2f100 0x5631eeb391c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.428+0000 7f36e659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2f100 0x5631eeb391c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39812/0 (socket says 172.21.15.116:39812) 2024-03-19T01:16:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.429+0000 7f36dffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2faf0 0x5631eeb39700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.429+0000 7f36dffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2faf0 0x5631eeb39700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43868/0 (socket says 172.21.15.116:43868) 2024-03-19T01:16:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.429+0000 7f36e659c640 1 -- 172.21.15.116:0/1460711760 learned_addr learned my addr 172.21.15.116:0/1460711760 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.429+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/1460711760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 0x5631eeb40740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.429+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1460711760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5631eeb37cb0 con 0x5631eeb2f100 2024-03-19T01:16:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.429+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1460711760 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5631eea4fdb0 con 0x5631eeb34c60 2024-03-19T01:16:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.429+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1460711760 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5631eeb37e30 con 0x5631eeb2faf0 2024-03-19T01:16:04.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.429+0000 7f36e6d9d640 1 --2- 172.21.15.116:0/1460711760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 0x5631eeb40740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.429+0000 7f36e659c640 1 -- 172.21.15.116:0/1460711760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2faf0 msgr2=0x5631eeb39700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.429+0000 7f36e659c640 1 --2- 172.21.15.116:0/1460711760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2faf0 0x5631eeb39700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.429+0000 7f36e659c640 1 -- 172.21.15.116:0/1460711760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 msgr2=0x5631eeb40740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.429+0000 7f36e659c640 1 --2- 172.21.15.116:0/1460711760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 0x5631eeb40740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.429+0000 7f36e659c640 1 -- 172.21.15.116:0/1460711760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631eea4ec60 con 0x5631eeb2f100 2024-03-19T01:16:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.430+0000 7f36ef3f1740 10 client.0 _ll_register_callbacks cb 0x5631eea54e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:16:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.430+0000 7f36e659c640 1 --2- 172.21.15.116:0/1460711760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2f100 0x5631eeb391c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009860 tx=0x7f36d802bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9e3f2aea1299a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.430+0000 7f36cdffb640 10 client.15603 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:16:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.430+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1460711760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36d00860c0 con 0x5631eeb2f100 2024-03-19T01:16:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.430+0000 7f36cdffb640 1 -- 172.21.15.116:0/1460711760 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8057050 con 0x5631eeb2f100 2024-03-19T01:16:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.430+0000 7f36cdffb640 1 -- 172.21.15.116:0/1460711760 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f36d8042030 con 0x5631eeb2f100 2024-03-19T01:16:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.430+0000 7f36cdffb640 1 -- 172.21.15.116:0/1460711760 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f36d8039040 con 0x5631eeb2f100 2024-03-19T01:16:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.430+0000 7f36ef3f1740 10 client.15603 fetch_fsmap learned FSMap version 35 2024-03-19T01:16:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.430+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1460711760 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=35}) -- 0x5631ee7a8210 con 0x5631eeb2f100 2024-03-19T01:16:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.431+0000 7f36cdffb640 1 -- 172.21.15.116:0/1460711760 <== mon.0 v2:172.21.15.116:3300/0 4 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f36d802c510 con 0x5631eeb2f100 2024-03-19T01:16:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.432+0000 7f36ef3f1740 10 client.15603 fetch_fsmap finished waiting for FSMap version 35 2024-03-19T01:16:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.432+0000 7f36ef3f1740 10 client.15603 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:16:04.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.432+0000 7f36ef3f1740 20 client.15603 populate_metadata read hostname 'smithi116' 2024-03-19T01:16:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.432+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/1460711760 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5631eeb44c30 0x5631eeb470c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.432+0000 7f36ef3f1740 4 client.15603 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:16:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.432+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1460711760 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f36d00860c0 con 0x5631eeb44c30 2024-03-19T01:16:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.433+0000 7f36dffff640 1 --2- 172.21.15.116:0/1460711760 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5631eeb44c30 0x5631eeb470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.433+0000 7f36dffff640 1 --2- 172.21.15.116:0/1460711760 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5631eeb44c30 0x5631eeb470c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff1f6f46d50d22f server_cookie=82bb54fd26a659c6 in_seq=0 out_seq=0 2024-03-19T01:16:04.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.433+0000 7f36cdffb640 10 client.15603 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.434+0000 7f36cdffb640 1 -- 172.21.15.116:0/1460711760 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f36d00860c0 con 0x5631eeb44c30 2024-03-19T01:16:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.434+0000 7f36cdffb640 10 client.15603 handle_command_reply: tid=0 2024-03-19T01:16:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.434+0000 7f36ef3f1740 1 client.15603 shutdown 2024-03-19T01:16:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.435+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1460711760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2f100 msgr2=0x5631eeb391c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.435+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/1460711760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2f100 0x5631eeb391c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009860 tx=0x7f36d802bb70 comp rx=0 tx=0).stop 2024-03-19T01:16:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.435+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1460711760 shutdown_connections 2024-03-19T01:16:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.435+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/1460711760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 0x5631eeb40740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.435+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/1460711760 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5631eeb44c30 0x5631eeb470c0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.435+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/1460711760 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2faf0 0x5631eeb39700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.435+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/1460711760 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2f100 0x5631eeb391c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.435+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1460711760 >> 172.21.15.116:0/1460711760 conn(0x5631eeb2a670 msgr2=0x5631eeb2b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.435+0000 7f36cdffb640 0 client.15603 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.435+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1460711760 shutdown_connections 2024-03-19T01:16:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.435+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1460711760 wait complete. 2024-03-19T01:16:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.435+0000 7f36ef3f1740 20 client.15603 trim_cache size 0 max 16384 2024-03-19T01:16:04.464 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.464+0000 7f36ef3f1740 1 Processor -- start 2024-03-19T01:16:04.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.464+0000 7f36ef3f1740 1 -- start start 2024-03-19T01:16:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.465+0000 7f36ef3f1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2f100 0x5631eeb38dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.465+0000 7f36ef3f1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb2faf0 0x5631eeb406d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.465+0000 7f36ef3f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb34c60 0x5631eeb39310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.465+0000 7f36dffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb2faf0 0x5631eeb406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.465+0000 7f36dffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb2faf0 0x5631eeb406d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:39372/0 (socket says 172.21.15.116:39372) 2024-03-19T01:16:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.465+0000 7f36e659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2f100 0x5631eeb38dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.465+0000 7f36e6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb34c60 0x5631eeb39310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.465+0000 7f36e6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb34c60 0x5631eeb39310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39814/0 (socket says 172.21.15.116:39814) 2024-03-19T01:16:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.465+0000 7f36dffff640 1 -- 172.21.15.116:0/1861694105 learned_addr learned my addr 172.21.15.116:0/1861694105 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.465+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1861694105 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36e81af490 con 0x5631eeb34c60 2024-03-19T01:16:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.465+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1861694105 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f36e81b1210 con 0x5631eeb2faf0 2024-03-19T01:16:04.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.465+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1861694105 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36e81afb60 con 0x5631eeb2f100 2024-03-19T01:16:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.466+0000 7f36e6d9d640 1 -- 172.21.15.116:0/1861694105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2f100 msgr2=0x5631eeb38dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.466+0000 7f36e6d9d640 1 --2- 172.21.15.116:0/1861694105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2f100 0x5631eeb38dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.466+0000 7f36e6d9d640 1 -- 172.21.15.116:0/1861694105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb2faf0 msgr2=0x5631eeb406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.466+0000 7f36e6d9d640 1 --2- 172.21.15.116:0/1861694105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb2faf0 0x5631eeb406d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.466+0000 7f36e6d9d640 1 -- 172.21.15.116:0/1861694105 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631ee874620 con 0x5631eeb34c60 2024-03-19T01:16:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.466+0000 7f36e6d9d640 1 --2- 172.21.15.116:0/1861694105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb34c60 0x5631eeb39310 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f36e0005fd0 tx=0x7f36e000f950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a557465577c5d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.467+0000 7f36cffff640 1 -- 172.21.15.116:0/1861694105 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e0027070 con 0x5631eeb34c60 2024-03-19T01:16:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.467+0000 7f36cffff640 1 -- 172.21.15.116:0/1861694105 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e000ea10 con 0x5631eeb34c60 2024-03-19T01:16:04.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.467+0000 7f36cffff640 1 -- 172.21.15.116:0/1861694105 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e0022aa0 con 0x5631eeb34c60 2024-03-19T01:16:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.468+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1861694105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb34c60 msgr2=0x5631eeb39310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:04.469 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.468+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/1861694105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb34c60 0x5631eeb39310 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f36e0005fd0 tx=0x7f36e000f950 comp rx=0 tx=0).stop 2024-03-19T01:16:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.470+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1861694105 shutdown_connections 2024-03-19T01:16:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.470+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/1861694105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb34c60 0x5631eeb39310 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.470+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/1861694105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb2faf0 0x5631eeb406d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.470+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/1861694105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2f100 0x5631eeb38dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.470+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1861694105 >> 172.21.15.116:0/1861694105 conn(0x5631eeb2a670 msgr2=0x5631eeb29df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.471+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1861694105 shutdown_connections 2024-03-19T01:16:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.471+0000 7f36ef3f1740 1 -- 172.21.15.116:0/1861694105 wait complete. 2024-03-19T01:16:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.472+0000 7f36ef3f1740 1 -- start start 2024-03-19T01:16:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.472+0000 7f36ef3f1740 1 Processor -- start 2024-03-19T01:16:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.472+0000 7f36ef3f1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2f100 0x5631eeb3dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.472+0000 7f36ef3f1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2faf0 0x5631eeb3e530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.472+0000 7f36e659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2f100 0x5631eeb3dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36e659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2f100 0x5631eeb3dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43874/0 (socket says 172.21.15.116:43874) 2024-03-19T01:16:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36e659c640 1 -- 172.21.15.116:0/3180564591 learned_addr learned my addr 172.21.15.116:0/3180564591 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 0x5631eeb3f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36ef3f1740 1 -- 172.21.15.116:0/3180564591 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36e81b0a80 con 0x5631eeb2faf0 2024-03-19T01:16:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36ef3f1740 1 -- 172.21.15.116:0/3180564591 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f36e81b02f0 con 0x5631eeb34c60 2024-03-19T01:16:04.474 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36ef3f1740 1 -- 172.21.15.116:0/3180564591 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36e81ad470 con 0x5631eeb2f100 2024-03-19T01:16:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36ef3f1740 10 client.0 _ll_register_callbacks cb 0x7f36e00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:16:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36dffff640 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2faf0 0x5631eeb3e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36e6d9d640 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 0x5631eeb3f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36e659c640 1 -- 172.21.15.116:0/3180564591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 msgr2=0x5631eeb3f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36e659c640 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 0x5631eeb3f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36e659c640 1 -- 172.21.15.116:0/3180564591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2faf0 msgr2=0x5631eeb3e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36e659c640 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2faf0 0x5631eeb3e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36e659c640 1 -- 172.21.15.116:0/3180564591 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631eeb3fae0 con 0x5631eeb2f100 2024-03-19T01:16:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36dffff640 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2faf0 0x5631eeb3e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:16:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36e659c640 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2f100 0x5631eeb3dff0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009780 tx=0x7f36d8044000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32e8bc344f95fc32 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:04.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.473+0000 7f36cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:16:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.474+0000 7f36cdffb640 1 -- 172.21.15.116:0/3180564591 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d8057050 con 0x5631eeb2f100 2024-03-19T01:16:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.474+0000 7f36cdffb640 1 -- 172.21.15.116:0/3180564591 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f36d8032850 con 0x5631eeb2f100 2024-03-19T01:16:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.474+0000 7f36ef3f1740 1 -- 172.21.15.116:0/3180564591 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36e000ea10 con 0x5631eeb2f100 2024-03-19T01:16:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.474+0000 7f36cdffb640 1 -- 172.21.15.116:0/3180564591 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f36d8032a50 con 0x5631eeb2f100 2024-03-19T01:16:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.474+0000 7f36ef3f1740 10 client.25420 fetch_fsmap learned FSMap version 35 2024-03-19T01:16:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.474+0000 7f36ef3f1740 1 -- 172.21.15.116:0/3180564591 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5631eebab1e0 con 0x5631eeb2f100 2024-03-19T01:16:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.475+0000 7f36cdffb640 1 -- 172.21.15.116:0/3180564591 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36d8032bf0 con 0x5631eeb2f100 2024-03-19T01:16:04.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.475+0000 7f36cdffb640 1 -- 172.21.15.116:0/3180564591 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f36d806e420 con 0x5631eeb2f100 2024-03-19T01:16:04.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.475+0000 7f36ef3f1740 10 client.25420 fetch_fsmap finished waiting for FSMap version 35 2024-03-19T01:16:04.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.475+0000 7f36ef3f1740 10 client.25420 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:16:04.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.475+0000 7f36ef3f1740 20 client.25420 populate_metadata read hostname 'smithi116' 2024-03-19T01:16:04.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.475+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5631eebabc40 0x5631eebae060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:04.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.475+0000 7f36ef3f1740 4 client.25420 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:16:04.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.475+0000 7f36ef3f1740 1 -- 172.21.15.116:0/3180564591 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f36e000ea10 con 0x5631eebabc40 2024-03-19T01:16:04.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.476+0000 7f36dffff640 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5631eebabc40 0x5631eebae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:04.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.476+0000 7f36cdffb640 10 client.25420 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:04.478 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.476+0000 7f36dffff640 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5631eebabc40 0x5631eebae060 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da9159ff8f15330a server_cookie=c40afb8449e499bc in_seq=0 out_seq=0 2024-03-19T01:16:04.485 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.484+0000 7f36cdffb640 1 -- 172.21.15.116:0/3180564591 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f36e000ea10 con 0x5631eebabc40 2024-03-19T01:16:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.484+0000 7f36cdffb640 10 client.25420 handle_command_reply: tid=0 2024-03-19T01:16:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.484+0000 7f36ef3f1740 1 client.25420 shutdown 2024-03-19T01:16:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.485+0000 7f36ef3f1740 1 -- 172.21.15.116:0/3180564591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2f100 msgr2=0x5631eeb3dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.485+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2f100 0x5631eeb3dff0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f36d8009780 tx=0x7f36d8044000 comp rx=0 tx=0).stop 2024-03-19T01:16:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.485+0000 7f36ef3f1740 1 -- 172.21.15.116:0/3180564591 shutdown_connections 2024-03-19T01:16:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.485+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5631eeb34c60 0x5631eeb3f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.485+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5631eeb2faf0 0x5631eeb3e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.485+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5631eebabc40 0x5631eebae060 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.485+0000 7f36ef3f1740 1 --2- 172.21.15.116:0/3180564591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5631eeb2f100 0x5631eeb3dff0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.485+0000 7f36ef3f1740 1 -- 172.21.15.116:0/3180564591 >> 172.21.15.116:0/3180564591 conn(0x5631eeb2a670 msgr2=0x5631eeb2e160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.485+0000 7f36cdffb640 0 client.25420 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.485+0000 7f36ef3f1740 1 -- 172.21.15.116:0/3180564591 shutdown_connections 2024-03-19T01:16:04.486 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.485+0000 7f36ef3f1740 1 -- 172.21.15.116:0/3180564591 wait complete. 2024-03-19T01:16:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.485+0000 7f36ef3f1740 20 client.25420 trim_cache size 0 max 16384 2024-03-19T01:16:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:16:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:16:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:16:04.487 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:16:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.486+0000 7f36ec80c640 1 -- 172.21.15.116:0/972036468 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f36d40a2970 msgr2=0x7f36d40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.486+0000 7f36ec80c640 1 --2- 172.21.15.116:0/972036468 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f36d40a2970 0x7f36d40a4e30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f36d802c180 tx=0x7f36d8034ab0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.486+0000 7f36ec80c640 1 -- 172.21.15.116:0/972036468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36e810c260 msgr2=0x7f36e81a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.486+0000 7f36ec80c640 1 --2- 172.21.15.116:0/972036468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36e810c260 0x7f36e81a0820 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f36d000e9a0 tx=0x7f36d000ee60 comp rx=0 tx=0).stop 2024-03-19T01:16:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.486+0000 7f36ec80c640 1 -- 172.21.15.116:0/972036468 shutdown_connections 2024-03-19T01:16:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.486+0000 7f36ec80c640 1 --2- 172.21.15.116:0/972036468 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36e810ebe0 0x7f36e81a7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.486+0000 7f36ec80c640 1 --2- 172.21.15.116:0/972036468 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f36d40a2970 0x7f36d40a4e30 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.486+0000 7f36ec80c640 1 --2- 172.21.15.116:0/972036468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36e810c260 0x7f36e81a0820 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.486+0000 7f36ec80c640 1 --2- 172.21.15.116:0/972036468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36e8109930 0x7f36e81a02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:04.488 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.486+0000 7f36ec80c640 1 -- 172.21.15.116:0/972036468 >> 172.21.15.116:0/972036468 conn(0x7f36e806e860 msgr2=0x7f36e8070c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:04.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.486+0000 7f36ec80c640 1 -- 172.21.15.116:0/972036468 shutdown_connections 2024-03-19T01:16:04.489 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:04.486+0000 7f36ec80c640 1 -- 172.21.15.116:0/972036468 wait complete. 2024-03-19T01:16:05.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:05 smithi162 ceph-mon[67178]: pgmap v456: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 MiB/s wr, 1.69k op/s 2024-03-19T01:16:05.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:05 smithi116 ceph-mon[62701]: pgmap v456: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 MiB/s wr, 1.69k op/s 2024-03-19T01:16:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:05 smithi156 ceph-mon[66799]: pgmap v456: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 MiB/s wr, 1.69k op/s 2024-03-19T01:16:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 200.0 s, 3576.3 tps, lat 28.007 ms stddev 30.597 2024-03-19T01:16:07.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:07 smithi162 ceph-mon[67178]: pgmap v457: 97 pgs: 97 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 MiB/s wr, 1.56k op/s 2024-03-19T01:16:07.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:07 smithi116 ceph-mon[62701]: pgmap v457: 97 pgs: 97 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 MiB/s wr, 1.56k op/s 2024-03-19T01:16:07.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:07 smithi156 ceph-mon[66799]: pgmap v457: 97 pgs: 97 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 MiB/s wr, 1.56k op/s 2024-03-19T01:16:09.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:09 smithi162 ceph-mon[67178]: pgmap v458: 97 pgs: 97 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 44 MiB/s wr, 1.48k op/s 2024-03-19T01:16:09.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:09 smithi116 ceph-mon[62701]: pgmap v458: 97 pgs: 97 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 44 MiB/s wr, 1.48k op/s 2024-03-19T01:16:09.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:09 smithi156 ceph-mon[66799]: pgmap v458: 97 pgs: 97 active+clean; 1.5 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 44 MiB/s wr, 1.48k op/s 2024-03-19T01:16:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 205.0 s, 3324.4 tps, lat 30.057 ms stddev 32.297 2024-03-19T01:16:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:11 smithi162 ceph-mon[67178]: pgmap v459: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 27 MiB/s wr, 1.08k op/s 2024-03-19T01:16:11.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:11 smithi116 ceph-mon[62701]: pgmap v459: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 27 MiB/s wr, 1.08k op/s 2024-03-19T01:16:11.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:11 smithi156 ceph-mon[66799]: pgmap v459: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 27 MiB/s wr, 1.08k op/s 2024-03-19T01:16:13.525 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:13 smithi156 ceph-mon[66799]: pgmap v460: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 14 MiB/s wr, 751 op/s 2024-03-19T01:16:13.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:13 smithi162 ceph-mon[67178]: pgmap v460: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 14 MiB/s wr, 751 op/s 2024-03-19T01:16:13.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:13 smithi116 ceph-mon[62701]: pgmap v460: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 14 MiB/s wr, 751 op/s 2024-03-19T01:16:15.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:15 smithi162 ceph-mon[67178]: pgmap v461: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 837 op/s 2024-03-19T01:16:15.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:15 smithi116 ceph-mon[62701]: pgmap v461: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 837 op/s 2024-03-19T01:16:15.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:15 smithi156 ceph-mon[66799]: pgmap v461: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 837 op/s 2024-03-19T01:16:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 210.0 s, 3076.6 tps, lat 32.492 ms stddev 34.179 2024-03-19T01:16:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.620+0000 7fe57967e640 1 -- 172.21.15.116:0/224606774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe574107560 msgr2=0x7fe5741099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.620+0000 7fe57967e640 1 --2- 172.21.15.116:0/224606774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe574107560 0x7fe5741099b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009ca0 tx=0x7fe56c02c100 comp rx=0 tx=0).stop 2024-03-19T01:16:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.620+0000 7fe57967e640 1 -- 172.21.15.116:0/224606774 shutdown_connections 2024-03-19T01:16:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.620+0000 7fe57967e640 1 --2- 172.21.15.116:0/224606774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe574107560 0x7fe5741099b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.620+0000 7fe57967e640 1 --2- 172.21.15.116:0/224606774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574104c30 0x7fe574107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.620+0000 7fe57967e640 1 --2- 172.21.15.116:0/224606774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe574102300 0x7fe5741046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.620+0000 7fe57967e640 1 -- 172.21.15.116:0/224606774 >> 172.21.15.116:0/224606774 conn(0x7fe5740fbcf0 msgr2=0x7fe5740fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.621+0000 7fe57967e640 1 -- 172.21.15.116:0/224606774 shutdown_connections 2024-03-19T01:16:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.621+0000 7fe57967e640 1 -- 172.21.15.116:0/224606774 wait complete. 2024-03-19T01:16:17.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.621+0000 7fe57967e640 1 Processor -- start 2024-03-19T01:16:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.622+0000 7fe57967e640 1 -- start start 2024-03-19T01:16:17.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.623+0000 7fe57967e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574102300 0x7fe574196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.623+0000 7fe57967e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe574104c30 0x7fe574196a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.623+0000 7fe57967e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe574107560 0x7fe5741930c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.623+0000 7fe572ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574102300 0x7fe574196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.623+0000 7fe572ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574102300 0x7fe574196550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55162/0 (socket says 172.21.15.116:55162) 2024-03-19T01:16:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.623+0000 7fe572ffd640 1 -- 172.21.15.116:0/2795648071 learned_addr learned my addr 172.21.15.116:0/2795648071 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:17.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.623+0000 7fe57967e640 1 -- 172.21.15.116:0/2795648071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe57410d500 con 0x7fe574102300 2024-03-19T01:16:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.623+0000 7fe57967e640 1 -- 172.21.15.116:0/2795648071 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe57410d380 con 0x7fe574104c30 2024-03-19T01:16:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.623+0000 7fe57967e640 1 -- 172.21.15.116:0/2795648071 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe57410d680 con 0x7fe574107560 2024-03-19T01:16:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.623+0000 7fe5737fe640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe574107560 0x7fe5741930c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.624+0000 7fe5737fe640 1 -- 172.21.15.116:0/2795648071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe574104c30 msgr2=0x7fe574196a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.623+0000 7fe5727fc640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe574104c30 0x7fe574196a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.624+0000 7fe5737fe640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe574104c30 0x7fe574196a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.624+0000 7fe5737fe640 1 -- 172.21.15.116:0/2795648071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574102300 msgr2=0x7fe574196550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.624+0000 7fe5737fe640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574102300 0x7fe574196550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.624+0000 7fe5737fe640 1 -- 172.21.15.116:0/2795648071 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe574193980 con 0x7fe574107560 2024-03-19T01:16:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.624+0000 7fe5727fc640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe574104c30 0x7fe574196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:16:17.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.624+0000 7fe5737fe640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe574107560 0x7fe5741930c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe56c037040 tx=0x7fe56c035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.624+0000 7fe55bfff640 1 -- 172.21.15.116:0/2795648071 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c039040 con 0x7fe574107560 2024-03-19T01:16:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.624+0000 7fe55bfff640 1 -- 172.21.15.116:0/2795648071 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe56c044070 con 0x7fe574107560 2024-03-19T01:16:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.624+0000 7fe55bfff640 1 -- 172.21.15.116:0/2795648071 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c004330 con 0x7fe574107560 2024-03-19T01:16:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.625+0000 7fe57967e640 1 -- 172.21.15.116:0/2795648071 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5741a6040 con 0x7fe574107560 2024-03-19T01:16:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.625+0000 7fe57967e640 1 -- 172.21.15.116:0/2795648071 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5741a63f0 con 0x7fe574107560 2024-03-19T01:16:17.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.627+0000 7fe55bfff640 1 -- 172.21.15.116:0/2795648071 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fe56c0044d0 con 0x7fe574107560 2024-03-19T01:16:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.628+0000 7fe57c268740 1 Processor -- start 2024-03-19T01:16:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.628+0000 7fe57c268740 1 -- start start 2024-03-19T01:16:17.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.628+0000 7fe57c268740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a4963520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.628+0000 7fe57c268740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4963af0 0x5567a49685f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.628+0000 7fe57c268740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4968c60 0x5567a496b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.628+0000 7fe57c268740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5567a48c7be0 con 0x5567a4963af0 2024-03-19T01:16:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.628+0000 7fe57c268740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5567a4870960 con 0x5567a4968c60 2024-03-19T01:16:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.628+0000 7fe57c268740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5567a48c3c80 con 0x5567a4963100 2024-03-19T01:16:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.628+0000 7fe5727fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a4963520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.628+0000 7fe5727fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a4963520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37780/0 (socket says 172.21.15.116:37780) 2024-03-19T01:16:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.628+0000 7fe5727fc640 1 -- 172.21.15.116:0/3931640023 learned_addr learned my addr 172.21.15.116:0/3931640023 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:17.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.628+0000 7fe572ffd640 1 --2- 172.21.15.116:0/3931640023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4963af0 0x5567a49685f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.628+0000 7fe5727fc640 1 --2- 172.21.15.116:0/3931640023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4968c60 0x5567a496b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.629+0000 7fe55bfff640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe55c0a2920 0x7fe55c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.629+0000 7fe5727fc640 1 -- 172.21.15.116:0/3931640023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4968c60 msgr2=0x5567a496b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.629+0000 7fe5727fc640 1 --2- 172.21.15.116:0/3931640023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4968c60 0x5567a496b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.629+0000 7fe5727fc640 1 -- 172.21.15.116:0/3931640023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4963af0 msgr2=0x5567a49685f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.629+0000 7fe5727fc640 1 --2- 172.21.15.116:0/3931640023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4963af0 0x5567a49685f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.629+0000 7fe5727fc640 1 -- 172.21.15.116:0/3931640023 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567a4867d20 con 0x5567a4963100 2024-03-19T01:16:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.629+0000 7fe55bfff640 1 -- 172.21.15.116:0/2795648071 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe56c0f0ff0 con 0x7fe574107560 2024-03-19T01:16:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.629+0000 7fe5727fc640 1 --2- 172.21.15.116:0/3931640023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a4963520 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe564025910 tx=0x7fe564009630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f17ffd0a4da0217 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.629+0000 7fe5737fe640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe55c0a2920 0x7fe55c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.629+0000 7fe5597fa640 1 -- 172.21.15.116:0/3931640023 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5640252d0 con 0x5567a4963100 2024-03-19T01:16:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.629+0000 7fe5597fa640 1 -- 172.21.15.116:0/3931640023 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe56400f760 con 0x5567a4963100 2024-03-19T01:16:17.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.630+0000 7fe5597fa640 1 -- 172.21.15.116:0/3931640023 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe564003850 con 0x5567a4963100 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.630+0000 7fe57c268740 1 -- 172.21.15.116:0/3931640023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 msgr2=0x5567a4963520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.630+0000 7fe57c268740 1 --2- 172.21.15.116:0/3931640023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a4963520 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe564025910 tx=0x7fe564009630 comp rx=0 tx=0).stop 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.630+0000 7fe5737fe640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe55c0a2920 0x7fe55c0a4de0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0eeb70 tx=0x7fe56c032a00 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.630+0000 7fe57c268740 1 -- 172.21.15.116:0/3931640023 shutdown_connections 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.630+0000 7fe57c268740 1 --2- 172.21.15.116:0/3931640023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4968c60 0x5567a496b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.630+0000 7fe57c268740 1 --2- 172.21.15.116:0/3931640023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4963af0 0x5567a49685f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.630+0000 7fe57c268740 1 --2- 172.21.15.116:0/3931640023 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a4963520 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.630+0000 7fe57c268740 1 -- 172.21.15.116:0/3931640023 >> 172.21.15.116:0/3931640023 conn(0x5567a495e670 msgr2=0x5567a495ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.630+0000 7fe57c268740 1 -- 172.21.15.116:0/3931640023 shutdown_connections 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.630+0000 7fe57c268740 1 -- 172.21.15.116:0/3931640023 wait complete. 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe57c268740 1 -- start start 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe57c268740 1 Processor -- start 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe57c268740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963100 0x5567a496d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe57c268740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963af0 0x5567a496d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe57c268740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4968c60 0x5567a4974640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe57c268740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5567a496bc50 con 0x5567a4968c60 2024-03-19T01:16:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe57c268740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5567a4883db0 con 0x5567a4963100 2024-03-19T01:16:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe57c268740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5567a496bdd0 con 0x5567a4963af0 2024-03-19T01:16:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe57c268740 10 client.0 _ll_register_callbacks cb 0x5567a4888e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:16:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe5727fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963100 0x5567a496d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe5727fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963100 0x5567a496d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:39300/0 (socket says 172.21.15.116:39300) 2024-03-19T01:16:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe5727fc640 1 -- 172.21.15.116:0/624251081 learned_addr learned my addr 172.21.15.116:0/624251081 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe5727fc640 1 --2- 172.21.15.116:0/624251081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4968c60 0x5567a4974640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.631+0000 7fe572ffd640 1 --2- 172.21.15.116:0/624251081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963af0 0x5567a496d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.632+0000 7fe5727fc640 1 -- 172.21.15.116:0/624251081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963af0 msgr2=0x5567a496d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.632+0000 7fe5727fc640 1 --2- 172.21.15.116:0/624251081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963af0 0x5567a496d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.632+0000 7fe5727fc640 1 -- 172.21.15.116:0/624251081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4968c60 msgr2=0x5567a4974640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:17.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.632+0000 7fe5727fc640 1 --2- 172.21.15.116:0/624251081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4968c60 0x5567a4974640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.632+0000 7fe5727fc640 1 -- 172.21.15.116:0/624251081 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567a4882c60 con 0x5567a4963100 2024-03-19T01:16:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.632+0000 7fe5727fc640 1 --2- 172.21.15.116:0/624251081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963100 0x5567a496d0c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe564025250 tx=0x7fe5640254e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aafdcc414ce6945 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.632+0000 7fe556ff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:16:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.632+0000 7fe556ff5640 1 -- 172.21.15.116:0/624251081 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56400f910 con 0x5567a4963100 2024-03-19T01:16:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.632+0000 7fe57c268740 1 -- 172.21.15.116:0/624251081 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe56400f760 con 0x5567a4963100 2024-03-19T01:16:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.632+0000 7fe556ff5640 1 -- 172.21.15.116:0/624251081 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe56400fab0 con 0x5567a4963100 2024-03-19T01:16:17.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.633+0000 7fe556ff5640 1 -- 172.21.15.116:0/624251081 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe564045050 con 0x5567a4963100 2024-03-19T01:16:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.634+0000 7fe57c268740 10 client.25457 fetch_fsmap learned FSMap version 35 2024-03-19T01:16:17.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.634+0000 7fe57c268740 1 -- 172.21.15.116:0/624251081 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5567a45dc210 con 0x5567a4963100 2024-03-19T01:16:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.635+0000 7fe556ff5640 1 -- 172.21.15.116:0/624251081 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe56400f760 con 0x5567a4963100 2024-03-19T01:16:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.635+0000 7fe556ff5640 1 -- 172.21.15.116:0/624251081 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7fe56403d020 con 0x5567a4963100 2024-03-19T01:16:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.635+0000 7fe57c268740 10 client.25457 fetch_fsmap finished waiting for FSMap version 35 2024-03-19T01:16:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.635+0000 7fe57c268740 10 client.25457 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:16:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.635+0000 7fe57c268740 20 client.25457 populate_metadata read hostname 'smithi116' 2024-03-19T01:16:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.635+0000 7fe57c268740 1 --2- 172.21.15.116:0/624251081 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5567a4978de0 0x5567a497b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.635+0000 7fe57c268740 4 client.25457 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:16:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.635+0000 7fe57c268740 1 -- 172.21.15.116:0/624251081 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567a48d5120 con 0x5567a4978de0 2024-03-19T01:16:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.635+0000 7fe572ffd640 1 --2- 172.21.15.116:0/624251081 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5567a4978de0 0x5567a497b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.636+0000 7fe556ff5640 10 client.25457 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.636+0000 7fe572ffd640 1 --2- 172.21.15.116:0/624251081 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5567a4978de0 0x5567a497b220 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65eabd6a14c9af6d server_cookie=d90da56fb99d9182 in_seq=0 out_seq=0 2024-03-19T01:16:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.636+0000 7fe556ff5640 1 -- 172.21.15.116:0/624251081 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5567a48d5120 con 0x5567a4978de0 2024-03-19T01:16:17.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.636+0000 7fe556ff5640 10 client.25457 handle_command_reply: tid=0 2024-03-19T01:16:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.636+0000 7fe57c268740 1 client.25457 shutdown 2024-03-19T01:16:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.637+0000 7fe57c268740 1 -- 172.21.15.116:0/624251081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963100 msgr2=0x5567a496d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.637+0000 7fe57c268740 1 --2- 172.21.15.116:0/624251081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963100 0x5567a496d0c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe564025250 tx=0x7fe5640254e0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.637+0000 7fe57c268740 1 -- 172.21.15.116:0/624251081 shutdown_connections 2024-03-19T01:16:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.637+0000 7fe57c268740 1 --2- 172.21.15.116:0/624251081 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4968c60 0x5567a4974640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.637+0000 7fe57c268740 1 --2- 172.21.15.116:0/624251081 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5567a4978de0 0x5567a497b220 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.637+0000 7fe57c268740 1 --2- 172.21.15.116:0/624251081 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963af0 0x5567a496d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.637+0000 7fe556ff5640 0 client.25457 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.637+0000 7fe57c268740 1 --2- 172.21.15.116:0/624251081 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963100 0x5567a496d0c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.637+0000 7fe57c268740 1 -- 172.21.15.116:0/624251081 >> 172.21.15.116:0/624251081 conn(0x5567a495e670 msgr2=0x5567a495f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.637+0000 7fe57c268740 1 -- 172.21.15.116:0/624251081 shutdown_connections 2024-03-19T01:16:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.637+0000 7fe57c268740 1 -- 172.21.15.116:0/624251081 wait complete. 2024-03-19T01:16:17.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.637+0000 7fe57c268740 20 client.25457 trim_cache size 0 max 16384 2024-03-19T01:16:17.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:17 smithi162 ceph-mon[67178]: pgmap v462: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 10 MiB/s wr, 714 op/s 2024-03-19T01:16:17.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:17 smithi116 ceph-mon[62701]: pgmap v462: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 10 MiB/s wr, 714 op/s 2024-03-19T01:16:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.662+0000 7fe57c268740 1 Processor -- start 2024-03-19T01:16:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.662+0000 7fe57c268740 1 -- start start 2024-03-19T01:16:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.662+0000 7fe57c268740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a496d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.662+0000 7fe57c268740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963af0 0x5567a49745d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.662+0000 7fe57c268740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4968c60 0x5567a4964590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.662+0000 7fe57c268740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe5741a69f0 con 0x5567a4968c60 2024-03-19T01:16:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.662+0000 7fe57c268740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe5741a8770 con 0x5567a4963af0 2024-03-19T01:16:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.662+0000 7fe57c268740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe5741a70c0 con 0x5567a4963100 2024-03-19T01:16:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe572ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963af0 0x5567a49745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe572ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963af0 0x5567a49745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:39308/0 (socket says 172.21.15.116:39308) 2024-03-19T01:16:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe572ffd640 1 -- 172.21.15.116:0/1312387985 learned_addr learned my addr 172.21.15.116:0/1312387985 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe5727fc640 1 --2- 172.21.15.116:0/1312387985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a496d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe5727fc640 1 --2- 172.21.15.116:0/1312387985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4968c60 0x5567a4964590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe572ffd640 1 -- 172.21.15.116:0/1312387985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 msgr2=0x5567a496d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe572ffd640 1 --2- 172.21.15.116:0/1312387985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a496d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe572ffd640 1 -- 172.21.15.116:0/1312387985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4968c60 msgr2=0x5567a4964590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe572ffd640 1 --2- 172.21.15.116:0/1312387985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4968c60 0x5567a4964590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe572ffd640 1 -- 172.21.15.116:0/1312387985 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567a46a8620 con 0x5567a4963af0 2024-03-19T01:16:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe5727fc640 1 --2- 172.21.15.116:0/1312387985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a496d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:16:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe572ffd640 1 --2- 172.21.15.116:0/1312387985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963af0 0x5567a49745d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5567a495f5e0 tx=0x7fe568005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=698e3064c5df1f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe558ff9640 1 -- 172.21.15.116:0/1312387985 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5680212e0 con 0x5567a4963af0 2024-03-19T01:16:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe558ff9640 1 -- 172.21.15.116:0/1312387985 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe568025070 con 0x5567a4963af0 2024-03-19T01:16:17.666 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.663+0000 7fe558ff9640 1 -- 172.21.15.116:0/1312387985 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe568011440 con 0x5567a4963af0 2024-03-19T01:16:17.669 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.664+0000 7fe57c268740 1 -- 172.21.15.116:0/1312387985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963af0 msgr2=0x5567a49745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:17.669 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.664+0000 7fe57c268740 1 --2- 172.21.15.116:0/1312387985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963af0 0x5567a49745d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5567a495f5e0 tx=0x7fe568005f90 comp rx=0 tx=0).stop 2024-03-19T01:16:17.669 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 -- 172.21.15.116:0/1312387985 shutdown_connections 2024-03-19T01:16:17.669 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 --2- 172.21.15.116:0/1312387985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4968c60 0x5567a4964590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.669 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 --2- 172.21.15.116:0/1312387985 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4963af0 0x5567a49745d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.669 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 --2- 172.21.15.116:0/1312387985 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a496d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 -- 172.21.15.116:0/1312387985 >> 172.21.15.116:0/1312387985 conn(0x5567a495e670 msgr2=0x5567a496ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 -- 172.21.15.116:0/1312387985 shutdown_connections 2024-03-19T01:16:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 -- 172.21.15.116:0/1312387985 wait complete. 2024-03-19T01:16:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 -- start start 2024-03-19T01:16:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 Processor -- start 2024-03-19T01:16:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a4971c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4963af0 0x5567a4972140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4968c60 0x5567a4973100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe5741a7fe0 con 0x5567a4963af0 2024-03-19T01:16:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe5741a7850 con 0x5567a4968c60 2024-03-19T01:16:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe574071e50 con 0x5567a4963100 2024-03-19T01:16:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe57c268740 10 client.0 _ll_register_callbacks cb 0x7fe568009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:16:17.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe5727fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a4971c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe572ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4963af0 0x5567a4972140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe5727fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a4971c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:37798/0 (socket says 172.21.15.116:37798) 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe5727fc640 1 -- 172.21.15.116:0/3639474144 learned_addr learned my addr 172.21.15.116:0/3639474144 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.665+0000 7fe5727fc640 1 --2- 172.21.15.116:0/3639474144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4968c60 0x5567a4973100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe5727fc640 1 -- 172.21.15.116:0/3639474144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4968c60 msgr2=0x5567a4973100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe5727fc640 1 --2- 172.21.15.116:0/3639474144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4968c60 0x5567a4973100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe5727fc640 1 -- 172.21.15.116:0/3639474144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4963af0 msgr2=0x5567a4972140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe5727fc640 1 --2- 172.21.15.116:0/3639474144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4963af0 0x5567a4972140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe5727fc640 1 -- 172.21.15.116:0/3639474144 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567a49736f0 con 0x5567a4963100 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe572ffd640 1 --2- 172.21.15.116:0/3639474144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4963af0 0x5567a4972140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe5727fc640 1 --2- 172.21.15.116:0/3639474144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a4971c00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe5640254b0 tx=0x7fe5640254e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbb821cdfc88fd4a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe556ff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe556ff5640 1 -- 172.21.15.116:0/3639474144 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5640410e0 con 0x5567a4963100 2024-03-19T01:16:17.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe556ff5640 1 -- 172.21.15.116:0/3639474144 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe564041280 con 0x5567a4963100 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe57c268740 1 -- 172.21.15.116:0/3639474144 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe568025070 con 0x5567a4963100 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe556ff5640 1 -- 172.21.15.116:0/3639474144 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe5640414c0 con 0x5567a4963100 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe57c268740 10 client.25438 fetch_fsmap learned FSMap version 35 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.666+0000 7fe57c268740 1 -- 172.21.15.116:0/3639474144 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5567a49df120 con 0x5567a4963100 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.667+0000 7fe556ff5640 1 -- 172.21.15.116:0/3639474144 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe568025070 con 0x5567a4963100 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.667+0000 7fe556ff5640 1 -- 172.21.15.116:0/3639474144 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7fe56403d040 con 0x5567a4963100 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.667+0000 7fe57c268740 10 client.25438 fetch_fsmap finished waiting for FSMap version 35 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.667+0000 7fe57c268740 10 client.25438 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.667+0000 7fe57c268740 20 client.25438 populate_metadata read hostname 'smithi116' 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.667+0000 7fe57c268740 1 --2- 172.21.15.116:0/3639474144 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5567a4978de0 0x5567a49e1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.667+0000 7fe57c268740 4 client.25438 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.667+0000 7fe57c268740 1 -- 172.21.15.116:0/3639474144 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5567a48cf470 con 0x5567a4978de0 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.667+0000 7fe572ffd640 1 --2- 172.21.15.116:0/3639474144 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5567a4978de0 0x5567a49e1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.668+0000 7fe572ffd640 1 --2- 172.21.15.116:0/3639474144 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5567a4978de0 0x5567a49e1b30 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49d4482178475083 server_cookie=f2608edb790768b4 in_seq=0 out_seq=0 2024-03-19T01:16:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.668+0000 7fe556ff5640 10 client.25438 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.676+0000 7fe556ff5640 1 -- 172.21.15.116:0/3639474144 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5567a48cf470 con 0x5567a4978de0 2024-03-19T01:16:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.676+0000 7fe556ff5640 10 client.25438 handle_command_reply: tid=0 2024-03-19T01:16:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.676+0000 7fe57c268740 1 client.25438 shutdown 2024-03-19T01:16:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.676+0000 7fe57c268740 1 -- 172.21.15.116:0/3639474144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 msgr2=0x5567a4971c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.676+0000 7fe57c268740 1 --2- 172.21.15.116:0/3639474144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a4971c00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe5640254b0 tx=0x7fe5640254e0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.676+0000 7fe57c268740 1 -- 172.21.15.116:0/3639474144 shutdown_connections 2024-03-19T01:16:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.676+0000 7fe57c268740 1 --2- 172.21.15.116:0/3639474144 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5567a4968c60 0x5567a4973100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.676+0000 7fe57c268740 1 --2- 172.21.15.116:0/3639474144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5567a4963af0 0x5567a4972140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.676+0000 7fe57c268740 1 --2- 172.21.15.116:0/3639474144 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5567a4978de0 0x5567a49e1b30 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.676+0000 7fe57c268740 1 --2- 172.21.15.116:0/3639474144 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5567a4963100 0x5567a4971c00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.676+0000 7fe556ff5640 0 client.25438 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.676+0000 7fe57c268740 1 -- 172.21.15.116:0/3639474144 >> 172.21.15.116:0/3639474144 conn(0x5567a495e670 msgr2=0x5567a4964590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.677+0000 7fe57c268740 1 -- 172.21.15.116:0/3639474144 shutdown_connections 2024-03-19T01:16:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.677+0000 7fe57c268740 1 -- 172.21.15.116:0/3639474144 wait complete. 2024-03-19T01:16:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.677+0000 7fe57c268740 20 client.25438 trim_cache size 0 max 16384 2024-03-19T01:16:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:16:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:16:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:16:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:16:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.679+0000 7fe57967e640 1 -- 172.21.15.116:0/2795648071 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe55c0a2920 msgr2=0x7fe55c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:17.680 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.679+0000 7fe57967e640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe55c0a2920 0x7fe55c0a4de0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe56c0eeb70 tx=0x7fe56c032a00 comp rx=0 tx=0).stop 2024-03-19T01:16:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.679+0000 7fe57967e640 1 -- 172.21.15.116:0/2795648071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe574107560 msgr2=0x7fe5741930c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.679+0000 7fe57967e640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe574107560 0x7fe5741930c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe56c037040 tx=0x7fe56c035d10 comp rx=0 tx=0).stop 2024-03-19T01:16:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.679+0000 7fe57967e640 1 -- 172.21.15.116:0/2795648071 shutdown_connections 2024-03-19T01:16:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.679+0000 7fe57967e640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe574107560 0x7fe5741930c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.679+0000 7fe57967e640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe55c0a2920 0x7fe55c0a4de0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.679+0000 7fe57967e640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe574104c30 0x7fe574196a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.679+0000 7fe57967e640 1 --2- 172.21.15.116:0/2795648071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574102300 0x7fe574196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.679+0000 7fe57967e640 1 -- 172.21.15.116:0/2795648071 >> 172.21.15.116:0/2795648071 conn(0x7fe5740fbcf0 msgr2=0x7fe5741018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.679+0000 7fe57967e640 1 -- 172.21.15.116:0/2795648071 shutdown_connections 2024-03-19T01:16:17.681 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:17.679+0000 7fe57967e640 1 -- 172.21.15.116:0/2795648071 wait complete. 2024-03-19T01:16:17.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:17 smithi156 ceph-mon[66799]: pgmap v462: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 10 MiB/s wr, 714 op/s 2024-03-19T01:16:19.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:19 smithi162 ceph-mon[67178]: pgmap v463: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 735 op/s 2024-03-19T01:16:19.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:19 smithi116 ceph-mon[62701]: pgmap v463: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 735 op/s 2024-03-19T01:16:19.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:19 smithi156 ceph-mon[66799]: pgmap v463: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 735 op/s 2024-03-19T01:16:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 215.0 s, 3293.6 tps, lat 30.400 ms stddev 34.633 2024-03-19T01:16:21.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:21 smithi162 ceph-mon[67178]: pgmap v464: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.8 MiB/s wr, 736 op/s 2024-03-19T01:16:21.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:21 smithi116 ceph-mon[62701]: pgmap v464: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.8 MiB/s wr, 736 op/s 2024-03-19T01:16:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:21 smithi156 ceph-mon[66799]: pgmap v464: 97 pgs: 97 active+clean; 1.5 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 9.8 MiB/s wr, 736 op/s 2024-03-19T01:16:23.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:23 smithi162 ceph-mon[67178]: pgmap v465: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.2 MiB/s wr, 647 op/s 2024-03-19T01:16:23.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:23 smithi116 ceph-mon[62701]: pgmap v465: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.2 MiB/s wr, 647 op/s 2024-03-19T01:16:23.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:23 smithi156 ceph-mon[66799]: pgmap v465: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.2 MiB/s wr, 647 op/s 2024-03-19T01:16:25.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:25 smithi162 ceph-mon[67178]: pgmap v466: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 804 op/s 2024-03-19T01:16:25.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:25 smithi116 ceph-mon[62701]: pgmap v466: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 804 op/s 2024-03-19T01:16:25.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:25 smithi156 ceph-mon[66799]: pgmap v466: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.9 MiB/s wr, 804 op/s 2024-03-19T01:16:26.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 220.0 s, 3634.6 tps, lat 27.404 ms stddev 29.306 2024-03-19T01:16:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:27 smithi162 ceph-mon[67178]: pgmap v467: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 746 op/s 2024-03-19T01:16:27.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:27 smithi116 ceph-mon[62701]: pgmap v467: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 746 op/s 2024-03-19T01:16:27.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:27 smithi156 ceph-mon[66799]: pgmap v467: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 746 op/s 2024-03-19T01:16:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:29 smithi162 ceph-mon[67178]: pgmap v468: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 876 op/s 2024-03-19T01:16:29.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:29 smithi116 ceph-mon[62701]: pgmap v468: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 876 op/s 2024-03-19T01:16:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:29 smithi156 ceph-mon[66799]: pgmap v468: 97 pgs: 97 active+clean; 1.5 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 MiB/s wr, 876 op/s 2024-03-19T01:16:30.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.815+0000 7f545d401640 1 -- 172.21.15.116:0/4071616179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5458109930 msgr2=0x7f545810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.815+0000 7f545d401640 1 --2- 172.21.15.116:0/4071616179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5458109930 0x7f545810bd20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5448009860 tx=0x7f544802bb70 comp rx=0 tx=0).stop 2024-03-19T01:16:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.815+0000 7f545d401640 1 -- 172.21.15.116:0/4071616179 shutdown_connections 2024-03-19T01:16:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.815+0000 7f545d401640 1 --2- 172.21.15.116:0/4071616179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f545810ebe0 0x7f5458111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.815+0000 7f545d401640 1 --2- 172.21.15.116:0/4071616179 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f545810c260 0x7f545810e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.815+0000 7f545d401640 1 --2- 172.21.15.116:0/4071616179 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5458109930 0x7f545810bd20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.815+0000 7f545d401640 1 -- 172.21.15.116:0/4071616179 >> 172.21.15.116:0/4071616179 conn(0x7f545806ea20 msgr2=0x7f5458070e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.815+0000 7f545d401640 1 -- 172.21.15.116:0/4071616179 shutdown_connections 2024-03-19T01:16:30.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.815+0000 7f545d401640 1 -- 172.21.15.116:0/4071616179 wait complete. 2024-03-19T01:16:30.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f545d401640 1 Processor -- start 2024-03-19T01:16:30.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f545d401640 1 -- start start 2024-03-19T01:16:30.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f545d401640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5458109930 0x7f54581a6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f545d401640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f545810c260 0x7f54581a6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f545d401640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f545810ebe0 0x7f54581adf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f545d401640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5458113ae0 con 0x7f545810c260 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f545d401640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5458113960 con 0x7f5458109930 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f545d401640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5458113c60 con 0x7f545810ebe0 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f544ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f545810c260 0x7f54581a6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f5456ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5458109930 0x7f54581a6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f5456ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5458109930 0x7f54581a6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:36766/0 (socket says 172.21.15.116:36766) 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f544ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f545810c260 0x7f54581a6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38706/0 (socket says 172.21.15.116:38706) 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f54577fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f545810ebe0 0x7f54581adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.816+0000 7f5456ffd640 1 -- 172.21.15.116:0/497098981 learned_addr learned my addr 172.21.15.116:0/497098981 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.817+0000 7f5456ffd640 1 -- 172.21.15.116:0/497098981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f545810ebe0 msgr2=0x7f54581adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.817+0000 7f5456ffd640 1 --2- 172.21.15.116:0/497098981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f545810ebe0 0x7f54581adf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.817+0000 7f5456ffd640 1 -- 172.21.15.116:0/497098981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f545810c260 msgr2=0x7f54581a6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.817+0000 7f5456ffd640 1 --2- 172.21.15.116:0/497098981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f545810c260 0x7f54581a6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.817+0000 7f5456ffd640 1 -- 172.21.15.116:0/497098981 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54581ae580 con 0x7f5458109930 2024-03-19T01:16:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.817+0000 7f54577fe640 1 --2- 172.21.15.116:0/497098981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f545810ebe0 0x7f54581adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:16:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.817+0000 7f5456ffd640 1 --2- 172.21.15.116:0/497098981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5458109930 0x7f54581a6960 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f54480349e0 tx=0x7f5448032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.817+0000 7f5454ff9640 1 -- 172.21.15.116:0/497098981 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5448041020 con 0x7f5458109930 2024-03-19T01:16:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.817+0000 7f545d401640 1 -- 172.21.15.116:0/497098981 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54581ae810 con 0x7f5458109930 2024-03-19T01:16:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.817+0000 7f5454ff9640 1 -- 172.21.15.116:0/497098981 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f544802c9f0 con 0x7f5458109930 2024-03-19T01:16:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.817+0000 7f5454ff9640 1 -- 172.21.15.116:0/497098981 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5448039040 con 0x7f5458109930 2024-03-19T01:16:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.817+0000 7f545d401640 1 -- 172.21.15.116:0/497098981 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54581aed00 con 0x7f5458109930 2024-03-19T01:16:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.820+0000 7f545ffe0740 1 Processor -- start 2024-03-19T01:16:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.820+0000 7f5454ff9640 1 -- 172.21.15.116:0/497098981 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f5448044020 con 0x7f5458109930 2024-03-19T01:16:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.820+0000 7f545ffe0740 1 -- start start 2024-03-19T01:16:30.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.820+0000 7f545ffe0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 0x564e309fd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f545ffe0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 0x564e30a025f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f545ffe0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e30a02c60 0x564e30a050f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f545ffe0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564e30961be0 con 0x564e30a02c60 2024-03-19T01:16:30.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f545ffe0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564e3090a960 con 0x564e309fdaf0 2024-03-19T01:16:30.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f545ffe0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564e3095dc80 con 0x564e309fd100 2024-03-19T01:16:30.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f54577fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 0x564e309fd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f54577fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 0x564e309fd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46842/0 (socket says 172.21.15.116:46842) 2024-03-19T01:16:30.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f54577fe640 1 -- 172.21.15.116:0/2522333706 learned_addr learned my addr 172.21.15.116:0/2522333706 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:30.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f5456ffd640 1 --2- 172.21.15.116:0/2522333706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 0x564e30a025f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f544ffff640 1 --2- 172.21.15.116:0/2522333706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e30a02c60 0x564e30a050f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f54577fe640 1 -- 172.21.15.116:0/2522333706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 msgr2=0x564e30a025f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:30.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f54577fe640 1 --2- 172.21.15.116:0/2522333706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 0x564e30a025f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f54577fe640 1 -- 172.21.15.116:0/2522333706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e30a02c60 msgr2=0x564e30a050f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:30.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f54577fe640 1 --2- 172.21.15.116:0/2522333706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e30a02c60 0x564e30a050f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f54577fe640 1 -- 172.21.15.116:0/2522333706 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e30901d20 con 0x564e309fd100 2024-03-19T01:16:30.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f54577fe640 1 --2- 172.21.15.116:0/2522333706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 0x564e309fd520 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f545000d780 tx=0x7f54500095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44409100c4e14a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:30.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f5456ffd640 1 --2- 172.21.15.116:0/2522333706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 0x564e30a025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:16:30.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.821+0000 7f544ffff640 1 --2- 172.21.15.116:0/2522333706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e30a02c60 0x564e30a050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:16:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.822+0000 7f5454ff9640 1 --2- 172.21.15.116:0/497098981 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f54440a2920 0x7f54440a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.822+0000 7f5456ffd640 1 --2- 172.21.15.116:0/497098981 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f54440a2920 0x7f54440a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.822+0000 7f5454ff9640 1 -- 172.21.15.116:0/497098981 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f54480f0300 con 0x7f5458109930 2024-03-19T01:16:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.823+0000 7f544d7fa640 1 -- 172.21.15.116:0/2522333706 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545000cc80 con 0x564e309fd100 2024-03-19T01:16:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.823+0000 7f5456ffd640 1 --2- 172.21.15.116:0/497098981 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f54440a2920 0x7f54440a4de0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f54480bc920 tx=0x7f54480bf040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.823+0000 7f544d7fa640 1 -- 172.21.15.116:0/2522333706 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5450003e50 con 0x564e309fd100 2024-03-19T01:16:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.823+0000 7f544d7fa640 1 -- 172.21.15.116:0/2522333706 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450017400 con 0x564e309fd100 2024-03-19T01:16:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.824+0000 7f545ffe0740 1 -- 172.21.15.116:0/2522333706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 msgr2=0x564e309fd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.824+0000 7f545ffe0740 1 --2- 172.21.15.116:0/2522333706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 0x564e309fd520 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f545000d780 tx=0x7f54500095c0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.824+0000 7f545ffe0740 1 -- 172.21.15.116:0/2522333706 shutdown_connections 2024-03-19T01:16:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.824+0000 7f545ffe0740 1 --2- 172.21.15.116:0/2522333706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e30a02c60 0x564e30a050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.824+0000 7f545ffe0740 1 --2- 172.21.15.116:0/2522333706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 0x564e30a025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.824+0000 7f545ffe0740 1 --2- 172.21.15.116:0/2522333706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 0x564e309fd520 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.824+0000 7f545ffe0740 1 -- 172.21.15.116:0/2522333706 >> 172.21.15.116:0/2522333706 conn(0x564e309f8670 msgr2=0x564e309f8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.824+0000 7f545ffe0740 1 -- 172.21.15.116:0/2522333706 shutdown_connections 2024-03-19T01:16:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.824+0000 7f545ffe0740 1 -- 172.21.15.116:0/2522333706 wait complete. 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.824+0000 7f545ffe0740 1 -- start start 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.824+0000 7f545ffe0740 1 Processor -- start 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.825+0000 7f545ffe0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 0x564e30a071d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.825+0000 7f545ffe0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 0x564e30a07710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.825+0000 7f54577fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 0x564e30a071d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.825+0000 7f54577fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 0x564e30a071d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46850/0 (socket says 172.21.15.116:46850) 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.825+0000 7f54577fe640 1 -- 172.21.15.116:0/4119898667 learned_addr learned my addr 172.21.15.116:0/4119898667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.825+0000 7f544ffff640 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 0x564e30a07710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.825+0000 7f545ffe0740 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e30a02c60 0x564e30a0e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.825+0000 7f545ffe0740 1 -- 172.21.15.116:0/4119898667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564e30a05c60 con 0x564e30a02c60 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.825+0000 7f545ffe0740 1 -- 172.21.15.116:0/4119898667 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564e3091ddb0 con 0x564e309fdaf0 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.825+0000 7f545ffe0740 1 -- 172.21.15.116:0/4119898667 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564e30a05de0 con 0x564e309fd100 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.825+0000 7f54577fe640 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e30a02c60 0x564e30a0e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.826+0000 7f54577fe640 1 -- 172.21.15.116:0/4119898667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 msgr2=0x564e30a07710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.826+0000 7f54577fe640 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 0x564e30a07710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.826+0000 7f54577fe640 1 -- 172.21.15.116:0/4119898667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e30a02c60 msgr2=0x564e30a0e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.826+0000 7f54577fe640 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e30a02c60 0x564e30a0e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.826+0000 7f54577fe640 1 -- 172.21.15.116:0/4119898667 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e3091cc60 con 0x564e309fd100 2024-03-19T01:16:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.826+0000 7f545ffe0740 10 client.0 _ll_register_callbacks cb 0x564e30922e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:16:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.826+0000 7f545ffe0740 1 -- 172.21.15.116:0/4119898667 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5450003e50 con 0x564e309fd100 2024-03-19T01:16:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.826+0000 7f54577fe640 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 0x564e30a071d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5450000f80 tx=0x7f5450004140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca313e542524602b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:30.830 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.826+0000 7f543e7fc640 10 client.25450 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:16:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.826+0000 7f543e7fc640 1 -- 172.21.15.116:0/4119898667 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450034050 con 0x564e309fd100 2024-03-19T01:16:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.826+0000 7f543e7fc640 1 -- 172.21.15.116:0/4119898667 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f545000cd40 con 0x564e309fd100 2024-03-19T01:16:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.826+0000 7f543e7fc640 1 -- 172.21.15.116:0/4119898667 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5450004a40 con 0x564e309fd100 2024-03-19T01:16:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.827+0000 7f545ffe0740 10 client.25450 fetch_fsmap learned FSMap version 35 2024-03-19T01:16:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.827+0000 7f545ffe0740 1 -- 172.21.15.116:0/4119898667 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({fsmap=35}) -- 0x564e30676210 con 0x564e309fd100 2024-03-19T01:16:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.828+0000 7f543e7fc640 1 -- 172.21.15.116:0/4119898667 <== mon.1 v2:172.21.15.162:3300/0 4 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f545000b030 con 0x564e309fd100 2024-03-19T01:16:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.828+0000 7f545ffe0740 10 client.25450 fetch_fsmap finished waiting for FSMap version 35 2024-03-19T01:16:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.828+0000 7f545ffe0740 10 client.25450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.828+0000 7f545ffe0740 20 client.25450 populate_metadata read hostname 'smithi116' 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.828+0000 7f545ffe0740 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e30a12a10 0x564e30a14ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.828+0000 7f545ffe0740 4 client.25450 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.828+0000 7f545ffe0740 1 -- 172.21.15.116:0/4119898667 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e3096f120 con 0x564e30a12a10 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.828+0000 7f544ffff640 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e30a12a10 0x564e30a14ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.829+0000 7f544ffff640 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e30a12a10 0x564e30a14ed0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1268dc48fb0cb22 server_cookie=cdd38135ba451d94 in_seq=0 out_seq=0 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.829+0000 7f543e7fc640 10 client.25450 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.830+0000 7f543e7fc640 1 -- 172.21.15.116:0/4119898667 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564e3096f120 con 0x564e30a12a10 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.830+0000 7f543e7fc640 10 client.25450 handle_command_reply: tid=0 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.830+0000 7f545ffe0740 1 client.25450 shutdown 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.830+0000 7f545ffe0740 1 -- 172.21.15.116:0/4119898667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 msgr2=0x564e30a071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.830+0000 7f545ffe0740 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 0x564e30a071d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5450000f80 tx=0x7f5450004140 comp rx=0 tx=0).stop 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.830+0000 7f545ffe0740 1 -- 172.21.15.116:0/4119898667 shutdown_connections 2024-03-19T01:16:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.830+0000 7f545ffe0740 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e30a02c60 0x564e30a0e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.830+0000 7f545ffe0740 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 0x564e30a07710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.830+0000 7f545ffe0740 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e30a12a10 0x564e30a14ed0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.830+0000 7f545ffe0740 1 --2- 172.21.15.116:0/4119898667 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fd100 0x564e30a071d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.830+0000 7f545ffe0740 1 -- 172.21.15.116:0/4119898667 >> 172.21.15.116:0/4119898667 conn(0x564e309f8670 msgr2=0x564e309f98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.830+0000 7f543e7fc640 0 client.25450 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.831+0000 7f545ffe0740 1 -- 172.21.15.116:0/4119898667 shutdown_connections 2024-03-19T01:16:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.831+0000 7f545ffe0740 1 -- 172.21.15.116:0/4119898667 wait complete. 2024-03-19T01:16:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.832+0000 7f545ffe0740 20 client.25450 trim_cache size 0 max 16384 2024-03-19T01:16:30.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.853+0000 7f545ffe0740 1 Processor -- start 2024-03-19T01:16:30.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.853+0000 7f545ffe0740 1 -- start start 2024-03-19T01:16:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f545ffe0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 0x564e30a07670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f545ffe0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fdaf0 0x564e309f7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f545ffe0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e30a02c60 0x564e309fe590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f545ffe0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f54581b7c70 con 0x564e309fd100 2024-03-19T01:16:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f545ffe0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f54581b99f0 con 0x564e30a02c60 2024-03-19T01:16:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f545ffe0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f54581b8340 con 0x564e309fdaf0 2024-03-19T01:16:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f54577fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 0x564e30a07670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f544ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fdaf0 0x564e309f7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f54577fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 0x564e30a07670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38752/0 (socket says 172.21.15.116:38752) 2024-03-19T01:16:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f54577fe640 1 -- 172.21.15.116:0/3579858725 learned_addr learned my addr 172.21.15.116:0/3579858725 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f544ffff640 1 --2- 172.21.15.116:0/3579858725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fdaf0 0x564e309f7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46858/0 (socket says 172.21.15.116:46858) 2024-03-19T01:16:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f54577fe640 1 --2- 172.21.15.116:0/3579858725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e30a02c60 0x564e309fe590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f54577fe640 1 -- 172.21.15.116:0/3579858725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fdaf0 msgr2=0x564e309f7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f54577fe640 1 --2- 172.21.15.116:0/3579858725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fdaf0 0x564e309f7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f54577fe640 1 -- 172.21.15.116:0/3579858725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e30a02c60 msgr2=0x564e309fe590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f54577fe640 1 --2- 172.21.15.116:0/3579858725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e30a02c60 0x564e309fe590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.854+0000 7f54577fe640 1 -- 172.21.15.116:0/3579858725 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e309f3ee0 con 0x564e309fd100 2024-03-19T01:16:30.858 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.855+0000 7f54577fe640 1 --2- 172.21.15.116:0/3579858725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 0x564e30a07670 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f545000a010 tx=0x7f5450025040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ec190df313bae9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.855+0000 7f544cff9640 1 -- 172.21.15.116:0/3579858725 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450034040 con 0x564e309fd100 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.855+0000 7f544cff9640 1 -- 172.21.15.116:0/3579858725 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5450003e50 con 0x564e309fd100 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.855+0000 7f544cff9640 1 -- 172.21.15.116:0/3579858725 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5450021bb0 con 0x564e309fd100 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.856+0000 7f545ffe0740 1 -- 172.21.15.116:0/3579858725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 msgr2=0x564e30a07670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.856+0000 7f545ffe0740 1 --2- 172.21.15.116:0/3579858725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 0x564e30a07670 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f545000a010 tx=0x7f5450025040 comp rx=0 tx=0).stop 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.856+0000 7f545ffe0740 1 -- 172.21.15.116:0/3579858725 shutdown_connections 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.856+0000 7f545ffe0740 1 --2- 172.21.15.116:0/3579858725 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e30a02c60 0x564e309fe590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.856+0000 7f545ffe0740 1 --2- 172.21.15.116:0/3579858725 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e309fdaf0 0x564e309f7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.856+0000 7f545ffe0740 1 --2- 172.21.15.116:0/3579858725 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 0x564e30a07670 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.856+0000 7f545ffe0740 1 -- 172.21.15.116:0/3579858725 >> 172.21.15.116:0/3579858725 conn(0x564e309f8670 msgr2=0x564e309f7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.856+0000 7f545ffe0740 1 -- 172.21.15.116:0/3579858725 shutdown_connections 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.856+0000 7f545ffe0740 1 -- 172.21.15.116:0/3579858725 wait complete. 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.857+0000 7f545ffe0740 1 -- start start 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.857+0000 7f545ffe0740 1 Processor -- start 2024-03-19T01:16:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.858+0000 7f545ffe0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 0x564e30a0b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.858+0000 7f54577fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 0x564e30a0b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.858+0000 7f54577fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 0x564e30a0b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38756/0 (socket says 172.21.15.116:38756) 2024-03-19T01:16:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.858+0000 7f54577fe640 1 -- 172.21.15.116:0/10022964 learned_addr learned my addr 172.21.15.116:0/10022964 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.858+0000 7f545ffe0740 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 0x564e30a0bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.858+0000 7f545ffe0740 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e30a02c60 0x564e30a0ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.859+0000 7f544ffff640 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 0x564e30a0bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.859+0000 7f545ffe0740 1 -- 172.21.15.116:0/10022964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f54581b9260 con 0x564e309fd100 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.859+0000 7f545ffe0740 1 -- 172.21.15.116:0/10022964 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f54581b8ad0 con 0x564e309fdaf0 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.859+0000 7f545ffe0740 1 -- 172.21.15.116:0/10022964 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f54581b5be0 con 0x564e30a02c60 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.859+0000 7f54577fe640 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e30a02c60 0x564e30a0ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.859+0000 7f54577fe640 1 -- 172.21.15.116:0/10022964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e30a02c60 msgr2=0x564e30a0ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.859+0000 7f54577fe640 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e30a02c60 0x564e30a0ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.859+0000 7f54577fe640 1 -- 172.21.15.116:0/10022964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 msgr2=0x564e30a0bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.859+0000 7f54577fe640 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 0x564e30a0bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.859+0000 7f54577fe640 1 -- 172.21.15.116:0/10022964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e30742620 con 0x564e309fd100 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.859+0000 7f54577fe640 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 0x564e30a0b960 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5450004d40 tx=0x7f5450004cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d858ac5fe9da6228 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.860+0000 7f543e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.860+0000 7f545ffe0740 10 client.0 _ll_register_callbacks cb 0x7f5450009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.860+0000 7f543e7fc640 1 -- 172.21.15.116:0/10022964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545003d030 con 0x564e309fd100 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.860+0000 7f543e7fc640 1 -- 172.21.15.116:0/10022964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f5450004380 con 0x564e309fd100 2024-03-19T01:16:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.860+0000 7f545ffe0740 1 -- 172.21.15.116:0/10022964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5450003e50 con 0x564e309fd100 2024-03-19T01:16:30.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.860+0000 7f543e7fc640 1 -- 172.21.15.116:0/10022964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5450034070 con 0x564e309fd100 2024-03-19T01:16:30.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.860+0000 7f545ffe0740 10 client.15645 fetch_fsmap learned FSMap version 35 2024-03-19T01:16:30.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.860+0000 7f545ffe0740 1 -- 172.21.15.116:0/10022964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({fsmap=35}) -- 0x564e30a0e390 con 0x564e309fd100 2024-03-19T01:16:30.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.860+0000 7f543e7fc640 1 -- 172.21.15.116:0/10022964 <== mon.0 v2:172.21.15.116:3300/0 4 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f545000b070 con 0x564e309fd100 2024-03-19T01:16:30.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.860+0000 7f545ffe0740 10 client.15645 fetch_fsmap finished waiting for FSMap version 35 2024-03-19T01:16:30.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.861+0000 7f545ffe0740 10 client.15645 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:16:30.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.861+0000 7f545ffe0740 20 client.15645 populate_metadata read hostname 'smithi116' 2024-03-19T01:16:30.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.861+0000 7f545ffe0740 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e30a792a0 0x564e30a7b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:30.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.861+0000 7f545ffe0740 4 client.15645 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:16:30.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.861+0000 7f545ffe0740 1 -- 172.21.15.116:0/10022964 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f5450003e50 con 0x564e30a792a0 2024-03-19T01:16:30.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.861+0000 7f544ffff640 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e30a792a0 0x564e30a7b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:30.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.861+0000 7f544ffff640 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e30a792a0 0x564e30a7b6c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77701b3964580de1 server_cookie=eaa33fe25ce521ca in_seq=0 out_seq=0 2024-03-19T01:16:30.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.861+0000 7f543e7fc640 10 client.15645 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.869+0000 7f543e7fc640 1 -- 172.21.15.116:0/10022964 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5450003e50 con 0x564e30a792a0 2024-03-19T01:16:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.869+0000 7f543e7fc640 10 client.15645 handle_command_reply: tid=0 2024-03-19T01:16:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.869+0000 7f545ffe0740 1 client.15645 shutdown 2024-03-19T01:16:30.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.869+0000 7f545ffe0740 1 -- 172.21.15.116:0/10022964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 msgr2=0x564e30a0b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.869+0000 7f545ffe0740 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 0x564e30a0b960 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5450004d40 tx=0x7f5450004cb0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.869+0000 7f545ffe0740 1 -- 172.21.15.116:0/10022964 shutdown_connections 2024-03-19T01:16:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.869+0000 7f545ffe0740 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e30a792a0 0x564e30a7b6c0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.869+0000 7f545ffe0740 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e30a02c60 0x564e30a0ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.869+0000 7f545ffe0740 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e309fdaf0 0x564e30a0bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.869+0000 7f545ffe0740 1 --2- 172.21.15.116:0/10022964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e309fd100 0x564e30a0b960 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.869+0000 7f545ffe0740 1 -- 172.21.15.116:0/10022964 >> 172.21.15.116:0/10022964 conn(0x564e309f8670 msgr2=0x564e30a074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f543e7fc640 0 client.15645 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545ffe0740 1 -- 172.21.15.116:0/10022964 shutdown_connections 2024-03-19T01:16:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545ffe0740 1 -- 172.21.15.116:0/10022964 wait complete. 2024-03-19T01:16:30.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545ffe0740 20 client.15645 trim_cache size 0 max 16384 2024-03-19T01:16:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:16:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:16:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:16:30.874 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:16:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545d401640 1 -- 172.21.15.116:0/497098981 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f54440a2920 msgr2=0x7f54440a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545d401640 1 --2- 172.21.15.116:0/497098981 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f54440a2920 0x7f54440a4de0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f54480bc920 tx=0x7f54480bf040 comp rx=0 tx=0).stop 2024-03-19T01:16:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545d401640 1 -- 172.21.15.116:0/497098981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5458109930 msgr2=0x7f54581a6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545d401640 1 --2- 172.21.15.116:0/497098981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5458109930 0x7f54581a6960 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f54480349e0 tx=0x7f5448032430 comp rx=0 tx=0).stop 2024-03-19T01:16:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545d401640 1 -- 172.21.15.116:0/497098981 shutdown_connections 2024-03-19T01:16:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545d401640 1 --2- 172.21.15.116:0/497098981 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f545810ebe0 0x7f54581adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545d401640 1 --2- 172.21.15.116:0/497098981 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f545810c260 0x7f54581a6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545d401640 1 --2- 172.21.15.116:0/497098981 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f54440a2920 0x7f54440a4de0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545d401640 1 --2- 172.21.15.116:0/497098981 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5458109930 0x7f54581a6960 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:30.875 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545d401640 1 -- 172.21.15.116:0/497098981 >> 172.21.15.116:0/497098981 conn(0x7f545806ea20 msgr2=0x7f545810a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:30.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545d401640 1 -- 172.21.15.116:0/497098981 shutdown_connections 2024-03-19T01:16:30.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:30.870+0000 7f545d401640 1 -- 172.21.15.116:0/497098981 wait complete. 2024-03-19T01:16:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 225.0 s, 3430.6 tps, lat 29.280 ms stddev 31.840 2024-03-19T01:16:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:31 smithi162 ceph-mon[67178]: pgmap v469: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 MiB/s wr, 884 op/s 2024-03-19T01:16:31.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:31 smithi116 ceph-mon[62701]: pgmap v469: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 MiB/s wr, 884 op/s 2024-03-19T01:16:31.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:31 smithi156 ceph-mon[66799]: pgmap v469: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 MiB/s wr, 884 op/s 2024-03-19T01:16:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:33 smithi162 ceph-mon[67178]: pgmap v470: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 9.7 MiB/s wr, 798 op/s 2024-03-19T01:16:33.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:33 smithi116 ceph-mon[62701]: pgmap v470: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 9.7 MiB/s wr, 798 op/s 2024-03-19T01:16:33.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:33 smithi156 ceph-mon[66799]: pgmap v470: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 9.7 MiB/s wr, 798 op/s 2024-03-19T01:16:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:35 smithi162 ceph-mon[67178]: pgmap v471: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 920 op/s 2024-03-19T01:16:35.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:35 smithi116 ceph-mon[62701]: pgmap v471: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 920 op/s 2024-03-19T01:16:35.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:35 smithi156 ceph-mon[66799]: pgmap v471: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 920 op/s 2024-03-19T01:16:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 230.0 s, 4060.4 tps, lat 24.616 ms stddev 26.194 2024-03-19T01:16:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:37 smithi162 ceph-mon[67178]: pgmap v472: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 9.9 MiB/s wr, 854 op/s 2024-03-19T01:16:37.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:37 smithi116 ceph-mon[62701]: pgmap v472: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 9.9 MiB/s wr, 854 op/s 2024-03-19T01:16:37.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:37 smithi156 ceph-mon[66799]: pgmap v472: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 9.9 MiB/s wr, 854 op/s 2024-03-19T01:16:39.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:39 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:16:39.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:39 smithi162 ceph-mon[67178]: pgmap v473: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.4 MiB/s wr, 839 op/s 2024-03-19T01:16:39.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:39 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:16:39.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:39 smithi116 ceph-mon[62701]: pgmap v473: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.4 MiB/s wr, 839 op/s 2024-03-19T01:16:39.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:39 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:16:39.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:39 smithi156 ceph-mon[66799]: pgmap v473: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.4 MiB/s wr, 839 op/s 2024-03-19T01:16:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 235.0 s, 2563.6 tps, lat 38.698 ms stddev 46.031 2024-03-19T01:16:41.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:41 smithi162 ceph-mon[67178]: pgmap v474: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 8.3 MiB/s wr, 760 op/s 2024-03-19T01:16:41.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:41 smithi116 ceph-mon[62701]: pgmap v474: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 8.3 MiB/s wr, 760 op/s 2024-03-19T01:16:41.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:41 smithi156 ceph-mon[66799]: pgmap v474: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 8.3 MiB/s wr, 760 op/s 2024-03-19T01:16:43.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:43 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:16:43.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:43 smithi156 ceph-mon[66799]: pgmap v475: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.8 MiB/s wr, 699 op/s 2024-03-19T01:16:43.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:43 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:16:43.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:43 smithi162 ceph-mon[67178]: pgmap v475: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.8 MiB/s wr, 699 op/s 2024-03-19T01:16:43.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:43 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:16:43.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:43 smithi116 ceph-mon[62701]: pgmap v475: 97 pgs: 97 active+clean; 1.5 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 7.8 MiB/s wr, 699 op/s 2024-03-19T01:16:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.001+0000 7f707f337640 1 -- 172.21.15.116:0/2884731875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707810b450 msgr2=0x7f707810d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.001+0000 7f707f337640 1 --2- 172.21.15.116:0/2884731875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707810b450 0x7f707810d860 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7074008570 tx=0x7f707400ac00 comp rx=0 tx=0).stop 2024-03-19T01:16:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.001+0000 7f707f337640 1 -- 172.21.15.116:0/2884731875 shutdown_connections 2024-03-19T01:16:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.001+0000 7f707f337640 1 --2- 172.21.15.116:0/2884731875 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f707810def0 0x7f70781103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.001+0000 7f707f337640 1 --2- 172.21.15.116:0/2884731875 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f707810b450 0x7f707810d860 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.001+0000 7f707f337640 1 --2- 172.21.15.116:0/2884731875 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7078108a90 0x7f707810ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.001+0000 7f707f337640 1 -- 172.21.15.116:0/2884731875 >> 172.21.15.116:0/2884731875 conn(0x7f707806ea20 msgr2=0x7f7078070e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.001+0000 7f707f337640 1 -- 172.21.15.116:0/2884731875 shutdown_connections 2024-03-19T01:16:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.001+0000 7f707f337640 1 -- 172.21.15.116:0/2884731875 wait complete. 2024-03-19T01:16:44.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.002+0000 7f707f337640 1 Processor -- start 2024-03-19T01:16:44.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.002+0000 7f707f337640 1 -- start start 2024-03-19T01:16:44.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.002+0000 7f707f337640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7078108a90 0x7f70781a6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.002+0000 7f707f337640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f707810b450 0x7f70781a6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.002+0000 7f707f337640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f707810def0 0x7f70781addd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.002+0000 7f707f337640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7078113a80 con 0x7f7078108a90 2024-03-19T01:16:44.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.002+0000 7f707f337640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7078113900 con 0x7f707810b450 2024-03-19T01:16:44.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.002+0000 7f707f337640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7078113c00 con 0x7f707810def0 2024-03-19T01:16:44.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.002+0000 7f707d8d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f707810def0 0x7f70781addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.003+0000 7f707c8d2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f707810b450 0x7f70781a6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.003+0000 7f707d8d4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f707810def0 0x7f70781addd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47820/0 (socket says 172.21.15.116:47820) 2024-03-19T01:16:44.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.003+0000 7f707d8d4640 1 -- 172.21.15.116:0/595245558 learned_addr learned my addr 172.21.15.116:0/595245558 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.003+0000 7f707c8d2640 1 -- 172.21.15.116:0/595245558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f707810def0 msgr2=0x7f70781addd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.003+0000 7f707c8d2640 1 --2- 172.21.15.116:0/595245558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f707810def0 0x7f70781addd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.003+0000 7f707c8d2640 1 -- 172.21.15.116:0/595245558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7078108a90 msgr2=0x7f70781a6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.003+0000 7f707c8d2640 1 --2- 172.21.15.116:0/595245558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7078108a90 0x7f70781a6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.003+0000 7f707c8d2640 1 -- 172.21.15.116:0/595245558 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70781ae440 con 0x7f707810b450 2024-03-19T01:16:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.003+0000 7f707c8d2640 1 --2- 172.21.15.116:0/595245558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f707810b450 0x7f70781a6d60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7074035040 tx=0x7f707400a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.003+0000 7f706e7fc640 1 -- 172.21.15.116:0/595245558 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7074041070 con 0x7f707810b450 2024-03-19T01:16:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.004+0000 7f707f337640 1 -- 172.21.15.116:0/595245558 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70781ae6d0 con 0x7f707810b450 2024-03-19T01:16:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.004+0000 7f707f337640 1 -- 172.21.15.116:0/595245558 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70781aec40 con 0x7f707810b450 2024-03-19T01:16:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.004+0000 7f706e7fc640 1 -- 172.21.15.116:0/595245558 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70740044d0 con 0x7f707810b450 2024-03-19T01:16:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.004+0000 7f706e7fc640 1 -- 172.21.15.116:0/595245558 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707403daf0 con 0x7f707810b450 2024-03-19T01:16:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.007+0000 7f7081ffc740 1 Processor -- start 2024-03-19T01:16:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.007+0000 7f706e7fc640 1 -- 172.21.15.116:0/595245558 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7074044070 con 0x7f707810b450 2024-03-19T01:16:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.007+0000 7f7081ffc740 1 -- start start 2024-03-19T01:16:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.008+0000 7f7081ffc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8100 0x56167a5d8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.008+0000 7f7081ffc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8af0 0x56167a5dd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.008+0000 7f7081ffc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5ddc60 0x56167a5e00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.008+0000 7f7081ffc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56167a53cbe0 con 0x56167a5d8100 2024-03-19T01:16:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.008+0000 7f7081ffc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56167a4e5960 con 0x56167a5d8af0 2024-03-19T01:16:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.008+0000 7f7081ffc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56167a538c80 con 0x56167a5ddc60 2024-03-19T01:16:44.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.009+0000 7f707d8d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8100 0x56167a5d8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.009+0000 7f707d8d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8100 0x56167a5d8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47508/0 (socket says 172.21.15.116:47508) 2024-03-19T01:16:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.009+0000 7f707d8d4640 1 -- 172.21.15.116:0/2230167844 learned_addr learned my addr 172.21.15.116:0/2230167844 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.009+0000 7f707d0d3640 1 --2- 172.21.15.116:0/2230167844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8af0 0x56167a5dd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.009+0000 7f707d8d4640 1 -- 172.21.15.116:0/2230167844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5ddc60 msgr2=0x56167a5e00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.009+0000 7f707c8d2640 1 --2- 172.21.15.116:0/2230167844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5ddc60 0x56167a5e00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.009+0000 7f707d8d4640 1 --2- 172.21.15.116:0/2230167844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5ddc60 0x56167a5e00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.009+0000 7f707d8d4640 1 -- 172.21.15.116:0/2230167844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8af0 msgr2=0x56167a5dd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.009+0000 7f707d8d4640 1 --2- 172.21.15.116:0/2230167844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8af0 0x56167a5dd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.009+0000 7f707d8d4640 1 -- 172.21.15.116:0/2230167844 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56167a4dcd20 con 0x56167a5d8100 2024-03-19T01:16:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.009+0000 7f707d0d3640 1 --2- 172.21.15.116:0/2230167844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8af0 0x56167a5dd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:16:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.009+0000 7f707c8d2640 1 --2- 172.21.15.116:0/2230167844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5ddc60 0x56167a5e00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:16:44.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.010+0000 7f706e7fc640 1 --2- 172.21.15.116:0/595245558 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70640a29a0 0x7f70640a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.010+0000 7f707d8d4640 1 --2- 172.21.15.116:0/2230167844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8100 0x56167a5d8520 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7070009370 tx=0x7f707000b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913a321fc27b3dc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.010+0000 7f706e7fc640 1 -- 172.21.15.116:0/595245558 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f70740efed0 con 0x7f707810b450 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.010+0000 7f707d0d3640 1 --2- 172.21.15.116:0/595245558 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70640a29a0 0x7f70640a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.010+0000 7f70637fe640 1 -- 172.21.15.116:0/2230167844 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070004b20 con 0x56167a5d8100 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.010+0000 7f70637fe640 1 -- 172.21.15.116:0/2230167844 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7070004cc0 con 0x56167a5d8100 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.010+0000 7f70637fe640 1 -- 172.21.15.116:0/2230167844 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70700187b0 con 0x56167a5d8100 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.011+0000 7f7081ffc740 1 -- 172.21.15.116:0/2230167844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8100 msgr2=0x56167a5d8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.011+0000 7f707d0d3640 1 --2- 172.21.15.116:0/595245558 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70640a29a0 0x7f70640a4e60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x56167a5d8ef0 tx=0x7f7068009530 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.011+0000 7f7081ffc740 1 --2- 172.21.15.116:0/2230167844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8100 0x56167a5d8520 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7070009370 tx=0x7f707000b710 comp rx=0 tx=0).stop 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.011+0000 7f7081ffc740 1 -- 172.21.15.116:0/2230167844 shutdown_connections 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.011+0000 7f7081ffc740 1 --2- 172.21.15.116:0/2230167844 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5ddc60 0x56167a5e00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.011+0000 7f7081ffc740 1 --2- 172.21.15.116:0/2230167844 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8af0 0x56167a5dd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.011+0000 7f7081ffc740 1 --2- 172.21.15.116:0/2230167844 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8100 0x56167a5d8520 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.011+0000 7f7081ffc740 1 -- 172.21.15.116:0/2230167844 >> 172.21.15.116:0/2230167844 conn(0x56167a5d3670 msgr2=0x56167a5d3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.011+0000 7f7081ffc740 1 -- 172.21.15.116:0/2230167844 shutdown_connections 2024-03-19T01:16:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.011+0000 7f7081ffc740 1 -- 172.21.15.116:0/2230167844 wait complete. 2024-03-19T01:16:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.011+0000 7f7081ffc740 1 -- start start 2024-03-19T01:16:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.011+0000 7f7081ffc740 1 Processor -- start 2024-03-19T01:16:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.012+0000 7f7081ffc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8100 0x56167a5e2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.012+0000 7f7081ffc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5d8af0 0x56167a5e26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.012+0000 7f7081ffc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5ddc60 0x56167a5e96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.012+0000 7f7081ffc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56167a5e0c10 con 0x56167a5ddc60 2024-03-19T01:16:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.012+0000 7f7081ffc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56167a4f8db0 con 0x56167a5d8100 2024-03-19T01:16:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.012+0000 7f7081ffc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56167a5e0d90 con 0x56167a5d8af0 2024-03-19T01:16:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.012+0000 7f7081ffc740 10 client.0 _ll_register_callbacks cb 0x56167a4fde70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:16:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.012+0000 7f707d0d3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8100 0x56167a5e2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.012+0000 7f707d0d3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8100 0x56167a5e2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:46034/0 (socket says 172.21.15.116:46034) 2024-03-19T01:16:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.012+0000 7f707d0d3640 1 -- 172.21.15.116:0/54514626 learned_addr learned my addr 172.21.15.116:0/54514626 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.012+0000 7f707c8d2640 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5d8af0 0x56167a5e26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.013+0000 7f707d8d4640 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5ddc60 0x56167a5e96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.013+0000 7f707d0d3640 1 -- 172.21.15.116:0/54514626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5d8af0 msgr2=0x56167a5e26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.013+0000 7f707d0d3640 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5d8af0 0x56167a5e26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.013+0000 7f707d0d3640 1 -- 172.21.15.116:0/54514626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5ddc60 msgr2=0x56167a5e96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.013+0000 7f707d0d3640 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5ddc60 0x56167a5e96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.013+0000 7f707d0d3640 1 -- 172.21.15.116:0/54514626 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56167a4f7c60 con 0x56167a5d8100 2024-03-19T01:16:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.013+0000 7f707d8d4640 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5ddc60 0x56167a5e96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.013+0000 7f707d0d3640 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8100 0x56167a5e2160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7068012870 tx=0x7f70680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89f14d235a3bcfcd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.013+0000 7f7060ff9640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.013+0000 7f7060ff9640 1 -- 172.21.15.116:0/54514626 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068023070 con 0x56167a5d8100 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.014+0000 7f7060ff9640 1 -- 172.21.15.116:0/54514626 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7068004460 con 0x56167a5d8100 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.014+0000 7f7081ffc740 1 -- 172.21.15.116:0/54514626 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7070004cc0 con 0x56167a5d8100 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.014+0000 7f7060ff9640 1 -- 172.21.15.116:0/54514626 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f706802f050 con 0x56167a5d8100 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.014+0000 7f7081ffc740 10 client.25490 fetch_fsmap learned FSMap version 37 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.014+0000 7f7081ffc740 1 -- 172.21.15.116:0/54514626 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x56167a251210 con 0x56167a5d8100 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.015+0000 7f7060ff9640 1 -- 172.21.15.116:0/54514626 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7070004cc0 con 0x56167a5d8100 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.015+0000 7f7060ff9640 1 -- 172.21.15.116:0/54514626 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 37) ==== 8390+0+0 (secure 0 0 0) 0x7f7068003b90 con 0x56167a5d8100 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.015+0000 7f7081ffc740 10 client.25490 fetch_fsmap finished waiting for FSMap version 37 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.015+0000 7f7081ffc740 10 client.25490 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.015+0000 7f7081ffc740 20 client.25490 populate_metadata read hostname 'smithi116' 2024-03-19T01:16:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.015+0000 7f7081ffc740 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56167a5edfc0 0x56167a5f0430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.015+0000 7f7081ffc740 4 client.25490 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:16:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.015+0000 7f7081ffc740 1 -- 172.21.15.116:0/54514626 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56167a54a120 con 0x56167a5edfc0 2024-03-19T01:16:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.015+0000 7f707c8d2640 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56167a5edfc0 0x56167a5f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.015+0000 7f7060ff9640 10 client.25490 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.016+0000 7f707c8d2640 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56167a5edfc0 0x56167a5f0430 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87d1a0eb87f35d6f server_cookie=f3742dce5309b50d in_seq=0 out_seq=0 2024-03-19T01:16:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.017+0000 7f7060ff9640 1 -- 172.21.15.116:0/54514626 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56167a54a120 con 0x56167a5edfc0 2024-03-19T01:16:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.017+0000 7f7060ff9640 10 client.25490 handle_command_reply: tid=0 2024-03-19T01:16:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.017+0000 7f7081ffc740 1 client.25490 shutdown 2024-03-19T01:16:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.017+0000 7f7081ffc740 1 -- 172.21.15.116:0/54514626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8100 msgr2=0x56167a5e2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.017+0000 7f7081ffc740 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8100 0x56167a5e2160 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7068012870 tx=0x7f70680046c0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.018+0000 7f7081ffc740 1 -- 172.21.15.116:0/54514626 shutdown_connections 2024-03-19T01:16:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.018+0000 7f7081ffc740 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5ddc60 0x56167a5e96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.018+0000 7f7081ffc740 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56167a5edfc0 0x56167a5f0430 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.018+0000 7f7081ffc740 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5d8af0 0x56167a5e26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.018+0000 7f7081ffc740 1 --2- 172.21.15.116:0/54514626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8100 0x56167a5e2160 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.018+0000 7f7060ff9640 0 client.25490 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.018+0000 7f7081ffc740 1 -- 172.21.15.116:0/54514626 >> 172.21.15.116:0/54514626 conn(0x56167a5d3670 msgr2=0x56167a5d48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.018+0000 7f7081ffc740 1 -- 172.21.15.116:0/54514626 shutdown_connections 2024-03-19T01:16:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.018+0000 7f7081ffc740 1 -- 172.21.15.116:0/54514626 wait complete. 2024-03-19T01:16:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.018+0000 7f7081ffc740 20 client.25490 trim_cache size 0 max 16384 2024-03-19T01:16:44.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.048+0000 7f7081ffc740 1 Processor -- start 2024-03-19T01:16:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.052+0000 7f7081ffc740 1 -- start start 2024-03-19T01:16:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.052+0000 7f7081ffc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5d8100 0x56167a5d2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.052+0000 7f7081ffc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8af0 0x56167a5e9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.052+0000 7f7081ffc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5ddc60 0x56167a5d6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.052+0000 7f7081ffc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f705c00e5f0 con 0x56167a5d8af0 2024-03-19T01:16:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.052+0000 7f7081ffc740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f705c0102e0 con 0x56167a5ddc60 2024-03-19T01:16:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.052+0000 7f7081ffc740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f705c00ecc0 con 0x56167a5d8100 2024-03-19T01:16:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.053+0000 7f707c8d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8af0 0x56167a5e9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.054+0000 7f707c8d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8af0 0x56167a5e9670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47522/0 (socket says 172.21.15.116:47522) 2024-03-19T01:16:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.054+0000 7f707c8d2640 1 -- 172.21.15.116:0/4218940429 learned_addr learned my addr 172.21.15.116:0/4218940429 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.054+0000 7f707c8d2640 1 -- 172.21.15.116:0/4218940429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5d8100 msgr2=0x56167a5d2ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:16:44.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.054+0000 7f707c8d2640 1 --2- 172.21.15.116:0/4218940429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5d8100 0x56167a5d2ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.054+0000 7f707c8d2640 1 -- 172.21.15.116:0/4218940429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5ddc60 msgr2=0x56167a5d6fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:16:44.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.054+0000 7f707c8d2640 1 --2- 172.21.15.116:0/4218940429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5ddc60 0x56167a5d6fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.054+0000 7f707c8d2640 1 -- 172.21.15.116:0/4218940429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56167a31d620 con 0x56167a5d8af0 2024-03-19T01:16:44.057 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.055+0000 7f707c8d2640 1 --2- 172.21.15.116:0/4218940429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8af0 0x56167a5e9670 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f707402bdb0 tx=0x7f707400c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62861e031c4a260 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.056+0000 7f7062ffd640 1 -- 172.21.15.116:0/4218940429 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707404b040 con 0x56167a5d8af0 2024-03-19T01:16:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.056+0000 7f7062ffd640 1 -- 172.21.15.116:0/4218940429 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70740c4f90 con 0x56167a5d8af0 2024-03-19T01:16:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.056+0000 7f7062ffd640 1 -- 172.21.15.116:0/4218940429 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70740c4b00 con 0x56167a5d8af0 2024-03-19T01:16:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.057+0000 7f7081ffc740 1 -- 172.21.15.116:0/4218940429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8af0 msgr2=0x56167a5e9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.057+0000 7f7081ffc740 1 --2- 172.21.15.116:0/4218940429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8af0 0x56167a5e9670 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f707402bdb0 tx=0x7f707400c040 comp rx=0 tx=0).stop 2024-03-19T01:16:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.057+0000 7f7081ffc740 1 -- 172.21.15.116:0/4218940429 shutdown_connections 2024-03-19T01:16:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.057+0000 7f7081ffc740 1 --2- 172.21.15.116:0/4218940429 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5ddc60 0x56167a5d6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.057+0000 7f7081ffc740 1 --2- 172.21.15.116:0/4218940429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8af0 0x56167a5e9670 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.057+0000 7f7081ffc740 1 --2- 172.21.15.116:0/4218940429 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5d8100 0x56167a5d2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.057+0000 7f7081ffc740 1 -- 172.21.15.116:0/4218940429 >> 172.21.15.116:0/4218940429 conn(0x56167a5d3670 msgr2=0x56167a5d68d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.060+0000 7f7081ffc740 1 -- 172.21.15.116:0/4218940429 shutdown_connections 2024-03-19T01:16:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.060+0000 7f7081ffc740 1 -- 172.21.15.116:0/4218940429 wait complete. 2024-03-19T01:16:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.060+0000 7f7081ffc740 1 -- start start 2024-03-19T01:16:44.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.060+0000 7f7081ffc740 1 Processor -- start 2024-03-19T01:16:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.061+0000 7f7081ffc740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8100 0x56167a5e74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.061+0000 7f7081ffc740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8af0 0x56167a5e7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.061+0000 7f7081ffc740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5ddc60 0x56167a5e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.061+0000 7f7081ffc740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f705c00fbe0 con 0x56167a5d8100 2024-03-19T01:16:44.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.062+0000 7f707c8d2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8af0 0x56167a5e7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.062+0000 7f707c8d2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8af0 0x56167a5e7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:46038/0 (socket says 172.21.15.116:46038) 2024-03-19T01:16:44.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.062+0000 7f707c8d2640 1 -- 172.21.15.116:0/2333265789 learned_addr learned my addr 172.21.15.116:0/2333265789 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:44.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.062+0000 7f707d8d4640 1 --2- 172.21.15.116:0/2333265789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5ddc60 0x56167a5e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.062+0000 7f7081ffc740 1 -- 172.21.15.116:0/2333265789 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f705c00f450 con 0x56167a5d8af0 2024-03-19T01:16:44.063 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.062+0000 7f7081ffc740 1 -- 172.21.15.116:0/2333265789 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f705c00c5d0 con 0x56167a5ddc60 2024-03-19T01:16:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.062+0000 7f7081ffc740 10 client.0 _ll_register_callbacks cb 0x7f7068009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:16:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.062+0000 7f707c8d2640 1 -- 172.21.15.116:0/2333265789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5ddc60 msgr2=0x56167a5e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.062+0000 7f707c8d2640 1 --2- 172.21.15.116:0/2333265789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5ddc60 0x56167a5e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.062+0000 7f707c8d2640 1 -- 172.21.15.116:0/2333265789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8100 msgr2=0x56167a5e74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:44.064 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.062+0000 7f707c8d2640 1 --2- 172.21.15.116:0/2333265789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8100 0x56167a5e74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.062+0000 7f707c8d2640 1 -- 172.21.15.116:0/2333265789 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56167a5e9020 con 0x56167a5d8af0 2024-03-19T01:16:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.063+0000 7f7081ffc740 1 -- 172.21.15.116:0/2333265789 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70740c4f90 con 0x56167a5d8af0 2024-03-19T01:16:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.063+0000 7f707c8d2640 1 --2- 172.21.15.116:0/2333265789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8af0 0x56167a5e7a20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f70740c4a60 tx=0x7f70740bc4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1429730b0a0381f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.063+0000 7f7060ff9640 10 client.25493 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:16:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.063+0000 7f7060ff9640 1 -- 172.21.15.116:0/2333265789 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f707404b040 con 0x56167a5d8af0 2024-03-19T01:16:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.063+0000 7f7060ff9640 1 -- 172.21.15.116:0/2333265789 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f70740bdb20 con 0x56167a5d8af0 2024-03-19T01:16:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.063+0000 7f7060ff9640 1 -- 172.21.15.116:0/2333265789 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f70740bdd20 con 0x56167a5d8af0 2024-03-19T01:16:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.064+0000 7f7081ffc740 10 client.25493 fetch_fsmap learned FSMap version 37 2024-03-19T01:16:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.064+0000 7f7081ffc740 1 -- 172.21.15.116:0/2333265789 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({fsmap=37}) -- 0x56167a6549d0 con 0x56167a5d8af0 2024-03-19T01:16:44.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.066+0000 7f7060ff9640 1 -- 172.21.15.116:0/2333265789 <== mon.2 v2:172.21.15.156:3300/0 4 ==== fsmap(e 37) ==== 8390+0+0 (secure 0 0 0) 0x7f70740f2070 con 0x56167a5d8af0 2024-03-19T01:16:44.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.066+0000 7f7081ffc740 10 client.25493 fetch_fsmap finished waiting for FSMap version 37 2024-03-19T01:16:44.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.066+0000 7f7081ffc740 10 client.25493 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:16:44.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.066+0000 7f7081ffc740 20 client.25493 populate_metadata read hostname 'smithi116' 2024-03-19T01:16:44.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.066+0000 7f7081ffc740 1 --2- 172.21.15.116:0/2333265789 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56167a654fd0 0x56167a6573f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:44.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.066+0000 7f7081ffc740 4 client.25493 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:16:44.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.066+0000 7f7081ffc740 1 -- 172.21.15.116:0/2333265789 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f70740c4f90 con 0x56167a654fd0 2024-03-19T01:16:44.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.066+0000 7f707d0d3640 1 --2- 172.21.15.116:0/2333265789 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56167a654fd0 0x56167a6573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:44.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.068+0000 7f707d0d3640 1 --2- 172.21.15.116:0/2333265789 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56167a654fd0 0x56167a6573f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c7c852db425caa4 server_cookie=e6c5c3e7686dd645 in_seq=0 out_seq=0 2024-03-19T01:16:44.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.068+0000 7f7060ff9640 10 client.25493 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7060ff9640 1 -- 172.21.15.116:0/2333265789 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f70740c4f90 con 0x56167a654fd0 2024-03-19T01:16:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7060ff9640 10 client.25493 handle_command_reply: tid=0 2024-03-19T01:16:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7081ffc740 1 client.25493 shutdown 2024-03-19T01:16:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7081ffc740 1 -- 172.21.15.116:0/2333265789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8af0 msgr2=0x56167a5e7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:44.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7081ffc740 1 --2- 172.21.15.116:0/2333265789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8af0 0x56167a5e7a20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f70740c4a60 tx=0x7f70740bc4e0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7081ffc740 1 -- 172.21.15.116:0/2333265789 shutdown_connections 2024-03-19T01:16:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7081ffc740 1 --2- 172.21.15.116:0/2333265789 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56167a654fd0 0x56167a6573f0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7081ffc740 1 --2- 172.21.15.116:0/2333265789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56167a5ddc60 0x56167a5e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7081ffc740 1 --2- 172.21.15.116:0/2333265789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56167a5d8af0 0x56167a5e7a20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7081ffc740 1 --2- 172.21.15.116:0/2333265789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56167a5d8100 0x56167a5e74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7081ffc740 1 -- 172.21.15.116:0/2333265789 >> 172.21.15.116:0/2333265789 conn(0x56167a5d3670 msgr2=0x56167a5d7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7060ff9640 0 client.25493 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7081ffc740 1 -- 172.21.15.116:0/2333265789 shutdown_connections 2024-03-19T01:16:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.078+0000 7f7081ffc740 1 -- 172.21.15.116:0/2333265789 wait complete. 2024-03-19T01:16:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.079+0000 7f7081ffc740 20 client.25493 trim_cache size 0 max 16384 2024-03-19T01:16:44.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:16:44.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:16:44.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:16:44.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:16:44.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.079+0000 7f707f337640 1 -- 172.21.15.116:0/595245558 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70640a29a0 msgr2=0x7f70640a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:44.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.079+0000 7f707f337640 1 --2- 172.21.15.116:0/595245558 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70640a29a0 0x7f70640a4e60 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x56167a5d8ef0 tx=0x7f7068009530 comp rx=0 tx=0).stop 2024-03-19T01:16:44.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.079+0000 7f707f337640 1 -- 172.21.15.116:0/595245558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f707810b450 msgr2=0x7f70781a6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.079+0000 7f707f337640 1 --2- 172.21.15.116:0/595245558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f707810b450 0x7f70781a6d60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7074035040 tx=0x7f707400a5b0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.079+0000 7f707f337640 1 -- 172.21.15.116:0/595245558 shutdown_connections 2024-03-19T01:16:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.079+0000 7f707f337640 1 --2- 172.21.15.116:0/595245558 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f707810def0 0x7f70781addd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.079+0000 7f707f337640 1 --2- 172.21.15.116:0/595245558 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70640a29a0 0x7f70640a4e60 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.079+0000 7f707f337640 1 --2- 172.21.15.116:0/595245558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f707810b450 0x7f70781a6d60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.079+0000 7f707f337640 1 --2- 172.21.15.116:0/595245558 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7078108a90 0x7f70781a6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.079+0000 7f707f337640 1 -- 172.21.15.116:0/595245558 >> 172.21.15.116:0/595245558 conn(0x7f707806ea20 msgr2=0x7f70781093f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.079+0000 7f707f337640 1 -- 172.21.15.116:0/595245558 shutdown_connections 2024-03-19T01:16:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:44.079+0000 7f707f337640 1 -- 172.21.15.116:0/595245558 wait complete. 2024-03-19T01:16:45.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:45 smithi162 ceph-mon[67178]: pgmap v476: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 798 op/s 2024-03-19T01:16:45.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:45 smithi116 ceph-mon[62701]: pgmap v476: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 798 op/s 2024-03-19T01:16:45.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:45 smithi156 ceph-mon[66799]: pgmap v476: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.1 MiB/s wr, 798 op/s 2024-03-19T01:16:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 240.0 s, 2867.0 tps, lat 35.096 ms stddev 42.972 2024-03-19T01:16:47.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:47 smithi162 ceph-mon[67178]: pgmap v477: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 769 op/s 2024-03-19T01:16:47.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:47 smithi116 ceph-mon[62701]: pgmap v477: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 769 op/s 2024-03-19T01:16:47.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:47 smithi156 ceph-mon[66799]: pgmap v477: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.5 MiB/s wr, 769 op/s 2024-03-19T01:16:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:49 smithi162 ceph-mon[67178]: pgmap v478: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.6 MiB/s wr, 726 op/s 2024-03-19T01:16:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:49 smithi116 ceph-mon[62701]: pgmap v478: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.6 MiB/s wr, 726 op/s 2024-03-19T01:16:49.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:49 smithi156 ceph-mon[66799]: pgmap v478: 97 pgs: 97 active+clean; 1.5 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 8.6 MiB/s wr, 726 op/s 2024-03-19T01:16:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 245.0 s, 3128.6 tps, lat 31.918 ms stddev 36.980 2024-03-19T01:16:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:51 smithi162 ceph-mon[67178]: pgmap v479: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 730 op/s 2024-03-19T01:16:51.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:51 smithi116 ceph-mon[62701]: pgmap v479: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 730 op/s 2024-03-19T01:16:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:51 smithi156 ceph-mon[66799]: pgmap v479: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 730 op/s 2024-03-19T01:16:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:53 smithi162 ceph-mon[67178]: pgmap v480: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 661 op/s 2024-03-19T01:16:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:53 smithi116 ceph-mon[62701]: pgmap v480: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 661 op/s 2024-03-19T01:16:53.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:53 smithi156 ceph-mon[66799]: pgmap v480: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.8 MiB/s wr, 661 op/s 2024-03-19T01:16:55.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:55 smithi162 ceph-mon[67178]: pgmap v481: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 787 op/s 2024-03-19T01:16:55.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:55 smithi116 ceph-mon[62701]: pgmap v481: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 787 op/s 2024-03-19T01:16:55.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:55 smithi156 ceph-mon[66799]: pgmap v481: 97 pgs: 97 active+clean; 1.5 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 787 op/s 2024-03-19T01:16:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 250.0 s, 3355.6 tps, lat 29.824 ms stddev 32.722 2024-03-19T01:16:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.201+0000 7f7fb87b3640 1 -- 172.21.15.116:0/2967448972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fb0103420 msgr2=0x7f7fb0109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.201+0000 7f7fb87b3640 1 --2- 172.21.15.116:0/2967448972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fb0103420 0x7f7fb0109cb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009ca0 tx=0x7f7fa802c100 comp rx=0 tx=0).stop 2024-03-19T01:16:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.201+0000 7f7fb87b3640 1 -- 172.21.15.116:0/2967448972 shutdown_connections 2024-03-19T01:16:57.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.201+0000 7f7fb87b3640 1 --2- 172.21.15.116:0/2967448972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fb0103420 0x7f7fb0109cb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.201+0000 7f7fb87b3640 1 --2- 172.21.15.116:0/2967448972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fb0102a60 0x7f7fb0102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.201+0000 7f7fb87b3640 1 --2- 172.21.15.116:0/2967448972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fb0101870 0x7f7fb0101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.201+0000 7f7fb87b3640 1 -- 172.21.15.116:0/2967448972 >> 172.21.15.116:0/2967448972 conn(0x7f7fb00fcde0 msgr2=0x7f7fb00ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.203+0000 7f7fb87b3640 1 -- 172.21.15.116:0/2967448972 shutdown_connections 2024-03-19T01:16:57.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.203+0000 7f7fb87b3640 1 -- 172.21.15.116:0/2967448972 wait complete. 2024-03-19T01:16:57.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.204+0000 7f7fb87b3640 1 Processor -- start 2024-03-19T01:16:57.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.204+0000 7f7fb87b3640 1 -- start start 2024-03-19T01:16:57.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb87b3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fb0101870 0x7f7fb0199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb87b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fb0102a60 0x7f7fb019a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb87b3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fb0103420 0x7f7fb01a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb87b3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb010f720 con 0x7f7fb0102a60 2024-03-19T01:16:57.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb87b3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb010f5a0 con 0x7f7fb0103420 2024-03-19T01:16:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb87b3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb010f8a0 con 0x7f7fb0101870 2024-03-19T01:16:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb5d4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fb0102a60 0x7f7fb019a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb5d4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fb0102a60 0x7f7fb019a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51896/0 (socket says 172.21.15.116:51896) 2024-03-19T01:16:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb654f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fb0101870 0x7f7fb0199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb5d4e640 1 -- 172.21.15.116:0/2391398885 learned_addr learned my addr 172.21.15.116:0/2391398885 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb6d50640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fb0103420 0x7f7fb01a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb654f640 1 -- 172.21.15.116:0/2391398885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fb0103420 msgr2=0x7f7fb01a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb654f640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fb0103420 0x7f7fb01a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb654f640 1 -- 172.21.15.116:0/2391398885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fb0102a60 msgr2=0x7f7fb019a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:57.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb654f640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fb0102a60 0x7f7fb019a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb654f640 1 -- 172.21.15.116:0/2391398885 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb01a1890 con 0x7f7fb0101870 2024-03-19T01:16:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb6d50640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fb0103420 0x7f7fb01a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:16:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb5d4e640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fb0102a60 0x7f7fb019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:16:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.205+0000 7f7fb654f640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fb0101870 0x7f7fb0199c80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00e9c0 tx=0x7f7fac00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.206+0000 7f7fa77fe640 1 -- 172.21.15.116:0/2391398885 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac00cd80 con 0x7f7fb0101870 2024-03-19T01:16:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.206+0000 7f7fa77fe640 1 -- 172.21.15.116:0/2391398885 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fac010910 con 0x7f7fb0101870 2024-03-19T01:16:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.206+0000 7f7fb87b3640 1 -- 172.21.15.116:0/2391398885 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb01a1b80 con 0x7f7fb0101870 2024-03-19T01:16:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.206+0000 7f7fa77fe640 1 -- 172.21.15.116:0/2391398885 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac00cd80 con 0x7f7fb0101870 2024-03-19T01:16:57.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.206+0000 7f7fb87b3640 1 -- 172.21.15.116:0/2391398885 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb01a2080 con 0x7f7fb0101870 2024-03-19T01:16:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.208+0000 7f7fa77fe640 1 -- 172.21.15.116:0/2391398885 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7fac01f020 con 0x7f7fb0101870 2024-03-19T01:16:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.209+0000 7f7fbb39a740 1 Processor -- start 2024-03-19T01:16:57.212 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.209+0000 7f7fa77fe640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7f9c0a2ab0 0x7f7f9c0a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.209+0000 7f7fb6d50640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7f9c0a2ab0 0x7f7f9c0a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.209+0000 7f7fa77fe640 1 -- 172.21.15.116:0/2391398885 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7fac0ce680 con 0x7f7fb0101870 2024-03-19T01:16:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.209+0000 7f7fbb39a740 1 -- start start 2024-03-19T01:16:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb6d50640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7f9c0a2ab0 0x7f7f9c0a4f70 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7fa802c640 tx=0x7f7fa8032590 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fbb39a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 0x55edd61e9520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fbb39a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9af0 0x55edd61ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fbb39a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61eec60 0x55edd61f10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fbb39a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55edd614dbe0 con 0x55edd61eec60 2024-03-19T01:16:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fbb39a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55edd60f6960 con 0x55edd61e9af0 2024-03-19T01:16:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fbb39a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55edd6149c80 con 0x55edd61e9100 2024-03-19T01:16:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb6d50640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61eec60 0x55edd61f10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb654f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 0x55edd61e9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb5d4e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9af0 0x55edd61ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb654f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 0x55edd61e9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42856/0 (socket says 172.21.15.116:42856) 2024-03-19T01:16:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb654f640 1 -- 172.21.15.116:0/3388273457 learned_addr learned my addr 172.21.15.116:0/3388273457 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb6d50640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61eec60 0x55edd61f10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51902/0 (socket says 172.21.15.116:51902) 2024-03-19T01:16:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb654f640 1 -- 172.21.15.116:0/3388273457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9af0 msgr2=0x55edd61ee5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb654f640 1 --2- 172.21.15.116:0/3388273457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9af0 0x55edd61ee5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb654f640 1 -- 172.21.15.116:0/3388273457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61eec60 msgr2=0x55edd61f10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:57.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb654f640 1 --2- 172.21.15.116:0/3388273457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61eec60 0x55edd61f10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb654f640 1 -- 172.21.15.116:0/3388273457 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edd60edd20 con 0x55edd61e9100 2024-03-19T01:16:57.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb5d4e640 1 --2- 172.21.15.116:0/3388273457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9af0 0x55edd61ee5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:16:57.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.210+0000 7f7fb654f640 1 --2- 172.21.15.116:0/3388273457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 0x55edd61e9520 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7fac09af90 tx=0x7f7fac09aa90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12bf3236c1f6fdc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:57.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.211+0000 7f7fa4ff9640 1 -- 172.21.15.116:0/3388273457 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac09b000 con 0x55edd61e9100 2024-03-19T01:16:57.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.211+0000 7f7fa4ff9640 1 -- 172.21.15.116:0/3388273457 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fac09f150 con 0x55edd61e9100 2024-03-19T01:16:57.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.211+0000 7f7fa4ff9640 1 -- 172.21.15.116:0/3388273457 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac0a3570 con 0x55edd61e9100 2024-03-19T01:16:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.213+0000 7f7fbb39a740 1 -- 172.21.15.116:0/3388273457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 msgr2=0x55edd61e9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.213+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/3388273457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 0x55edd61e9520 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7fac09af90 tx=0x7f7fac09aa90 comp rx=0 tx=0).stop 2024-03-19T01:16:57.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.214+0000 7f7fbb39a740 1 -- 172.21.15.116:0/3388273457 shutdown_connections 2024-03-19T01:16:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.214+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/3388273457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61eec60 0x55edd61f10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.214+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/3388273457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9af0 0x55edd61ee5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.214+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/3388273457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 0x55edd61e9520 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.214+0000 7f7fbb39a740 1 -- 172.21.15.116:0/3388273457 >> 172.21.15.116:0/3388273457 conn(0x55edd61e4670 msgr2=0x55edd61e4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.214+0000 7f7fbb39a740 1 -- 172.21.15.116:0/3388273457 shutdown_connections 2024-03-19T01:16:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.214+0000 7f7fbb39a740 1 -- 172.21.15.116:0/3388273457 wait complete. 2024-03-19T01:16:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.215+0000 7f7fbb39a740 1 -- start start 2024-03-19T01:16:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.215+0000 7f7fbb39a740 1 Processor -- start 2024-03-19T01:16:57.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.215+0000 7f7fbb39a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9100 0x55edd61f3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.215+0000 7f7fbb39a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9af0 0x55edd61f3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.215+0000 7f7fbb39a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61eec60 0x55edd61fa6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.215+0000 7f7fbb39a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55edd61f1c10 con 0x55edd61eec60 2024-03-19T01:16:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.215+0000 7f7fbb39a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55edd6109db0 con 0x55edd61e9100 2024-03-19T01:16:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.215+0000 7f7fbb39a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55edd61f1d90 con 0x55edd61e9af0 2024-03-19T01:16:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.215+0000 7f7fb5d4e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9af0 0x55edd61f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.215+0000 7f7fb5d4e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9af0 0x55edd61f3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42866/0 (socket says 172.21.15.116:42866) 2024-03-19T01:16:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.215+0000 7f7fb5d4e640 1 -- 172.21.15.116:0/3495528137 learned_addr learned my addr 172.21.15.116:0/3495528137 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.215+0000 7f7fb654f640 1 --2- 172.21.15.116:0/3495528137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9100 0x55edd61f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.216+0000 7f7fb6d50640 1 --2- 172.21.15.116:0/3495528137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61eec60 0x55edd61fa6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.215+0000 7f7fbb39a740 10 client.0 _ll_register_callbacks cb 0x55edd610ee70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:16:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.216+0000 7f7fb6d50640 1 -- 172.21.15.116:0/3495528137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9af0 msgr2=0x55edd61f3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.216+0000 7f7fb6d50640 1 --2- 172.21.15.116:0/3495528137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9af0 0x55edd61f3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.216+0000 7f7fb6d50640 1 -- 172.21.15.116:0/3495528137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9100 msgr2=0x55edd61f3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.216+0000 7f7fb6d50640 1 --2- 172.21.15.116:0/3495528137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9100 0x55edd61f3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.216+0000 7f7fb6d50640 1 -- 172.21.15.116:0/3495528137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edd6108c60 con 0x55edd61eec60 2024-03-19T01:16:57.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.217+0000 7f7fb6d50640 1 --2- 172.21.15.116:0/3495528137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61eec60 0x55edd61fa6b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8037040 tx=0x7f7fa802c100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57744333460027b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.221+0000 7f7f99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:16:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.221+0000 7f7f99ffb640 1 -- 172.21.15.116:0/3495528137 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8050020 con 0x55edd61eec60 2024-03-19T01:16:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.222+0000 7f7f99ffb640 1 -- 172.21.15.116:0/3495528137 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7fa8034c00 con 0x55edd61eec60 2024-03-19T01:16:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.222+0000 7f7fbb39a740 1 -- 172.21.15.116:0/3495528137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fac09f150 con 0x55edd61eec60 2024-03-19T01:16:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.222+0000 7f7f99ffb640 1 -- 172.21.15.116:0/3495528137 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f7fa805b050 con 0x55edd61eec60 2024-03-19T01:16:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.222+0000 7f7fbb39a740 10 client.15666 fetch_fsmap learned FSMap version 38 2024-03-19T01:16:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.222+0000 7f7fbb39a740 1 -- 172.21.15.116:0/3495528137 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55edd5e62210 con 0x55edd61eec60 2024-03-19T01:16:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.223+0000 7f7f99ffb640 1 -- 172.21.15.116:0/3495528137 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fac09f150 con 0x55edd61eec60 2024-03-19T01:16:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.223+0000 7f7f99ffb640 1 -- 172.21.15.116:0/3495528137 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f7fa804c560 con 0x55edd61eec60 2024-03-19T01:16:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.223+0000 7f7fbb39a740 10 client.15666 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T01:16:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.223+0000 7f7fbb39a740 10 client.15666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:16:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.223+0000 7f7fbb39a740 20 client.15666 populate_metadata read hostname 'smithi116' 2024-03-19T01:16:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.223+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/3495528137 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55edd61ff430 0x55edd6201890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.223+0000 7f7fbb39a740 4 client.15666 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:16:57.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.223+0000 7f7fbb39a740 1 -- 172.21.15.116:0/3495528137 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55edd615b120 con 0x55edd61ff430 2024-03-19T01:16:57.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.225+0000 7f7fb654f640 1 --2- 172.21.15.116:0/3495528137 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55edd61ff430 0x55edd6201890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.235+0000 7f7fb654f640 1 --2- 172.21.15.116:0/3495528137 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55edd61ff430 0x55edd6201890 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=798a5255ea1f93c6 server_cookie=1a10c9361fd169a5 in_seq=0 out_seq=0 2024-03-19T01:16:57.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.235+0000 7f7f99ffb640 10 client.15666 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.237+0000 7f7f99ffb640 1 -- 172.21.15.116:0/3495528137 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55edd615b120 con 0x55edd61ff430 2024-03-19T01:16:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.237+0000 7f7f99ffb640 10 client.15666 handle_command_reply: tid=0 2024-03-19T01:16:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.237+0000 7f7fbb39a740 1 client.15666 shutdown 2024-03-19T01:16:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.242+0000 7f7fbb39a740 1 -- 172.21.15.116:0/3495528137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61eec60 msgr2=0x55edd61fa6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:57.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.242+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/3495528137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61eec60 0x55edd61fa6b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8037040 tx=0x7f7fa802c100 comp rx=0 tx=0).stop 2024-03-19T01:16:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.242+0000 7f7fbb39a740 1 -- 172.21.15.116:0/3495528137 shutdown_connections 2024-03-19T01:16:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.242+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/3495528137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61eec60 0x55edd61fa6b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.242+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/3495528137 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55edd61ff430 0x55edd6201890 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.242+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/3495528137 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9af0 0x55edd61f3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.242+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/3495528137 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9100 0x55edd61f3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.242+0000 7f7fbb39a740 1 -- 172.21.15.116:0/3495528137 >> 172.21.15.116:0/3495528137 conn(0x55edd61e4670 msgr2=0x55edd61e58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.242+0000 7f7f99ffb640 0 client.15666 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.242+0000 7f7fbb39a740 1 -- 172.21.15.116:0/3495528137 shutdown_connections 2024-03-19T01:16:57.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.242+0000 7f7fbb39a740 1 -- 172.21.15.116:0/3495528137 wait complete. 2024-03-19T01:16:57.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.243+0000 7f7fbb39a740 20 client.15666 trim_cache size 0 max 16384 2024-03-19T01:16:57.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.265+0000 7f7fbb39a740 1 Processor -- start 2024-03-19T01:16:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.265+0000 7f7fbb39a740 1 -- start start 2024-03-19T01:16:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.265+0000 7f7fbb39a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9100 0x55edd61e3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.265+0000 7f7fbb39a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61e9af0 0x55edd61ea380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.265+0000 7f7fbb39a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61eec60 0x55edd61ea8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.265+0000 7f7fbb39a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01ab000 con 0x55edd61e9af0 2024-03-19T01:16:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.265+0000 7f7fbb39a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01acc90 con 0x55edd61e9100 2024-03-19T01:16:57.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.265+0000 7f7fbb39a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01ab6d0 con 0x55edd61eec60 2024-03-19T01:16:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.265+0000 7f7fb6d50640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61eec60 0x55edd61ea8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.265+0000 7f7fb6d50640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61eec60 0x55edd61ea8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42878/0 (socket says 172.21.15.116:42878) 2024-03-19T01:16:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.265+0000 7f7fb6d50640 1 -- 172.21.15.116:0/4054515075 learned_addr learned my addr 172.21.15.116:0/4054515075 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:57.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.266+0000 7f7fb654f640 1 --2- 172.21.15.116:0/4054515075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9100 0x55edd61e3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.266+0000 7f7fb5d4e640 1 --2- 172.21.15.116:0/4054515075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61e9af0 0x55edd61ea380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.266+0000 7f7fb6d50640 1 -- 172.21.15.116:0/4054515075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9100 msgr2=0x55edd61e3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:57.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.266+0000 7f7fb6d50640 1 --2- 172.21.15.116:0/4054515075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9100 0x55edd61e3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.266+0000 7f7fb6d50640 1 -- 172.21.15.116:0/4054515075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61e9af0 msgr2=0x55edd61ea380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:57.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.266+0000 7f7fb6d50640 1 --2- 172.21.15.116:0/4054515075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61e9af0 0x55edd61ea380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.266+0000 7f7fb6d50640 1 -- 172.21.15.116:0/4054515075 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55edd5f2e620 con 0x55edd61eec60 2024-03-19T01:16:57.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.267+0000 7f7fb6d50640 1 --2- 172.21.15.116:0/4054515075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61eec60 0x55edd61ea8c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009970 tx=0x7f7fa8004a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aef1aa8c3c23b11b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:57.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.267+0000 7f7f9bfff640 1 -- 172.21.15.116:0/4054515075 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8063070 con 0x55edd61eec60 2024-03-19T01:16:57.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.267+0000 7f7f9bfff640 1 -- 172.21.15.116:0/4054515075 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fa804cdf0 con 0x55edd61eec60 2024-03-19T01:16:57.269 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.267+0000 7f7f9bfff640 1 -- 172.21.15.116:0/4054515075 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8050040 con 0x55edd61eec60 2024-03-19T01:16:57.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.268+0000 7f7fbb39a740 1 -- 172.21.15.116:0/4054515075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61eec60 msgr2=0x55edd61ea8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:57.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.268+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/4054515075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61eec60 0x55edd61ea8c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8009970 tx=0x7f7fa8004a60 comp rx=0 tx=0).stop 2024-03-19T01:16:57.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.268+0000 7f7fbb39a740 1 -- 172.21.15.116:0/4054515075 shutdown_connections 2024-03-19T01:16:57.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.268+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/4054515075 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61eec60 0x55edd61ea8c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.268+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/4054515075 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61e9af0 0x55edd61ea380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.268+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/4054515075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61e9100 0x55edd61e3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.268+0000 7f7fbb39a740 1 -- 172.21.15.116:0/4054515075 >> 172.21.15.116:0/4054515075 conn(0x55edd61e4670 msgr2=0x55edd61e3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:57.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fbb39a740 1 -- 172.21.15.116:0/4054515075 shutdown_connections 2024-03-19T01:16:57.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fbb39a740 1 -- 172.21.15.116:0/4054515075 wait complete. 2024-03-19T01:16:57.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fbb39a740 1 -- start start 2024-03-19T01:16:57.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fbb39a740 1 Processor -- start 2024-03-19T01:16:57.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fbb39a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 0x55edd625f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fbb39a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61e9af0 0x55edd6261c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fbb39a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61eec60 0x55edd62641b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fbb39a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01ac590 con 0x55edd61e9af0 2024-03-19T01:16:57.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fbb39a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01abe00 con 0x55edd61eec60 2024-03-19T01:16:57.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fbb39a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb01a8fe0 con 0x55edd61e9100 2024-03-19T01:16:57.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fb5d4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61e9af0 0x55edd6261c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fb654f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 0x55edd625f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fb5d4e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61e9af0 0x55edd6261c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51936/0 (socket says 172.21.15.116:51936) 2024-03-19T01:16:57.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fb654f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 0x55edd625f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42894/0 (socket says 172.21.15.116:42894) 2024-03-19T01:16:57.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fb5d4e640 1 -- 172.21.15.116:0/549572155 learned_addr learned my addr 172.21.15.116:0/549572155 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:16:57.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fbb39a740 10 client.0 _ll_register_callbacks cb 0x7f7fa0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:16:57.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.269+0000 7f7fb6d50640 1 --2- 172.21.15.116:0/549572155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61eec60 0x55edd62641b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7fb654f640 1 -- 172.21.15.116:0/549572155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61eec60 msgr2=0x55edd62641b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7fb654f640 1 --2- 172.21.15.116:0/549572155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61eec60 0x55edd62641b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7fb654f640 1 -- 172.21.15.116:0/549572155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61e9af0 msgr2=0x55edd6261c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:16:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7fb654f640 1 --2- 172.21.15.116:0/549572155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61e9af0 0x55edd6261c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7fb654f640 1 -- 172.21.15.116:0/549572155 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55edd62648f0 con 0x55edd61e9100 2024-03-19T01:16:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7fb5d4e640 1 --2- 172.21.15.116:0/549572155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61e9af0 0x55edd6261c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:16:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7fb654f640 1 --2- 172.21.15.116:0/549572155 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 0x55edd625f710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0cc1b0 tx=0x7f7fac09b700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebe3256eebd4632f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:16:57.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7f99ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:16:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7f99ffb640 1 -- 172.21.15.116:0/549572155 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac009430 con 0x55edd61e9100 2024-03-19T01:16:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7f99ffb640 1 -- 172.21.15.116:0/549572155 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7fac0975e0 con 0x55edd61e9100 2024-03-19T01:16:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7fbb39a740 1 -- 172.21.15.116:0/549572155 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7fa804cdf0 con 0x55edd61e9100 2024-03-19T01:16:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7f99ffb640 1 -- 172.21.15.116:0/549572155 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) ==== 24+0+0 (secure 0 0 0) 0x7f7fac09b000 con 0x55edd61e9100 2024-03-19T01:16:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7fbb39a740 10 client.25486 fetch_fsmap learned FSMap version 38 2024-03-19T01:16:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.270+0000 7f7fbb39a740 1 -- 172.21.15.116:0/549572155 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) -- 0x55edd6265bc0 con 0x55edd61e9100 2024-03-19T01:16:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.271+0000 7f7f99ffb640 1 -- 172.21.15.116:0/549572155 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fa804cdf0 con 0x55edd61e9100 2024-03-19T01:16:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.271+0000 7f7f99ffb640 1 -- 172.21.15.116:0/549572155 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 38) ==== 8382+0+0 (secure 0 0 0) 0x7f7fac09a5c0 con 0x55edd61e9100 2024-03-19T01:16:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.271+0000 7f7fbb39a740 10 client.25486 fetch_fsmap finished waiting for FSMap version 38 2024-03-19T01:16:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.271+0000 7f7fbb39a740 10 client.25486 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:16:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.271+0000 7f7fbb39a740 20 client.25486 populate_metadata read hostname 'smithi116' 2024-03-19T01:16:57.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.271+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/549572155 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55edd6266190 0x55edd6268580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:16:57.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.271+0000 7f7fbb39a740 4 client.25486 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:16:57.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.271+0000 7f7fbb39a740 1 -- 172.21.15.116:0/549572155 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55edd6155470 con 0x55edd6266190 2024-03-19T01:16:57.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.271+0000 7f7fb5d4e640 1 --2- 172.21.15.116:0/549572155 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55edd6266190 0x55edd6268580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:16:57.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.272+0000 7f7fb5d4e640 1 --2- 172.21.15.116:0/549572155 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55edd6266190 0x55edd6268580 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8205a95dfb506016 server_cookie=69c6899babd00e31 in_seq=0 out_seq=0 2024-03-19T01:16:57.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.272+0000 7f7f99ffb640 10 client.25486 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:57.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7f99ffb640 1 -- 172.21.15.116:0/549572155 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55edd6155470 con 0x55edd6266190 2024-03-19T01:16:57.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7f99ffb640 10 client.25486 handle_command_reply: tid=0 2024-03-19T01:16:57.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7fbb39a740 1 client.25486 shutdown 2024-03-19T01:16:57.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7fbb39a740 1 -- 172.21.15.116:0/549572155 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 msgr2=0x55edd625f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:57.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/549572155 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 0x55edd625f710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7fac0cc1b0 tx=0x7f7fac09b700 comp rx=0 tx=0).stop 2024-03-19T01:16:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7fbb39a740 1 -- 172.21.15.116:0/549572155 shutdown_connections 2024-03-19T01:16:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/549572155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55edd61eec60 0x55edd62641b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/549572155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55edd61e9af0 0x55edd6261c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/549572155 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55edd6266190 0x55edd6268580 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7f99ffb640 0 client.25486 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:16:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7fbb39a740 1 --2- 172.21.15.116:0/549572155 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55edd61e9100 0x55edd625f710 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7fbb39a740 1 -- 172.21.15.116:0/549572155 >> 172.21.15.116:0/549572155 conn(0x55edd61e4670 msgr2=0x55edd61e78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7fbb39a740 1 -- 172.21.15.116:0/549572155 shutdown_connections 2024-03-19T01:16:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.279+0000 7f7fbb39a740 1 -- 172.21.15.116:0/549572155 wait complete. 2024-03-19T01:16:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.280+0000 7f7fbb39a740 20 client.25486 trim_cache size 0 max 16384 2024-03-19T01:16:57.282 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:16:57.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:16:57.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:16:57.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:16:57.283 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.280+0000 7f7fb87b3640 1 -- 172.21.15.116:0/2391398885 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7f9c0a2ab0 msgr2=0x7f7f9c0a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.280+0000 7f7fb87b3640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7f9c0a2ab0 0x7f7f9c0a4f70 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7fa802c640 tx=0x7f7fa8032590 comp rx=0 tx=0).stop 2024-03-19T01:16:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.280+0000 7f7fb87b3640 1 -- 172.21.15.116:0/2391398885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fb0101870 msgr2=0x7f7fb0199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:16:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.280+0000 7f7fb87b3640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fb0101870 0x7f7fb0199c80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00e9c0 tx=0x7f7fac00ee80 comp rx=0 tx=0).stop 2024-03-19T01:16:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.280+0000 7f7fb87b3640 1 -- 172.21.15.116:0/2391398885 shutdown_connections 2024-03-19T01:16:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.280+0000 7f7fb87b3640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7f9c0a2ab0 0x7f7f9c0a4f70 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.280+0000 7f7fb87b3640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7fb0103420 0x7f7fb01a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.280+0000 7f7fb87b3640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7fb0102a60 0x7f7fb019a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.280+0000 7f7fb87b3640 1 --2- 172.21.15.116:0/2391398885 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7fb0101870 0x7f7fb0199c80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:16:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.280+0000 7f7fb87b3640 1 -- 172.21.15.116:0/2391398885 >> 172.21.15.116:0/2391398885 conn(0x7f7fb00fcde0 msgr2=0x7f7fb0105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:16:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.280+0000 7f7fb87b3640 1 -- 172.21.15.116:0/2391398885 shutdown_connections 2024-03-19T01:16:57.284 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:16:57.280+0000 7f7fb87b3640 1 -- 172.21.15.116:0/2391398885 wait complete. 2024-03-19T01:16:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:57 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:16:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:57 smithi162 ceph-mon[67178]: pgmap v482: 97 pgs: 97 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 790 op/s 2024-03-19T01:16:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:57 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:16:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:57 smithi116 ceph-mon[62701]: pgmap v482: 97 pgs: 97 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 790 op/s 2024-03-19T01:16:57.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:57 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:16:57.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:57 smithi156 ceph-mon[66799]: pgmap v482: 97 pgs: 97 active+clean; 1.6 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 790 op/s 2024-03-19T01:16:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:16:59 smithi162 ceph-mon[67178]: pgmap v483: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.2 MiB/s wr, 768 op/s 2024-03-19T01:16:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:16:59 smithi116 ceph-mon[62701]: pgmap v483: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.2 MiB/s wr, 768 op/s 2024-03-19T01:16:59.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:16:59 smithi156 ceph-mon[66799]: pgmap v483: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.2 MiB/s wr, 768 op/s 2024-03-19T01:17:00.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:00 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:17:00.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:00 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:17:00.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:00 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:17:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 255.0 s, 2694.8 tps, lat 37.040 ms stddev 40.351 2024-03-19T01:17:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:17:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:17:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:17:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:01 smithi162 ceph-mon[67178]: pgmap v484: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.7 MiB/s wr, 740 op/s 2024-03-19T01:17:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:17:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:17:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:17:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:01 smithi116 ceph-mon[62701]: pgmap v484: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.7 MiB/s wr, 740 op/s 2024-03-19T01:17:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:17:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:17:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:17:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:01 smithi156 ceph-mon[66799]: pgmap v484: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.7 MiB/s wr, 740 op/s 2024-03-19T01:17:02.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:02 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:02.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:02 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:02.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:02 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:03.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:03 smithi162 ceph-mon[67178]: pgmap v485: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 750 op/s 2024-03-19T01:17:03.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:03 smithi116 ceph-mon[62701]: pgmap v485: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 750 op/s 2024-03-19T01:17:03.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:03 smithi156 ceph-mon[66799]: pgmap v485: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.0 MiB/s wr, 750 op/s 2024-03-19T01:17:04.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:04 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:04.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:04 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:04.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:04 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:05.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:05 smithi162 ceph-mon[67178]: pgmap v486: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.3 MiB/s wr, 862 op/s 2024-03-19T01:17:05.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:05 smithi116 ceph-mon[62701]: pgmap v486: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.3 MiB/s wr, 862 op/s 2024-03-19T01:17:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:05 smithi156 ceph-mon[66799]: pgmap v486: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 9.3 MiB/s wr, 862 op/s 2024-03-19T01:17:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 260.0 s, 3456.2 tps, lat 28.966 ms stddev 31.687 2024-03-19T01:17:07.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:07 smithi162 ceph-mon[67178]: pgmap v487: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.1 MiB/s wr, 711 op/s 2024-03-19T01:17:07.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:07 smithi116 ceph-mon[62701]: pgmap v487: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.1 MiB/s wr, 711 op/s 2024-03-19T01:17:07.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:07 smithi156 ceph-mon[66799]: pgmap v487: 97 pgs: 97 active+clean; 1.6 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.1 MiB/s wr, 711 op/s 2024-03-19T01:17:09.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:09 smithi162 ceph-mon[67178]: pgmap v488: 97 pgs: 97 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 720 op/s 2024-03-19T01:17:09.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:09 smithi116 ceph-mon[62701]: pgmap v488: 97 pgs: 97 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 720 op/s 2024-03-19T01:17:09.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:09 smithi156 ceph-mon[66799]: pgmap v488: 97 pgs: 97 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.0 MiB/s wr, 720 op/s 2024-03-19T01:17:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.413+0000 7f1d270a1640 1 -- 172.21.15.116:0/467712739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d20102a20 msgr2=0x7f1d20104e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.413+0000 7f1d270a1640 1 --2- 172.21.15.116:0/467712739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d20102a20 0x7f1d20104e90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c01a910 tx=0x7f1d1c03cc80 comp rx=0 tx=0).stop 2024-03-19T01:17:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.413+0000 7f1d270a1640 1 -- 172.21.15.116:0/467712739 shutdown_connections 2024-03-19T01:17:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.413+0000 7f1d270a1640 1 --2- 172.21.15.116:0/467712739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d201053d0 0x7f1d20105850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.413+0000 7f1d270a1640 1 --2- 172.21.15.116:0/467712739 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d20102a20 0x7f1d20104e90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.413+0000 7f1d270a1640 1 --2- 172.21.15.116:0/467712739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d201000b0 0x7f1d201024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.413+0000 7f1d270a1640 1 -- 172.21.15.116:0/467712739 >> 172.21.15.116:0/467712739 conn(0x7f1d200fbcf0 msgr2=0x7f1d200fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.413+0000 7f1d270a1640 1 -- 172.21.15.116:0/467712739 shutdown_connections 2024-03-19T01:17:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.413+0000 7f1d270a1640 1 -- 172.21.15.116:0/467712739 wait complete. 2024-03-19T01:17:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.414+0000 7f1d270a1640 1 Processor -- start 2024-03-19T01:17:10.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.414+0000 7f1d270a1640 1 -- start start 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.414+0000 7f1d270a1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d201000b0 0x7f1d201958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.414+0000 7f1d270a1640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d20102a20 0x7f1d20195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.414+0000 7f1d270a1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d201053d0 0x7f1d2019ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.414+0000 7f1d270a1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20109300 con 0x7f1d201053d0 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.414+0000 7f1d270a1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20109180 con 0x7f1d201000b0 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.414+0000 7f1d270a1640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20109480 con 0x7f1d20102a20 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.414+0000 7f1d24e3d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d201000b0 0x7f1d201958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.414+0000 7f1d24e3d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d201000b0 0x7f1d201958d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:59362/0 (socket says 172.21.15.116:59362) 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.414+0000 7f1d24e3d640 1 -- 172.21.15.116:0/1387825258 learned_addr learned my addr 172.21.15.116:0/1387825258 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.414+0000 7f1d17fff640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d20102a20 0x7f1d20195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.414+0000 7f1d2563e640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d201053d0 0x7f1d2019ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.415+0000 7f1d17fff640 1 -- 172.21.15.116:0/1387825258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d201000b0 msgr2=0x7f1d201958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.415+0000 7f1d17fff640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d201000b0 0x7f1d201958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.415+0000 7f1d17fff640 1 -- 172.21.15.116:0/1387825258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d201053d0 msgr2=0x7f1d2019ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:10.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.415+0000 7f1d17fff640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d201053d0 0x7f1d2019ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.415+0000 7f1d17fff640 1 -- 172.21.15.116:0/1387825258 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d2019d4f0 con 0x7f1d20102a20 2024-03-19T01:17:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.415+0000 7f1d2563e640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d201053d0 0x7f1d2019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:17:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.415+0000 7f1d17fff640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d20102a20 0x7f1d20195e10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c047040 tx=0x7f1d1c0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:10.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.416+0000 7f1d15ffb640 1 -- 172.21.15.116:0/1387825258 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c052020 con 0x7f1d20102a20 2024-03-19T01:17:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.416+0000 7f1d15ffb640 1 -- 172.21.15.116:0/1387825258 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d1c003da0 con 0x7f1d20102a20 2024-03-19T01:17:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.416+0000 7f1d270a1640 1 -- 172.21.15.116:0/1387825258 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d2019d780 con 0x7f1d20102a20 2024-03-19T01:17:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.416+0000 7f1d15ffb640 1 -- 172.21.15.116:0/1387825258 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c04e480 con 0x7f1d20102a20 2024-03-19T01:17:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.416+0000 7f1d270a1640 1 -- 172.21.15.116:0/1387825258 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d2019dce0 con 0x7f1d20102a20 2024-03-19T01:17:10.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.418+0000 7f1d15ffb640 1 -- 172.21.15.116:0/1387825258 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f1d1c055020 con 0x7f1d20102a20 2024-03-19T01:17:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.419+0000 7f1d15ffb640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1d080a29a0 0x7f1d080a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.419+0000 7f1d15ffb640 1 -- 172.21.15.116:0/1387825258 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1d1c101890 con 0x7f1d20102a20 2024-03-19T01:17:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.419+0000 7f1d29d69740 1 Processor -- start 2024-03-19T01:17:10.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.420+0000 7f1d29d69740 1 -- start start 2024-03-19T01:17:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.420+0000 7f1d29d69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb133100 0x5647cb133520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.420+0000 7f1d29d69740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133af0 0x5647cb1385f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.420+0000 7f1d29d69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb138c60 0x5647cb13b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.420+0000 7f1d29d69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5647cb097be0 con 0x5647cb133100 2024-03-19T01:17:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.420+0000 7f1d29d69740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5647cb040960 con 0x5647cb133af0 2024-03-19T01:17:10.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.420+0000 7f1d29d69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5647cb093c80 con 0x5647cb138c60 2024-03-19T01:17:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.421+0000 7f1d24e3d640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1d080a29a0 0x7f1d080a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.421+0000 7f1d2563e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb138c60 0x5647cb13b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.421+0000 7f1d2563e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb138c60 0x5647cb13b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36098/0 (socket says 172.21.15.116:36098) 2024-03-19T01:17:10.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.421+0000 7f1d2563e640 1 -- 172.21.15.116:0/2344891370 learned_addr learned my addr 172.21.15.116:0/2344891370 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.421+0000 7f1d24e3d640 1 --2- 172.21.15.116:0/2344891370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb133100 0x5647cb133520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.421+0000 7f1d2563e640 1 -- 172.21.15.116:0/2344891370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133af0 msgr2=0x5647cb1385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.421+0000 7f1d2563e640 1 --2- 172.21.15.116:0/2344891370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133af0 0x5647cb1385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.421+0000 7f1d2563e640 1 -- 172.21.15.116:0/2344891370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb133100 msgr2=0x5647cb133520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.421+0000 7f1d2563e640 1 --2- 172.21.15.116:0/2344891370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb133100 0x5647cb133520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.421+0000 7f1d2563e640 1 -- 172.21.15.116:0/2344891370 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647cb037d20 con 0x5647cb138c60 2024-03-19T01:17:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.421+0000 7f1d24e3d640 1 --2- 172.21.15.116:0/2344891370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb133100 0x5647cb133520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:17:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.422+0000 7f1d2563e640 1 --2- 172.21.15.116:0/2344891370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb138c60 0x5647cb13b0f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1d18007f30 tx=0x7f1d1800c750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e83dae3eb21ab83 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:10.423 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.422+0000 7f1d06ffd640 1 -- 172.21.15.116:0/2344891370 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d18014070 con 0x5647cb138c60 2024-03-19T01:17:10.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.422+0000 7f1d06ffd640 1 -- 172.21.15.116:0/2344891370 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d1800fd30 con 0x5647cb138c60 2024-03-19T01:17:10.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.422+0000 7f1d06ffd640 1 -- 172.21.15.116:0/2344891370 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1800ee20 con 0x5647cb138c60 2024-03-19T01:17:10.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.423+0000 7f1d29d69740 1 -- 172.21.15.116:0/2344891370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb138c60 msgr2=0x5647cb13b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:10.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.423+0000 7f1d24e3d640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1d080a29a0 0x7f1d080a4e60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x5647cb134350 tx=0x7f1d1000b890 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:10.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.423+0000 7f1d29d69740 1 --2- 172.21.15.116:0/2344891370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb138c60 0x5647cb13b0f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1d18007f30 tx=0x7f1d1800c750 comp rx=0 tx=0).stop 2024-03-19T01:17:10.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.423+0000 7f1d29d69740 1 -- 172.21.15.116:0/2344891370 shutdown_connections 2024-03-19T01:17:10.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.423+0000 7f1d29d69740 1 --2- 172.21.15.116:0/2344891370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb138c60 0x5647cb13b0f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.423+0000 7f1d29d69740 1 --2- 172.21.15.116:0/2344891370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133af0 0x5647cb1385f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.423+0000 7f1d29d69740 1 --2- 172.21.15.116:0/2344891370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb133100 0x5647cb133520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.423+0000 7f1d29d69740 1 -- 172.21.15.116:0/2344891370 >> 172.21.15.116:0/2344891370 conn(0x5647cb12e670 msgr2=0x5647cb12ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.423+0000 7f1d29d69740 1 -- 172.21.15.116:0/2344891370 shutdown_connections 2024-03-19T01:17:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.423+0000 7f1d29d69740 1 -- 172.21.15.116:0/2344891370 wait complete. 2024-03-19T01:17:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.423+0000 7f1d29d69740 1 -- start start 2024-03-19T01:17:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.423+0000 7f1d29d69740 1 Processor -- start 2024-03-19T01:17:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d29d69740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133100 0x5647cb14e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.428 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d29d69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb14e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d29d69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb138c60 0x5647cb155720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d29d69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5647cb14cc20 con 0x5647cb138c60 2024-03-19T01:17:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d29d69740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5647cb053db0 con 0x5647cb133100 2024-03-19T01:17:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d29d69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5647cb14cda0 con 0x5647cb133af0 2024-03-19T01:17:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d29d69740 10 client.0 _ll_register_callbacks cb 0x5647cb058e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:17:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d17fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb14e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d24e3d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133100 0x5647cb14e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d17fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb14e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36112/0 (socket says 172.21.15.116:36112) 2024-03-19T01:17:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d17fff640 1 -- 172.21.15.116:0/1304748664 learned_addr learned my addr 172.21.15.116:0/1304748664 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d24e3d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133100 0x5647cb14e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:59388/0 (socket says 172.21.15.116:59388) 2024-03-19T01:17:10.429 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d2563e640 1 --2- 172.21.15.116:0/1304748664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb138c60 0x5647cb155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d17fff640 1 -- 172.21.15.116:0/1304748664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133100 msgr2=0x5647cb14e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d17fff640 1 --2- 172.21.15.116:0/1304748664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133100 0x5647cb14e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d17fff640 1 -- 172.21.15.116:0/1304748664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb138c60 msgr2=0x5647cb155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d17fff640 1 --2- 172.21.15.116:0/1304748664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb138c60 0x5647cb155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.424+0000 7f1d17fff640 1 -- 172.21.15.116:0/1304748664 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647cb052c60 con 0x5647cb133af0 2024-03-19T01:17:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.425+0000 7f1d17fff640 1 --2- 172.21.15.116:0/1304748664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb14e6e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0ca610 tx=0x7f1d1c0cdbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfc4f1b2d69fdc5a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.425+0000 7f1cfffff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:17:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.425+0000 7f1cfffff640 1 -- 172.21.15.116:0/1304748664 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0d2300 con 0x5647cb133af0 2024-03-19T01:17:10.430 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.425+0000 7f1cfffff640 1 -- 172.21.15.116:0/1304748664 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1d1c0ff420 con 0x5647cb133af0 2024-03-19T01:17:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.425+0000 7f1d29d69740 1 -- 172.21.15.116:0/1304748664 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d1800fd30 con 0x5647cb133af0 2024-03-19T01:17:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.425+0000 7f1cfffff640 1 -- 172.21.15.116:0/1304748664 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1d1c0d6950 con 0x5647cb133af0 2024-03-19T01:17:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.425+0000 7f1d29d69740 10 client.25498 fetch_fsmap learned FSMap version 40 2024-03-19T01:17:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.425+0000 7f1d29d69740 1 -- 172.21.15.116:0/1304748664 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5647cadac210 con 0x5647cb133af0 2024-03-19T01:17:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.426+0000 7f1cfffff640 1 -- 172.21.15.116:0/1304748664 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d1800fd30 con 0x5647cb133af0 2024-03-19T01:17:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.426+0000 7f1cfffff640 1 -- 172.21.15.116:0/1304748664 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7f1d1c0cd600 con 0x5647cb133af0 2024-03-19T01:17:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.426+0000 7f1d29d69740 10 client.25498 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T01:17:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.426+0000 7f1d29d69740 10 client.25498 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:17:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.426+0000 7f1d29d69740 20 client.25498 populate_metadata read hostname 'smithi116' 2024-03-19T01:17:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.426+0000 7f1d29d69740 1 --2- 172.21.15.116:0/1304748664 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5647cb159f60 0x5647cb15c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.426+0000 7f1d29d69740 4 client.25498 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:17:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.426+0000 7f1d29d69740 1 -- 172.21.15.116:0/1304748664 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647cb0a5120 con 0x5647cb159f60 2024-03-19T01:17:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.426+0000 7f1d24e3d640 1 --2- 172.21.15.116:0/1304748664 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5647cb159f60 0x5647cb15c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.426+0000 7f1cfffff640 10 client.25498 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.426+0000 7f1d24e3d640 1 --2- 172.21.15.116:0/1304748664 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5647cb159f60 0x5647cb15c350 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d03939ec793229f0 server_cookie=3d9f7ed26e0122db in_seq=0 out_seq=0 2024-03-19T01:17:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.427+0000 7f1cfffff640 1 -- 172.21.15.116:0/1304748664 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5647cb0a5120 con 0x5647cb159f60 2024-03-19T01:17:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.427+0000 7f1cfffff640 10 client.25498 handle_command_reply: tid=0 2024-03-19T01:17:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.427+0000 7f1d29d69740 1 client.25498 shutdown 2024-03-19T01:17:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.427+0000 7f1d29d69740 1 -- 172.21.15.116:0/1304748664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 msgr2=0x5647cb14e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.427+0000 7f1d29d69740 1 --2- 172.21.15.116:0/1304748664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb14e6e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0ca610 tx=0x7f1d1c0cdbd0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.427+0000 7f1d29d69740 1 -- 172.21.15.116:0/1304748664 shutdown_connections 2024-03-19T01:17:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.427+0000 7f1d29d69740 1 --2- 172.21.15.116:0/1304748664 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb138c60 0x5647cb155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.427+0000 7f1d29d69740 1 --2- 172.21.15.116:0/1304748664 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5647cb159f60 0x5647cb15c350 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.427+0000 7f1d29d69740 1 --2- 172.21.15.116:0/1304748664 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb14e6e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.427+0000 7f1d29d69740 1 --2- 172.21.15.116:0/1304748664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133100 0x5647cb14e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.427+0000 7f1cfffff640 0 client.25498 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.427+0000 7f1d29d69740 1 -- 172.21.15.116:0/1304748664 >> 172.21.15.116:0/1304748664 conn(0x5647cb12e670 msgr2=0x5647cb12f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.428+0000 7f1d29d69740 1 -- 172.21.15.116:0/1304748664 shutdown_connections 2024-03-19T01:17:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.428+0000 7f1d29d69740 1 -- 172.21.15.116:0/1304748664 wait complete. 2024-03-19T01:17:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.428+0000 7f1d29d69740 20 client.25498 trim_cache size 0 max 16384 2024-03-19T01:17:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.447+0000 7f1d29d69740 1 Processor -- start 2024-03-19T01:17:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.447+0000 7f1d29d69740 1 -- start start 2024-03-19T01:17:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.447+0000 7f1d29d69740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133100 0x5647cb12d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.447+0000 7f1d29d69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb1556e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.447+0000 7f1d29d69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb138c60 0x5647cb134560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.447+0000 7f1d24e3d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133100 0x5647cb12d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.447+0000 7f1d24e3d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133100 0x5647cb12d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:59404/0 (socket says 172.21.15.116:59404) 2024-03-19T01:17:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.447+0000 7f1d17fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb1556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.447+0000 7f1d17fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb1556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36116/0 (socket says 172.21.15.116:36116) 2024-03-19T01:17:10.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.447+0000 7f1d29d69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2019e340 con 0x5647cb138c60 2024-03-19T01:17:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.447+0000 7f1d29d69740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2019ff70 con 0x5647cb133100 2024-03-19T01:17:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.447+0000 7f1d29d69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2019ea10 con 0x5647cb133af0 2024-03-19T01:17:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.447+0000 7f1d24e3d640 1 -- 172.21.15.116:0/191568279 learned_addr learned my addr 172.21.15.116:0/191568279 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.448+0000 7f1d2563e640 1 --2- 172.21.15.116:0/191568279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb138c60 0x5647cb134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.448+0000 7f1d24e3d640 1 -- 172.21.15.116:0/191568279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 msgr2=0x5647cb1556e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.448+0000 7f1d24e3d640 1 --2- 172.21.15.116:0/191568279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb1556e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.448+0000 7f1d24e3d640 1 -- 172.21.15.116:0/191568279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb138c60 msgr2=0x5647cb134560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.448+0000 7f1d24e3d640 1 --2- 172.21.15.116:0/191568279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb138c60 0x5647cb134560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.448+0000 7f1d24e3d640 1 -- 172.21.15.116:0/191568279 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647cae78620 con 0x5647cb133100 2024-03-19T01:17:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.448+0000 7f1d24e3d640 1 --2- 172.21.15.116:0/191568279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133100 0x5647cb12d930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x5647cb15cbb0 tx=0x7f1d10014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b4d96eb98a9c879 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.448+0000 7f1d067fc640 1 -- 172.21.15.116:0/191568279 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d10020340 con 0x5647cb133100 2024-03-19T01:17:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.448+0000 7f1d067fc640 1 -- 172.21.15.116:0/191568279 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d1002e070 con 0x5647cb133100 2024-03-19T01:17:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.448+0000 7f1d067fc640 1 -- 172.21.15.116:0/191568279 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1001f7c0 con 0x5647cb133100 2024-03-19T01:17:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.449+0000 7f1d29d69740 1 -- 172.21.15.116:0/191568279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133100 msgr2=0x5647cb12d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.449+0000 7f1d29d69740 1 --2- 172.21.15.116:0/191568279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133100 0x5647cb12d930 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x5647cb15cbb0 tx=0x7f1d10014040 comp rx=0 tx=0).stop 2024-03-19T01:17:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.449+0000 7f1d29d69740 1 -- 172.21.15.116:0/191568279 shutdown_connections 2024-03-19T01:17:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.449+0000 7f1d29d69740 1 --2- 172.21.15.116:0/191568279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb138c60 0x5647cb134560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.449+0000 7f1d29d69740 1 --2- 172.21.15.116:0/191568279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb1556e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.449+0000 7f1d29d69740 1 --2- 172.21.15.116:0/191568279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb133100 0x5647cb12d930 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.449+0000 7f1d29d69740 1 -- 172.21.15.116:0/191568279 >> 172.21.15.116:0/191568279 conn(0x5647cb12e670 msgr2=0x5647cb12d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.449+0000 7f1d29d69740 1 -- 172.21.15.116:0/191568279 shutdown_connections 2024-03-19T01:17:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.449+0000 7f1d29d69740 1 -- 172.21.15.116:0/191568279 wait complete. 2024-03-19T01:17:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.450+0000 7f1d29d69740 1 -- start start 2024-03-19T01:17:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.450+0000 7f1d29d69740 1 Processor -- start 2024-03-19T01:17:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.450+0000 7f1d29d69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb133100 0x5647cb152d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.450+0000 7f1d29d69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb153260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.450+0000 7f1d29d69740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb138c60 0x5647cb154270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.450+0000 7f1d29d69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2019f8a0 con 0x5647cb133100 2024-03-19T01:17:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.450+0000 7f1d29d69740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2019f110 con 0x5647cb138c60 2024-03-19T01:17:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.450+0000 7f1d29d69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1d20071e70 con 0x5647cb133af0 2024-03-19T01:17:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.450+0000 7f1d17fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.451+0000 7f1d17fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb153260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36130/0 (socket says 172.21.15.116:36130) 2024-03-19T01:17:10.454 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.451+0000 7f1d17fff640 1 -- 172.21.15.116:0/1379782053 learned_addr learned my addr 172.21.15.116:0/1379782053 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.451+0000 7f1d29d69740 10 client.0 _ll_register_callbacks cb 0x7f1d1c005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.451+0000 7f1d17fff640 1 -- 172.21.15.116:0/1379782053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb138c60 msgr2=0x5647cb154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.451+0000 7f1d17fff640 1 --2- 172.21.15.116:0/1379782053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb138c60 0x5647cb154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.451+0000 7f1d24e3d640 1 --2- 172.21.15.116:0/1379782053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb133100 0x5647cb152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.451+0000 7f1d17fff640 1 -- 172.21.15.116:0/1379782053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb133100 msgr2=0x5647cb152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.451+0000 7f1d17fff640 1 --2- 172.21.15.116:0/1379782053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb133100 0x5647cb152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.451+0000 7f1d17fff640 1 -- 172.21.15.116:0/1379782053 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647cb154860 con 0x5647cb133af0 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.451+0000 7f1d24e3d640 1 --2- 172.21.15.116:0/1379782053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb133100 0x5647cb152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.452+0000 7f1d17fff640 1 --2- 172.21.15.116:0/1379782053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb153260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0cde60 tx=0x7f1d1c03cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7b71a240d51682a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.452+0000 7f1cfffff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.452+0000 7f1cfffff640 1 -- 172.21.15.116:0/1379782053 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d1c0b1020 con 0x5647cb133af0 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.452+0000 7f1cfffff640 1 -- 172.21.15.116:0/1379782053 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1d1c0c9fd0 con 0x5647cb133af0 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.452+0000 7f1d29d69740 1 -- 172.21.15.116:0/1379782053 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d1002e070 con 0x5647cb133af0 2024-03-19T01:17:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.452+0000 7f1cfffff640 1 -- 172.21.15.116:0/1379782053 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f1d1c0d64c0 con 0x5647cb133af0 2024-03-19T01:17:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.453+0000 7f1d29d69740 10 client.25504 fetch_fsmap learned FSMap version 40 2024-03-19T01:17:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.453+0000 7f1d29d69740 1 -- 172.21.15.116:0/1379782053 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5647cb1c0270 con 0x5647cb133af0 2024-03-19T01:17:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.454+0000 7f1cfffff640 1 -- 172.21.15.116:0/1379782053 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d1c0e7060 con 0x5647cb133af0 2024-03-19T01:17:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.454+0000 7f1cfffff640 1 -- 172.21.15.116:0/1379782053 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 40) ==== 8374+0+0 (secure 0 0 0) 0x7f1d1c0cd140 con 0x5647cb133af0 2024-03-19T01:17:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.454+0000 7f1d29d69740 10 client.25504 fetch_fsmap finished waiting for FSMap version 40 2024-03-19T01:17:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.454+0000 7f1d29d69740 10 client.25504 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:17:10.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.454+0000 7f1d29d69740 20 client.25504 populate_metadata read hostname 'smithi116' 2024-03-19T01:17:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.454+0000 7f1d29d69740 1 --2- 172.21.15.116:0/1379782053 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5647cb1c0840 0x5647cb1c2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.454+0000 7f1d29d69740 4 client.25504 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:17:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.454+0000 7f1d29d69740 1 -- 172.21.15.116:0/1379782053 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f1d1002e070 con 0x5647cb1c0840 2024-03-19T01:17:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.455+0000 7f1d24e3d640 1 --2- 172.21.15.116:0/1379782053 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5647cb1c0840 0x5647cb1c2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.455+0000 7f1d24e3d640 1 --2- 172.21.15.116:0/1379782053 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5647cb1c0840 0x5647cb1c2ce0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af4c0d2496b83b0d server_cookie=cd29279ddd2eb09b in_seq=0 out_seq=0 2024-03-19T01:17:10.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.455+0000 7f1cfffff640 10 client.25504 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:10.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.465+0000 7f1cfffff640 1 -- 172.21.15.116:0/1379782053 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f1d1002e070 con 0x5647cb1c0840 2024-03-19T01:17:10.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.465+0000 7f1cfffff640 10 client.25504 handle_command_reply: tid=0 2024-03-19T01:17:10.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.465+0000 7f1d29d69740 1 client.25504 shutdown 2024-03-19T01:17:10.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.465+0000 7f1d29d69740 1 -- 172.21.15.116:0/1379782053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 msgr2=0x5647cb153260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:10.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.465+0000 7f1d29d69740 1 --2- 172.21.15.116:0/1379782053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb153260 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c0cde60 tx=0x7f1d1c03cc80 comp rx=0 tx=0).stop 2024-03-19T01:17:10.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.465+0000 7f1d29d69740 1 -- 172.21.15.116:0/1379782053 shutdown_connections 2024-03-19T01:17:10.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.465+0000 7f1d29d69740 1 --2- 172.21.15.116:0/1379782053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5647cb138c60 0x5647cb154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.465+0000 7f1d29d69740 1 --2- 172.21.15.116:0/1379782053 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5647cb1c0840 0x5647cb1c2ce0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.465+0000 7f1d29d69740 1 --2- 172.21.15.116:0/1379782053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5647cb133af0 0x5647cb153260 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.465+0000 7f1d29d69740 1 --2- 172.21.15.116:0/1379782053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5647cb133100 0x5647cb152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.465+0000 7f1cfffff640 0 client.25504 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:10.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.465+0000 7f1d29d69740 1 -- 172.21.15.116:0/1379782053 >> 172.21.15.116:0/1379782053 conn(0x5647cb12e670 msgr2=0x5647cb1318d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.466+0000 7f1d29d69740 1 -- 172.21.15.116:0/1379782053 shutdown_connections 2024-03-19T01:17:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.466+0000 7f1d29d69740 1 -- 172.21.15.116:0/1379782053 wait complete. 2024-03-19T01:17:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.466+0000 7f1d29d69740 20 client.25504 trim_cache size 0 max 16384 2024-03-19T01:17:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:17:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:17:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:17:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:17:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.466+0000 7f1d270a1640 1 -- 172.21.15.116:0/1387825258 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1d080a29a0 msgr2=0x7f1d080a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.466+0000 7f1d270a1640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1d080a29a0 0x7f1d080a4e60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x5647cb134350 tx=0x7f1d1000b890 comp rx=0 tx=0).stop 2024-03-19T01:17:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.466+0000 7f1d270a1640 1 -- 172.21.15.116:0/1387825258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d20102a20 msgr2=0x7f1d20195e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.466+0000 7f1d270a1640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d20102a20 0x7f1d20195e10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1d1c047040 tx=0x7f1d1c0045e0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.466+0000 7f1d270a1640 1 -- 172.21.15.116:0/1387825258 shutdown_connections 2024-03-19T01:17:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.466+0000 7f1d270a1640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d201053d0 0x7f1d2019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.466+0000 7f1d270a1640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1d20102a20 0x7f1d20195e10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.469 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.466+0000 7f1d270a1640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1d080a29a0 0x7f1d080a4e60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.466+0000 7f1d270a1640 1 --2- 172.21.15.116:0/1387825258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d201000b0 0x7f1d201958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.466+0000 7f1d270a1640 1 -- 172.21.15.116:0/1387825258 >> 172.21.15.116:0/1387825258 conn(0x7f1d200fbcf0 msgr2=0x7f1d200fc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.467+0000 7f1d270a1640 1 -- 172.21.15.116:0/1387825258 shutdown_connections 2024-03-19T01:17:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:10.467+0000 7f1d270a1640 1 -- 172.21.15.116:0/1387825258 wait complete. 2024-03-19T01:17:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 265.0 s, 2283.0 tps, lat 43.747 ms stddev 45.690 2024-03-19T01:17:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:11 smithi162 ceph-mon[67178]: pgmap v489: 97 pgs: 97 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.2 MiB/s wr, 726 op/s 2024-03-19T01:17:11.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:11 smithi116 ceph-mon[62701]: pgmap v489: 97 pgs: 97 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.2 MiB/s wr, 726 op/s 2024-03-19T01:17:11.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:11 smithi156 ceph-mon[66799]: pgmap v489: 97 pgs: 97 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 7.2 MiB/s wr, 726 op/s 2024-03-19T01:17:13.525 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:13 smithi156 ceph-mon[66799]: pgmap v490: 97 pgs: 97 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 7.3 MiB/s wr, 712 op/s 2024-03-19T01:17:13.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:13 smithi162 ceph-mon[67178]: pgmap v490: 97 pgs: 97 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 7.3 MiB/s wr, 712 op/s 2024-03-19T01:17:13.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:13 smithi116 ceph-mon[62701]: pgmap v490: 97 pgs: 97 active+clean; 1.6 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 7.3 MiB/s wr, 712 op/s 2024-03-19T01:17:15.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:15 smithi162 ceph-mon[67178]: pgmap v491: 97 pgs: 97 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 774 op/s 2024-03-19T01:17:15.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:15 smithi116 ceph-mon[62701]: pgmap v491: 97 pgs: 97 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 774 op/s 2024-03-19T01:17:15.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:15 smithi156 ceph-mon[66799]: pgmap v491: 97 pgs: 97 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 774 op/s 2024-03-19T01:17:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 270.0 s, 2754.4 tps, lat 36.245 ms stddev 41.927 2024-03-19T01:17:17.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:17 smithi162 ceph-mon[67178]: pgmap v492: 97 pgs: 97 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 MiB/s wr, 697 op/s 2024-03-19T01:17:17.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:17 smithi116 ceph-mon[62701]: pgmap v492: 97 pgs: 97 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 MiB/s wr, 697 op/s 2024-03-19T01:17:17.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:17 smithi156 ceph-mon[66799]: pgmap v492: 97 pgs: 97 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.0 MiB/s wr, 697 op/s 2024-03-19T01:17:19.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:19 smithi162 ceph-mon[67178]: pgmap v493: 97 pgs: 97 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 740 op/s 2024-03-19T01:17:19.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:19 smithi116 ceph-mon[62701]: pgmap v493: 97 pgs: 97 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 740 op/s 2024-03-19T01:17:19.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:19 smithi156 ceph-mon[66799]: pgmap v493: 97 pgs: 97 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 740 op/s 2024-03-19T01:17:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 275.0 s, 2635.6 tps, lat 37.851 ms stddev 42.108 2024-03-19T01:17:22.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:21 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:22.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:21 smithi156 ceph-mon[66799]: pgmap v494: 97 pgs: 97 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 689 op/s 2024-03-19T01:17:22.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:21 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:22.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:21 smithi162 ceph-mon[67178]: pgmap v494: 97 pgs: 97 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 689 op/s 2024-03-19T01:17:22.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:21 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:22.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:21 smithi116 ceph-mon[62701]: pgmap v494: 97 pgs: 97 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.8 MiB/s wr, 689 op/s 2024-03-19T01:17:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.586+0000 7f3454a9a640 1 -- 172.21.15.116:0/390345100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3450103a70 msgr2=0x7f3450103e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.586+0000 7f3454a9a640 1 --2- 172.21.15.116:0/390345100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3450103a70 0x7f3450103e90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3448009ca0 tx=0x7f344802c0f0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.586+0000 7f3454a9a640 1 -- 172.21.15.116:0/390345100 shutdown_connections 2024-03-19T01:17:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.586+0000 7f3454a9a640 1 --2- 172.21.15.116:0/390345100 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3450105620 0x7f345010beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.586+0000 7f3454a9a640 1 --2- 172.21.15.116:0/390345100 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3450104c60 0x7f34501050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.586+0000 7f3454a9a640 1 --2- 172.21.15.116:0/390345100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3450103a70 0x7f3450103e90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.586+0000 7f3454a9a640 1 -- 172.21.15.116:0/390345100 >> 172.21.15.116:0/390345100 conn(0x7f34500fefe0 msgr2=0x7f3450101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.586+0000 7f3454a9a640 1 -- 172.21.15.116:0/390345100 shutdown_connections 2024-03-19T01:17:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.586+0000 7f3454a9a640 1 -- 172.21.15.116:0/390345100 wait complete. 2024-03-19T01:17:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.587+0000 7f3454a9a640 1 Processor -- start 2024-03-19T01:17:23.590 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.587+0000 7f3454a9a640 1 -- start start 2024-03-19T01:17:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.587+0000 7f3454a9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3450103a70 0x7f3450199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.587+0000 7f3454a9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3450104c60 0x7f345019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.587+0000 7f3454a9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3450105620 0x7f34501a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.587+0000 7f3454a9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f345010f6f0 con 0x7f3450105620 2024-03-19T01:17:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.587+0000 7f3454a9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f345010f570 con 0x7f3450103a70 2024-03-19T01:17:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.587+0000 7f3454a9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f345010f870 con 0x7f3450104c60 2024-03-19T01:17:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.587+0000 7f344e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3450103a70 0x7f3450199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.587+0000 7f344e59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3450103a70 0x7f3450199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:49500/0 (socket says 172.21.15.116:49500) 2024-03-19T01:17:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.587+0000 7f344e59c640 1 -- 172.21.15.116:0/2545803493 learned_addr learned my addr 172.21.15.116:0/2545803493 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.587+0000 7f344dd9b640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3450104c60 0x7f345019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.587+0000 7f344ed9d640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3450105620 0x7f34501a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.588+0000 7f344e59c640 1 -- 172.21.15.116:0/2545803493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3450104c60 msgr2=0x7f345019a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:23.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.588+0000 7f344e59c640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3450104c60 0x7f345019a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.588+0000 7f344e59c640 1 -- 172.21.15.116:0/2545803493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3450105620 msgr2=0x7f34501a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.588+0000 7f344e59c640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3450105620 0x7f34501a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.588+0000 7f344e59c640 1 -- 172.21.15.116:0/2545803493 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34501a1910 con 0x7f3450103a70 2024-03-19T01:17:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.588+0000 7f344dd9b640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3450104c60 0x7f345019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:17:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.588+0000 7f344e59c640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3450103a70 0x7f3450199d00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f34480324c0 tx=0x7f3448032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.588+0000 7f343f7fe640 1 -- 172.21.15.116:0/2545803493 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3448041020 con 0x7f3450103a70 2024-03-19T01:17:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.588+0000 7f3454a9a640 1 -- 172.21.15.116:0/2545803493 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3450075c60 con 0x7f3450103a70 2024-03-19T01:17:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.588+0000 7f3454a9a640 1 -- 172.21.15.116:0/2545803493 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34500761a0 con 0x7f3450103a70 2024-03-19T01:17:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.588+0000 7f343f7fe640 1 -- 172.21.15.116:0/2545803493 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34480348f0 con 0x7f3450103a70 2024-03-19T01:17:23.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.588+0000 7f343f7fe640 1 -- 172.21.15.116:0/2545803493 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3448039040 con 0x7f3450103a70 2024-03-19T01:17:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.591+0000 7f3457763740 1 Processor -- start 2024-03-19T01:17:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.592+0000 7f343f7fe640 1 -- 172.21.15.116:0/2545803493 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f3448044020 con 0x7f3450103a70 2024-03-19T01:17:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.592+0000 7f3457763740 1 -- start start 2024-03-19T01:17:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.592+0000 7f3457763740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e47520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.592+0000 7f3457763740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e47af0 0x5602f7e4c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.592+0000 7f3457763740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e4cc60 0x5602f7e4f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.592+0000 7f3457763740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5602f7dabbe0 con 0x5602f7e47af0 2024-03-19T01:17:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.592+0000 7f3457763740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5602f7d54960 con 0x5602f7e47100 2024-03-19T01:17:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.592+0000 7f3457763740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5602f7da7c80 con 0x5602f7e4cc60 2024-03-19T01:17:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.592+0000 7f344ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e47520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.592+0000 7f344e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e47af0 0x5602f7e4c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.592+0000 7f344ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e47520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:49512/0 (socket says 172.21.15.116:49512) 2024-03-19T01:17:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.592+0000 7f344ed9d640 1 -- 172.21.15.116:0/4226152042 learned_addr learned my addr 172.21.15.116:0/4226152042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f344dd9b640 1 --2- 172.21.15.116:0/4226152042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e4cc60 0x5602f7e4f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f344ed9d640 1 -- 172.21.15.116:0/4226152042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e4cc60 msgr2=0x5602f7e4f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f344ed9d640 1 --2- 172.21.15.116:0/4226152042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e4cc60 0x5602f7e4f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f344ed9d640 1 -- 172.21.15.116:0/4226152042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e47af0 msgr2=0x5602f7e4c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f344ed9d640 1 --2- 172.21.15.116:0/4226152042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e47af0 0x5602f7e4c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f344ed9d640 1 -- 172.21.15.116:0/4226152042 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602f7d4bd20 con 0x5602f7e47100 2024-03-19T01:17:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f344dd9b640 1 --2- 172.21.15.116:0/4226152042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e4cc60 0x5602f7e4f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:17:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f344ed9d640 1 --2- 172.21.15.116:0/4226152042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e47520 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3444009660 tx=0x7f3444009b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed5798d1ee89684b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f343f7fe640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f34380a28a0 0x7f34380a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f343cff9640 1 -- 172.21.15.116:0/4226152042 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444024070 con 0x5602f7e47100 2024-03-19T01:17:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f343cff9640 1 -- 172.21.15.116:0/4226152042 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f344400bd80 con 0x5602f7e47100 2024-03-19T01:17:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f343cff9640 1 -- 172.21.15.116:0/4226152042 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344401e3a0 con 0x5602f7e47100 2024-03-19T01:17:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f344e59c640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f34380a28a0 0x7f34380a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.593+0000 7f343f7fe640 1 -- 172.21.15.116:0/2545803493 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f34480f0420 con 0x7f3450103a70 2024-03-19T01:17:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.594+0000 7f3457763740 1 -- 172.21.15.116:0/4226152042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 msgr2=0x5602f7e47520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.594+0000 7f3457763740 1 --2- 172.21.15.116:0/4226152042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e47520 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3444009660 tx=0x7f3444009b20 comp rx=0 tx=0).stop 2024-03-19T01:17:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.594+0000 7f344e59c640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f34380a28a0 0x7f34380a4d60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x5602f7e47ef0 tx=0x7f34480f3e60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.594+0000 7f3457763740 1 -- 172.21.15.116:0/4226152042 shutdown_connections 2024-03-19T01:17:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.594+0000 7f3457763740 1 --2- 172.21.15.116:0/4226152042 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e4cc60 0x5602f7e4f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.594+0000 7f3457763740 1 --2- 172.21.15.116:0/4226152042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e47af0 0x5602f7e4c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.594+0000 7f3457763740 1 --2- 172.21.15.116:0/4226152042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e47520 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.594+0000 7f3457763740 1 -- 172.21.15.116:0/4226152042 >> 172.21.15.116:0/4226152042 conn(0x5602f7e42670 msgr2=0x5602f7e42a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.594+0000 7f3457763740 1 -- 172.21.15.116:0/4226152042 shutdown_connections 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.594+0000 7f3457763740 1 -- 172.21.15.116:0/4226152042 wait complete. 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f3457763740 1 -- start start 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f3457763740 1 Processor -- start 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f3457763740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e51140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f3457763740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47af0 0x5602f7e51680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f3457763740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e4cc60 0x5602f7e586c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f3457763740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5602f7e4fc10 con 0x5602f7e4cc60 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f3457763740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5602f7d67db0 con 0x5602f7e47100 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f3457763740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5602f7e4fd90 con 0x5602f7e47af0 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f344ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e51140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f344ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e51140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:49524/0 (socket says 172.21.15.116:49524) 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f344ed9d640 1 -- 172.21.15.116:0/3654665070 learned_addr learned my addr 172.21.15.116:0/3654665070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f344dd9b640 1 --2- 172.21.15.116:0/3654665070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47af0 0x5602f7e51680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f344ed9d640 1 --2- 172.21.15.116:0/3654665070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e4cc60 0x5602f7e586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f3457763740 10 client.0 _ll_register_callbacks cb 0x5602f7d6ce70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:17:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f344dd9b640 1 -- 172.21.15.116:0/3654665070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 msgr2=0x5602f7e51140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f344dd9b640 1 --2- 172.21.15.116:0/3654665070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e51140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f344dd9b640 1 -- 172.21.15.116:0/3654665070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e4cc60 msgr2=0x5602f7e586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.595+0000 7f344dd9b640 1 --2- 172.21.15.116:0/3654665070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e4cc60 0x5602f7e586c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.596+0000 7f344dd9b640 1 -- 172.21.15.116:0/3654665070 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602f7d66c60 con 0x5602f7e47af0 2024-03-19T01:17:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.596+0000 7f344dd9b640 1 --2- 172.21.15.116:0/3654665070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47af0 0x5602f7e51680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f34400129e0 tx=0x7f3440012ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c4ba301c24837ca server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.596+0000 7f3435ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:17:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.596+0000 7f3435ffb640 1 -- 172.21.15.116:0/3654665070 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3440004030 con 0x5602f7e47af0 2024-03-19T01:17:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.596+0000 7f3457763740 1 -- 172.21.15.116:0/3654665070 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f344400bd80 con 0x5602f7e47af0 2024-03-19T01:17:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.596+0000 7f3435ffb640 1 -- 172.21.15.116:0/3654665070 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f34400041d0 con 0x5602f7e47af0 2024-03-19T01:17:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.596+0000 7f3435ffb640 1 -- 172.21.15.116:0/3654665070 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f3440021070 con 0x5602f7e47af0 2024-03-19T01:17:23.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.596+0000 7f3457763740 10 client.25516 fetch_fsmap learned FSMap version 41 2024-03-19T01:17:23.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.596+0000 7f3457763740 1 -- 172.21.15.116:0/3654665070 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5602f7ac0210 con 0x5602f7e47af0 2024-03-19T01:17:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.596+0000 7f3435ffb640 1 -- 172.21.15.116:0/3654665070 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f344400bd80 con 0x5602f7e47af0 2024-03-19T01:17:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.597+0000 7f3435ffb640 1 -- 172.21.15.116:0/3654665070 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f3440003a60 con 0x5602f7e47af0 2024-03-19T01:17:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.597+0000 7f3457763740 10 client.25516 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T01:17:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.597+0000 7f3457763740 10 client.25516 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:17:23.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.597+0000 7f3457763740 20 client.25516 populate_metadata read hostname 'smithi116' 2024-03-19T01:17:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.597+0000 7f3457763740 1 --2- 172.21.15.116:0/3654665070 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5602f7e5d020 0x5602f7e5f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.597+0000 7f3457763740 4 client.25516 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:17:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.597+0000 7f3457763740 1 -- 172.21.15.116:0/3654665070 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602f7db9120 con 0x5602f7e5d020 2024-03-19T01:17:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.597+0000 7f344ed9d640 1 --2- 172.21.15.116:0/3654665070 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5602f7e5d020 0x5602f7e5f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.598+0000 7f3435ffb640 10 client.25516 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.598+0000 7f344ed9d640 1 --2- 172.21.15.116:0/3654665070 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5602f7e5d020 0x5602f7e5f4c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dc4d784b530702b server_cookie=9f0e60c355ae71b5 in_seq=0 out_seq=0 2024-03-19T01:17:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3435ffb640 1 -- 172.21.15.116:0/3654665070 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5602f7db9120 con 0x5602f7e5d020 2024-03-19T01:17:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3435ffb640 10 client.25516 handle_command_reply: tid=0 2024-03-19T01:17:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3457763740 1 client.25516 shutdown 2024-03-19T01:17:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3457763740 1 -- 172.21.15.116:0/3654665070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47af0 msgr2=0x5602f7e51680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3457763740 1 --2- 172.21.15.116:0/3654665070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47af0 0x5602f7e51680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f34400129e0 tx=0x7f3440012ea0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3457763740 1 -- 172.21.15.116:0/3654665070 shutdown_connections 2024-03-19T01:17:23.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3457763740 1 --2- 172.21.15.116:0/3654665070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e4cc60 0x5602f7e586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3457763740 1 --2- 172.21.15.116:0/3654665070 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5602f7e5d020 0x5602f7e5f4c0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3435ffb640 0 client.25516 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3457763740 1 --2- 172.21.15.116:0/3654665070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47af0 0x5602f7e51680 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3457763740 1 --2- 172.21.15.116:0/3654665070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e51140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3457763740 1 -- 172.21.15.116:0/3654665070 >> 172.21.15.116:0/3654665070 conn(0x5602f7e42670 msgr2=0x5602f7e438c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3457763740 1 -- 172.21.15.116:0/3654665070 shutdown_connections 2024-03-19T01:17:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3457763740 1 -- 172.21.15.116:0/3654665070 wait complete. 2024-03-19T01:17:23.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.599+0000 7f3457763740 20 client.25516 trim_cache size 0 max 16384 2024-03-19T01:17:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.621+0000 7f3457763740 1 Processor -- start 2024-03-19T01:17:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.621+0000 7f3457763740 1 -- start start 2024-03-19T01:17:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.621+0000 7f3457763740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e50b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.621+0000 7f3457763740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47af0 0x5602f7e58680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.621+0000 7f3457763740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e4cc60 0x5602f7e41220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.621+0000 7f3457763740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f343000e530 con 0x5602f7e4cc60 2024-03-19T01:17:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.621+0000 7f3457763740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3430010130 con 0x5602f7e47100 2024-03-19T01:17:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.622+0000 7f344dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47af0 0x5602f7e58680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.622+0000 7f344dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47af0 0x5602f7e58680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52370/0 (socket says 172.21.15.116:52370) 2024-03-19T01:17:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.622+0000 7f344dd9b640 1 -- 172.21.15.116:0/3002619871 learned_addr learned my addr 172.21.15.116:0/3002619871 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.622+0000 7f3457763740 1 -- 172.21.15.116:0/3002619871 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f343000ec00 con 0x5602f7e47af0 2024-03-19T01:17:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.622+0000 7f344ed9d640 1 --2- 172.21.15.116:0/3002619871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e50b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.623 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.622+0000 7f344ed9d640 1 --2- 172.21.15.116:0/3002619871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e4cc60 0x5602f7e41220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.622+0000 7f344dd9b640 1 -- 172.21.15.116:0/3002619871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 msgr2=0x5602f7e50b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.622+0000 7f344dd9b640 1 --2- 172.21.15.116:0/3002619871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e50b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.622+0000 7f344dd9b640 1 -- 172.21.15.116:0/3002619871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e4cc60 msgr2=0x5602f7e41220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.622+0000 7f344dd9b640 1 --2- 172.21.15.116:0/3002619871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e4cc60 0x5602f7e41220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.622+0000 7f344dd9b640 1 -- 172.21.15.116:0/3002619871 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602f7b8c620 con 0x5602f7e47af0 2024-03-19T01:17:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.622+0000 7f344ed9d640 1 --2- 172.21.15.116:0/3002619871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e4cc60 0x5602f7e41220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:17:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.622+0000 7f344dd9b640 1 --2- 172.21.15.116:0/3002619871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47af0 0x5602f7e58680 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3440000c00 tx=0x7f3440010780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13a4ff731c6a060b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:23.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.623+0000 7f3437fff640 1 -- 172.21.15.116:0/3002619871 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3440010c20 con 0x5602f7e47af0 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.623+0000 7f3437fff640 1 -- 172.21.15.116:0/3002619871 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f344000fe90 con 0x5602f7e47af0 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.623+0000 7f3437fff640 1 -- 172.21.15.116:0/3002619871 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34400035f0 con 0x5602f7e47af0 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.623+0000 7f3457763740 1 -- 172.21.15.116:0/3002619871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47af0 msgr2=0x5602f7e58680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.623+0000 7f3457763740 1 --2- 172.21.15.116:0/3002619871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47af0 0x5602f7e58680 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3440000c00 tx=0x7f3440010780 comp rx=0 tx=0).stop 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.623+0000 7f3457763740 1 -- 172.21.15.116:0/3002619871 shutdown_connections 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.623+0000 7f3457763740 1 --2- 172.21.15.116:0/3002619871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e4cc60 0x5602f7e41220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.623+0000 7f3457763740 1 --2- 172.21.15.116:0/3002619871 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47af0 0x5602f7e58680 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.623+0000 7f3457763740 1 --2- 172.21.15.116:0/3002619871 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e47100 0x5602f7e50b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.623+0000 7f3457763740 1 -- 172.21.15.116:0/3002619871 >> 172.21.15.116:0/3002619871 conn(0x5602f7e42670 msgr2=0x5602f7e50430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.624+0000 7f3457763740 1 -- 172.21.15.116:0/3002619871 shutdown_connections 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.624+0000 7f3457763740 1 -- 172.21.15.116:0/3002619871 wait complete. 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.624+0000 7f3457763740 1 -- start start 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.624+0000 7f3457763740 1 Processor -- start 2024-03-19T01:17:23.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.624+0000 7f3457763740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47100 0x5602f7e55650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.624+0000 7f3457763740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e47af0 0x5602f7e55b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.624+0000 7f3457763740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e4cc60 0x5602f7ec13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.624+0000 7f344ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47100 0x5602f7e55650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.625+0000 7f344ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47100 0x5602f7e55650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52376/0 (socket says 172.21.15.116:52376) 2024-03-19T01:17:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.625+0000 7f344ed9d640 1 -- 172.21.15.116:0/1598813532 learned_addr learned my addr 172.21.15.116:0/1598813532 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.625+0000 7f344dd9b640 1 --2- 172.21.15.116:0/1598813532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e47af0 0x5602f7e55b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.625+0000 7f3457763740 1 -- 172.21.15.116:0/1598813532 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f343000fa60 con 0x5602f7e47af0 2024-03-19T01:17:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.625+0000 7f3457763740 1 -- 172.21.15.116:0/1598813532 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f343000f2d0 con 0x5602f7e4cc60 2024-03-19T01:17:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.625+0000 7f3457763740 1 -- 172.21.15.116:0/1598813532 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f343000c510 con 0x5602f7e47100 2024-03-19T01:17:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.625+0000 7f3457763740 10 client.0 _ll_register_callbacks cb 0x7f3440009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:17:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.625+0000 7f344ed9d640 1 --2- 172.21.15.116:0/1598813532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e4cc60 0x5602f7ec13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.625+0000 7f344ed9d640 1 -- 172.21.15.116:0/1598813532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e4cc60 msgr2=0x5602f7ec13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:23.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.625+0000 7f344ed9d640 1 --2- 172.21.15.116:0/1598813532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e4cc60 0x5602f7ec13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.625+0000 7f344ed9d640 1 -- 172.21.15.116:0/1598813532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e47af0 msgr2=0x5602f7e55b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.625+0000 7f344ed9d640 1 --2- 172.21.15.116:0/1598813532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e47af0 0x5602f7e55b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.625+0000 7f344ed9d640 1 -- 172.21.15.116:0/1598813532 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602f7ec19e0 con 0x5602f7e47100 2024-03-19T01:17:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.626+0000 7f344ed9d640 1 --2- 172.21.15.116:0/1598813532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47100 0x5602f7e55650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3444005890 tx=0x7f3444005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36959ae8d584bda4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:23.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.626+0000 7f3435ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:17:23.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.626+0000 7f3435ffb640 1 -- 172.21.15.116:0/1598813532 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444024040 con 0x5602f7e47100 2024-03-19T01:17:23.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.626+0000 7f3435ffb640 1 -- 172.21.15.116:0/1598813532 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f344400b8a0 con 0x5602f7e47100 2024-03-19T01:17:23.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.626+0000 7f3457763740 1 -- 172.21.15.116:0/1598813532 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f344000fe90 con 0x5602f7e47100 2024-03-19T01:17:23.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.626+0000 7f3435ffb640 1 -- 172.21.15.116:0/1598813532 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f344400baa0 con 0x5602f7e47100 2024-03-19T01:17:23.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.627+0000 7f3457763740 10 client.25522 fetch_fsmap learned FSMap version 41 2024-03-19T01:17:23.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.627+0000 7f3457763740 1 -- 172.21.15.116:0/1598813532 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x5602f7ec2cd0 con 0x5602f7e47100 2024-03-19T01:17:23.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.627+0000 7f3435ffb640 1 -- 172.21.15.116:0/1598813532 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f344000fe90 con 0x5602f7e47100 2024-03-19T01:17:23.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.627+0000 7f3435ffb640 1 -- 172.21.15.116:0/1598813532 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 41) ==== 8370+0+0 (secure 0 0 0) 0x7f344403a020 con 0x5602f7e47100 2024-03-19T01:17:23.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.628+0000 7f3457763740 10 client.25522 fetch_fsmap finished waiting for FSMap version 41 2024-03-19T01:17:23.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.628+0000 7f3457763740 10 client.25522 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:17:23.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.628+0000 7f3457763740 20 client.25522 populate_metadata read hostname 'smithi116' 2024-03-19T01:17:23.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.628+0000 7f3457763740 1 --2- 172.21.15.116:0/1598813532 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5602f7ec3490 0x5602f7ec58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:23.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.628+0000 7f3457763740 4 client.25522 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:17:23.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.628+0000 7f3457763740 1 -- 172.21.15.116:0/1598813532 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5602f7db3470 con 0x5602f7ec3490 2024-03-19T01:17:23.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.628+0000 7f344dd9b640 1 --2- 172.21.15.116:0/1598813532 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5602f7ec3490 0x5602f7ec58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:23.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.629+0000 7f344dd9b640 1 --2- 172.21.15.116:0/1598813532 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5602f7ec3490 0x5602f7ec58b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d72126b226717e7 server_cookie=c2a5c29616805982 in_seq=0 out_seq=0 2024-03-19T01:17:23.631 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.629+0000 7f3435ffb640 10 client.25522 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:23.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.637+0000 7f3435ffb640 1 -- 172.21.15.116:0/1598813532 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5602f7db3470 con 0x5602f7ec3490 2024-03-19T01:17:23.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.637+0000 7f3435ffb640 10 client.25522 handle_command_reply: tid=0 2024-03-19T01:17:23.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.637+0000 7f3457763740 1 client.25522 shutdown 2024-03-19T01:17:23.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.637+0000 7f3457763740 1 -- 172.21.15.116:0/1598813532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47100 msgr2=0x5602f7e55650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:23.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.637+0000 7f3457763740 1 --2- 172.21.15.116:0/1598813532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47100 0x5602f7e55650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3444005890 tx=0x7f3444005f90 comp rx=0 tx=0).stop 2024-03-19T01:17:23.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.639+0000 7f3457763740 1 -- 172.21.15.116:0/1598813532 shutdown_connections 2024-03-19T01:17:23.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.639+0000 7f3457763740 1 --2- 172.21.15.116:0/1598813532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5602f7e4cc60 0x5602f7ec13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.639+0000 7f3457763740 1 --2- 172.21.15.116:0/1598813532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5602f7e47af0 0x5602f7e55b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.639+0000 7f3457763740 1 --2- 172.21.15.116:0/1598813532 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5602f7ec3490 0x5602f7ec58b0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.639+0000 7f3457763740 1 --2- 172.21.15.116:0/1598813532 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5602f7e47100 0x5602f7e55650 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.639+0000 7f3457763740 1 -- 172.21.15.116:0/1598813532 >> 172.21.15.116:0/1598813532 conn(0x5602f7e42670 msgr2=0x5602f7e41220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.639+0000 7f3435ffb640 0 client.25522 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.639+0000 7f3457763740 1 -- 172.21.15.116:0/1598813532 shutdown_connections 2024-03-19T01:17:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.640+0000 7f3457763740 1 -- 172.21.15.116:0/1598813532 wait complete. 2024-03-19T01:17:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.640+0000 7f3457763740 20 client.25522 trim_cache size 0 max 16384 2024-03-19T01:17:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:17:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:17:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:17:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:17:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.641+0000 7f3454a9a640 1 -- 172.21.15.116:0/2545803493 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f34380a28a0 msgr2=0x7f34380a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.641+0000 7f3454a9a640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f34380a28a0 0x7f34380a4d60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x5602f7e47ef0 tx=0x7f34480f3e60 comp rx=0 tx=0).stop 2024-03-19T01:17:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.641+0000 7f3454a9a640 1 -- 172.21.15.116:0/2545803493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3450103a70 msgr2=0x7f3450199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:23.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.641+0000 7f3454a9a640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3450103a70 0x7f3450199d00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f34480324c0 tx=0x7f3448032f40 comp rx=0 tx=0).stop 2024-03-19T01:17:23.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.642+0000 7f3454a9a640 1 -- 172.21.15.116:0/2545803493 shutdown_connections 2024-03-19T01:17:23.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.642+0000 7f3454a9a640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3450105620 0x7f34501a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.642+0000 7f3454a9a640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3450104c60 0x7f345019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.642+0000 7f3454a9a640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f34380a28a0 0x7f34380a4d60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.643+0000 7f3454a9a640 1 --2- 172.21.15.116:0/2545803493 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3450103a70 0x7f3450199d00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:23.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.643+0000 7f3454a9a640 1 -- 172.21.15.116:0/2545803493 >> 172.21.15.116:0/2545803493 conn(0x7f34500fefe0 msgr2=0x7f3450107fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:23.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:23 smithi162 ceph-mon[67178]: pgmap v495: 97 pgs: 97 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.9 MiB/s wr, 607 op/s 2024-03-19T01:17:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.649+0000 7f3454a9a640 1 -- 172.21.15.116:0/2545803493 shutdown_connections 2024-03-19T01:17:23.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:23.652+0000 7f3454a9a640 1 -- 172.21.15.116:0/2545803493 wait complete. 2024-03-19T01:17:23.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:23 smithi116 ceph-mon[62701]: pgmap v495: 97 pgs: 97 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.9 MiB/s wr, 607 op/s 2024-03-19T01:17:23.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:23 smithi156 ceph-mon[66799]: pgmap v495: 97 pgs: 97 active+clean; 1.6 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.9 MiB/s wr, 607 op/s 2024-03-19T01:17:25.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:25 smithi162 ceph-mon[67178]: pgmap v496: 97 pgs: 97 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.6 MiB/s wr, 755 op/s 2024-03-19T01:17:25.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:25 smithi116 ceph-mon[62701]: pgmap v496: 97 pgs: 97 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.6 MiB/s wr, 755 op/s 2024-03-19T01:17:25.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:25 smithi156 ceph-mon[66799]: pgmap v496: 97 pgs: 97 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.6 MiB/s wr, 755 op/s 2024-03-19T01:17:26.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 280.0 s, 2342.8 tps, lat 42.846 ms stddev 46.484 2024-03-19T01:17:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:27 smithi162 ceph-mon[67178]: pgmap v497: 97 pgs: 97 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.7 MiB/s wr, 650 op/s 2024-03-19T01:17:27.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:27 smithi116 ceph-mon[62701]: pgmap v497: 97 pgs: 97 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.7 MiB/s wr, 650 op/s 2024-03-19T01:17:27.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:27 smithi156 ceph-mon[66799]: pgmap v497: 97 pgs: 97 active+clean; 1.6 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.7 MiB/s wr, 650 op/s 2024-03-19T01:17:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:29 smithi162 ceph-mon[67178]: pgmap v498: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.1 MiB/s wr, 711 op/s 2024-03-19T01:17:29.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:29 smithi116 ceph-mon[62701]: pgmap v498: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.1 MiB/s wr, 711 op/s 2024-03-19T01:17:29.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:29 smithi156 ceph-mon[66799]: pgmap v498: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.1 MiB/s wr, 711 op/s 2024-03-19T01:17:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 285.0 s, 1791.0 tps, lat 55.635 ms stddev 57.152 2024-03-19T01:17:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:31 smithi162 ceph-mon[67178]: pgmap v499: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.0 MiB/s wr, 693 op/s 2024-03-19T01:17:31.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:31 smithi116 ceph-mon[62701]: pgmap v499: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.0 MiB/s wr, 693 op/s 2024-03-19T01:17:31.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:31 smithi156 ceph-mon[66799]: pgmap v499: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.0 MiB/s wr, 693 op/s 2024-03-19T01:17:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:33 smithi162 ceph-mon[67178]: pgmap v500: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.1 MiB/s wr, 672 op/s 2024-03-19T01:17:33.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:33 smithi116 ceph-mon[62701]: pgmap v500: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.1 MiB/s wr, 672 op/s 2024-03-19T01:17:33.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:33 smithi156 ceph-mon[66799]: pgmap v500: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.1 MiB/s wr, 672 op/s 2024-03-19T01:17:35.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:35 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:35.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:35 smithi162 ceph-mon[67178]: pgmap v501: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 762 op/s 2024-03-19T01:17:35.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:35 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:35.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:35 smithi116 ceph-mon[62701]: pgmap v501: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 762 op/s 2024-03-19T01:17:35.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:35 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:35.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:35 smithi156 ceph-mon[66799]: pgmap v501: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.7 MiB/s wr, 762 op/s 2024-03-19T01:17:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 290.0 s, 2340.8 tps, lat 42.829 ms stddev 46.575 2024-03-19T01:17:36.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.773+0000 7f202e6dc640 1 -- 172.21.15.116:0/651881959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f202810db40 msgr2=0x7f202810dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.773+0000 7f202e6dc640 1 --2- 172.21.15.116:0/651881959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f202810db40 0x7f202810dfc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2020009ca0 tx=0x7f202002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.773+0000 7f202e6dc640 1 -- 172.21.15.116:0/651881959 shutdown_connections 2024-03-19T01:17:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.773+0000 7f202e6dc640 1 --2- 172.21.15.116:0/651881959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f202810db40 0x7f202810dfc0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.773+0000 7f202e6dc640 1 --2- 172.21.15.116:0/651881959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f202810b190 0x7f202810d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.773+0000 7f202e6dc640 1 --2- 172.21.15.116:0/651881959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2028108820 0x7f202810ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.773+0000 7f202e6dc640 1 -- 172.21.15.116:0/651881959 >> 172.21.15.116:0/651881959 conn(0x7f202806e860 msgr2=0x7f2028070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.773+0000 7f202e6dc640 1 -- 172.21.15.116:0/651881959 shutdown_connections 2024-03-19T01:17:36.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.773+0000 7f202e6dc640 1 -- 172.21.15.116:0/651881959 wait complete. 2024-03-19T01:17:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.773+0000 7f202e6dc640 1 Processor -- start 2024-03-19T01:17:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.773+0000 7f202e6dc640 1 -- start start 2024-03-19T01:17:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f202e6dc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2028108820 0x7f20281a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f202e6dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f202810b190 0x7f20281a6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f202e6dc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f202810db40 0x7f20281adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f202e6dc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f202811c2c0 con 0x7f202810b190 2024-03-19T01:17:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f202e6dc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f202811c140 con 0x7f2028108820 2024-03-19T01:17:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f202e6dc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f202811c440 con 0x7f202810db40 2024-03-19T01:17:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f20277fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f202810b190 0x7f20281a6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f20277fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f202810b190 0x7f20281a6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49774/0 (socket says 172.21.15.116:49774) 2024-03-19T01:17:36.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f2027fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2028108820 0x7f20281a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f20277fe640 1 -- 172.21.15.116:0/1804863912 learned_addr learned my addr 172.21.15.116:0/1804863912 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:36.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f202cc79640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f202810db40 0x7f20281adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f20277fe640 1 -- 172.21.15.116:0/1804863912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f202810db40 msgr2=0x7f20281adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:36.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f20277fe640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f202810db40 0x7f20281adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f20277fe640 1 -- 172.21.15.116:0/1804863912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2028108820 msgr2=0x7f20281a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:36.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f20277fe640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2028108820 0x7f20281a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f20277fe640 1 -- 172.21.15.116:0/1804863912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20281ae450 con 0x7f202810b190 2024-03-19T01:17:36.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f2027fff640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2028108820 0x7f20281a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:17:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.774+0000 7f20277fe640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f202810b190 0x7f20281a6dc0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e9f0 tx=0x7f201c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.775+0000 7f20257fa640 1 -- 172.21.15.116:0/1804863912 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c00cd80 con 0x7f202810b190 2024-03-19T01:17:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.775+0000 7f20257fa640 1 -- 172.21.15.116:0/1804863912 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f201c005500 con 0x7f202810b190 2024-03-19T01:17:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.775+0000 7f202e6dc640 1 -- 172.21.15.116:0/1804863912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20281ae740 con 0x7f202810b190 2024-03-19T01:17:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.775+0000 7f20257fa640 1 -- 172.21.15.116:0/1804863912 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c010690 con 0x7f202810b190 2024-03-19T01:17:36.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.775+0000 7f202e6dc640 1 -- 172.21.15.116:0/1804863912 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20281aec40 con 0x7f202810b190 2024-03-19T01:17:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.777+0000 7f20257fa640 1 -- 172.21.15.116:0/1804863912 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f201c010830 con 0x7f202810b190 2024-03-19T01:17:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.778+0000 7f20313a1740 1 Processor -- start 2024-03-19T01:17:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.778+0000 7f20313a1740 1 -- start start 2024-03-19T01:17:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.778+0000 7f20313a1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7e1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.778+0000 7f20313a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7e65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.778+0000 7f20313a1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7e90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.778+0000 7f20313a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fbfa745be0 con 0x55fbfa7e1af0 2024-03-19T01:17:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.778+0000 7f20313a1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fbfa6ee960 con 0x55fbfa7e1100 2024-03-19T01:17:36.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.778+0000 7f20313a1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fbfa741c80 con 0x55fbfa7e6c60 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.778+0000 7f202cc79640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7e1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.778+0000 7f202cc79640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7e1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:55704/0 (socket says 172.21.15.116:55704) 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.778+0000 7f202cc79640 1 -- 172.21.15.116:0/595976175 learned_addr learned my addr 172.21.15.116:0/595976175 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.779+0000 7f2027fff640 1 --2- 172.21.15.116:0/595976175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7e65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.779+0000 7f20277fe640 1 --2- 172.21.15.116:0/595976175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7e90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.779+0000 7f202cc79640 1 -- 172.21.15.116:0/595976175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 msgr2=0x55fbfa7e90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.779+0000 7f202cc79640 1 --2- 172.21.15.116:0/595976175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7e90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.779+0000 7f202cc79640 1 -- 172.21.15.116:0/595976175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1af0 msgr2=0x55fbfa7e65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.779+0000 7f202cc79640 1 --2- 172.21.15.116:0/595976175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7e65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.779+0000 7f202cc79640 1 -- 172.21.15.116:0/595976175 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbfa6e5d20 con 0x55fbfa7e1100 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.779+0000 7f20277fe640 1 --2- 172.21.15.116:0/595976175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7e90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.779+0000 7f202cc79640 1 --2- 172.21.15.116:0/595976175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7e1520 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f202002c700 tx=0x7f20200323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73999eb706a144b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.780+0000 7f20257fa640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20100a28a0 0x7f20100a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.780+0000 7f200e7fc640 1 -- 172.21.15.116:0/595976175 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020043070 con 0x55fbfa7e1100 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.780+0000 7f200e7fc640 1 -- 172.21.15.116:0/595976175 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f202002cca0 con 0x55fbfa7e1100 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.780+0000 7f200e7fc640 1 -- 172.21.15.116:0/595976175 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020039040 con 0x55fbfa7e1100 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.780+0000 7f2027fff640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20100a28a0 0x7f20100a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.780+0000 7f20257fa640 1 -- 172.21.15.116:0/1804863912 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f201c0cee70 con 0x7f202810b190 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.780+0000 7f2027fff640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20100a28a0 0x7f20100a4d60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55fbfa7e1ef0 tx=0x7f2018005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.780+0000 7f20313a1740 1 -- 172.21.15.116:0/595976175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 msgr2=0x55fbfa7e1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.780+0000 7f20313a1740 1 --2- 172.21.15.116:0/595976175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7e1520 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f202002c700 tx=0x7f20200323d0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 -- 172.21.15.116:0/595976175 shutdown_connections 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 --2- 172.21.15.116:0/595976175 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7e90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 --2- 172.21.15.116:0/595976175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7e65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 --2- 172.21.15.116:0/595976175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7e1520 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 -- 172.21.15.116:0/595976175 >> 172.21.15.116:0/595976175 conn(0x55fbfa7dc670 msgr2=0x55fbfa7dca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 -- 172.21.15.116:0/595976175 shutdown_connections 2024-03-19T01:17:36.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 -- 172.21.15.116:0/595976175 wait complete. 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 -- start start 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 Processor -- start 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1100 0x55fbfa7eb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7eb600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7f2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fbfa7e9c00 con 0x55fbfa7e1100 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fbfa701db0 con 0x55fbfa7e1af0 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fbfa7e9d80 con 0x55fbfa7e6c60 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.781+0000 7f20313a1740 10 client.0 _ll_register_callbacks cb 0x55fbfa706e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f2027fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1100 0x55fbfa7eb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f2027fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1100 0x55fbfa7eb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49796/0 (socket says 172.21.15.116:49796) 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f2027fff640 1 -- 172.21.15.116:0/3000804913 learned_addr learned my addr 172.21.15.116:0/3000804913 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f20277fe640 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7eb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f202cc79640 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f2027fff640 1 -- 172.21.15.116:0/3000804913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 msgr2=0x55fbfa7f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f2027fff640 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f2027fff640 1 -- 172.21.15.116:0/3000804913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1af0 msgr2=0x55fbfa7eb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f2027fff640 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7eb600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f2027fff640 1 -- 172.21.15.116:0/3000804913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbfa700c60 con 0x55fbfa7e1100 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f20277fe640 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7eb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f2027fff640 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1100 0x55fbfa7eb0c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2018009560 tx=0x7f2018004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab73728fa315080 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f20077fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f20077fe640 1 -- 172.21.15.116:0/3000804913 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2018004020 con 0x55fbfa7e1100 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f20313a1740 1 -- 172.21.15.116:0/3000804913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f202002cca0 con 0x55fbfa7e1100 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f20077fe640 1 -- 172.21.15.116:0/3000804913 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2018018030 con 0x55fbfa7e1100 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f20077fe640 1 -- 172.21.15.116:0/3000804913 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f20180041c0 con 0x55fbfa7e1100 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f20313a1740 10 client.15711 fetch_fsmap learned FSMap version 42 2024-03-19T01:17:36.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.782+0000 7f20313a1740 1 -- 172.21.15.116:0/3000804913 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55fbfa45a210 con 0x55fbfa7e1100 2024-03-19T01:17:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.783+0000 7f20077fe640 1 -- 172.21.15.116:0/3000804913 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f202002cca0 con 0x55fbfa7e1100 2024-03-19T01:17:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.783+0000 7f20077fe640 1 -- 172.21.15.116:0/3000804913 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 42) ==== 8366+0+0 (secure 0 0 0) 0x7f2018003730 con 0x55fbfa7e1100 2024-03-19T01:17:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.783+0000 7f20313a1740 10 client.15711 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T01:17:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.783+0000 7f20313a1740 10 client.15711 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:17:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.783+0000 7f20313a1740 20 client.15711 populate_metadata read hostname 'smithi116' 2024-03-19T01:17:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.783+0000 7f20313a1740 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fbfa7f7310 0x55fbfa7f97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.783+0000 7f20313a1740 4 client.15711 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:17:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.783+0000 7f20313a1740 1 -- 172.21.15.116:0/3000804913 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbfa753120 con 0x55fbfa7f7310 2024-03-19T01:17:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.783+0000 7f20277fe640 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fbfa7f7310 0x55fbfa7f97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.783+0000 7f20077fe640 10 client.15711 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.783+0000 7f20277fe640 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fbfa7f7310 0x55fbfa7f97b0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78b74fa1b2beaee6 server_cookie=2d22bb2bd1d25051 in_seq=0 out_seq=0 2024-03-19T01:17:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20077fe640 1 -- 172.21.15.116:0/3000804913 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fbfa753120 con 0x55fbfa7f7310 2024-03-19T01:17:36.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20077fe640 10 client.15711 handle_command_reply: tid=0 2024-03-19T01:17:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20313a1740 1 client.15711 shutdown 2024-03-19T01:17:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20313a1740 1 -- 172.21.15.116:0/3000804913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1100 msgr2=0x55fbfa7eb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20313a1740 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1100 0x55fbfa7eb0c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2018009560 tx=0x7f2018004730 comp rx=0 tx=0).stop 2024-03-19T01:17:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20313a1740 1 -- 172.21.15.116:0/3000804913 shutdown_connections 2024-03-19T01:17:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20313a1740 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fbfa7f7310 0x55fbfa7f97b0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20313a1740 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20313a1740 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7eb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20077fe640 0 client.15711 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20313a1740 1 --2- 172.21.15.116:0/3000804913 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1100 0x55fbfa7eb0c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20313a1740 1 -- 172.21.15.116:0/3000804913 >> 172.21.15.116:0/3000804913 conn(0x55fbfa7dc670 msgr2=0x55fbfa7dd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20313a1740 1 -- 172.21.15.116:0/3000804913 shutdown_connections 2024-03-19T01:17:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20313a1740 1 -- 172.21.15.116:0/3000804913 wait complete. 2024-03-19T01:17:36.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.784+0000 7f20313a1740 20 client.15711 trim_cache size 0 max 16384 2024-03-19T01:17:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.804+0000 7f20313a1740 1 Processor -- start 2024-03-19T01:17:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.804+0000 7f20313a1740 1 -- start start 2024-03-19T01:17:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f20313a1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7eb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f20313a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7f2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f20313a1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7db930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f20277fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f20277fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7f2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49808/0 (socket says 172.21.15.116:49808) 2024-03-19T01:17:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f20277fe640 1 -- 172.21.15.116:0/254837494 learned_addr learned my addr 172.21.15.116:0/254837494 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f20313a1740 1 -- 172.21.15.116:0/254837494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f20281b7af0 con 0x55fbfa7e1af0 2024-03-19T01:17:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f20313a1740 1 -- 172.21.15.116:0/254837494 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f20281b96c0 con 0x55fbfa7e1100 2024-03-19T01:17:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f20313a1740 1 -- 172.21.15.116:0/254837494 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f20281b81c0 con 0x55fbfa7e6c60 2024-03-19T01:17:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f202cc79640 1 --2- 172.21.15.116:0/254837494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7db930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f202cc79640 1 -- 172.21.15.116:0/254837494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 msgr2=0x55fbfa7eb560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:17:36.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f202cc79640 1 --2- 172.21.15.116:0/254837494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7eb560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f202cc79640 1 -- 172.21.15.116:0/254837494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1af0 msgr2=0x55fbfa7f2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:36.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.805+0000 7f202cc79640 1 --2- 172.21.15.116:0/254837494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7f2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.806+0000 7f202cc79640 1 -- 172.21.15.116:0/254837494 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbfa526620 con 0x55fbfa7e6c60 2024-03-19T01:17:36.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.806+0000 7f20277fe640 1 --2- 172.21.15.116:0/254837494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:17:36.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.806+0000 7f202cc79640 1 --2- 172.21.15.116:0/254837494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7db930 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f20200324c0 tx=0x7f2020004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78d4394286a758e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.806+0000 7f200dffb640 1 -- 172.21.15.116:0/254837494 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020039070 con 0x55fbfa7e6c60 2024-03-19T01:17:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.806+0000 7f200dffb640 1 -- 172.21.15.116:0/254837494 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2020003970 con 0x55fbfa7e6c60 2024-03-19T01:17:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.806+0000 7f200dffb640 1 -- 172.21.15.116:0/254837494 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020043040 con 0x55fbfa7e6c60 2024-03-19T01:17:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.807+0000 7f20313a1740 1 -- 172.21.15.116:0/254837494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 msgr2=0x55fbfa7db930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.807+0000 7f20313a1740 1 --2- 172.21.15.116:0/254837494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7db930 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f20200324c0 tx=0x7f2020004a40 comp rx=0 tx=0).stop 2024-03-19T01:17:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.807+0000 7f20313a1740 1 -- 172.21.15.116:0/254837494 shutdown_connections 2024-03-19T01:17:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.807+0000 7f20313a1740 1 --2- 172.21.15.116:0/254837494 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7db930 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.807+0000 7f20313a1740 1 --2- 172.21.15.116:0/254837494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.807+0000 7f20313a1740 1 --2- 172.21.15.116:0/254837494 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7eb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.807+0000 7f20313a1740 1 -- 172.21.15.116:0/254837494 >> 172.21.15.116:0/254837494 conn(0x55fbfa7dc670 msgr2=0x55fbfa7db220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.807+0000 7f20313a1740 1 -- 172.21.15.116:0/254837494 shutdown_connections 2024-03-19T01:17:36.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.807+0000 7f20313a1740 1 -- 172.21.15.116:0/254837494 wait complete. 2024-03-19T01:17:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.807+0000 7f20313a1740 1 -- start start 2024-03-19T01:17:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20313a1740 1 Processor -- start 2024-03-19T01:17:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20313a1740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7ef830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20313a1740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7efd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20313a1740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20313a1740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f20281b8ff0 con 0x55fbfa7e6c60 2024-03-19T01:17:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20313a1740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f20281b8890 con 0x55fbfa7e1100 2024-03-19T01:17:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20313a1740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f20281b5ad0 con 0x55fbfa7e1af0 2024-03-19T01:17:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20313a1740 10 client.0 _ll_register_callbacks cb 0x7f2018009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:17:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f2027fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7ef830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f202cc79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20277fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7efd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f2027fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7ef830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:55726/0 (socket says 172.21.15.116:55726) 2024-03-19T01:17:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f2027fff640 1 -- 172.21.15.116:0/1061777949 learned_addr learned my addr 172.21.15.116:0/1061777949 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20277fe640 1 -- 172.21.15.116:0/1061777949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 msgr2=0x55fbfa7ef830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20277fe640 1 --2- 172.21.15.116:0/1061777949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7ef830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20277fe640 1 -- 172.21.15.116:0/1061777949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e6c60 msgr2=0x55fbfa7f0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20277fe640 1 --2- 172.21.15.116:0/1061777949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7f0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f20277fe640 1 -- 172.21.15.116:0/1061777949 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbfa7f1370 con 0x55fbfa7e1af0 2024-03-19T01:17:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.808+0000 7f2027fff640 1 --2- 172.21.15.116:0/1061777949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7ef830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:17:36.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20277fe640 1 --2- 172.21.15.116:0/1061777949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7efd70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f201c09b780 tx=0x7f201c086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dbf0da2fd350c57 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20077fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:17:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20077fe640 1 -- 172.21.15.116:0/1061777949 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c09f940 con 0x55fbfa7e1af0 2024-03-19T01:17:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20077fe640 1 -- 172.21.15.116:0/1061777949 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f201c099af0 con 0x55fbfa7e1af0 2024-03-19T01:17:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20313a1740 1 -- 172.21.15.116:0/1061777949 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2020003970 con 0x55fbfa7e1af0 2024-03-19T01:17:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20077fe640 1 -- 172.21.15.116:0/1061777949 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f201c0a3f90 con 0x55fbfa7e1af0 2024-03-19T01:17:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20313a1740 10 client.25537 fetch_fsmap learned FSMap version 42 2024-03-19T01:17:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20313a1740 1 -- 172.21.15.116:0/1061777949 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55fbfa7f26f0 con 0x55fbfa7e1af0 2024-03-19T01:17:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20077fe640 1 -- 172.21.15.116:0/1061777949 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2020003970 con 0x55fbfa7e1af0 2024-03-19T01:17:36.818 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20077fe640 1 -- 172.21.15.116:0/1061777949 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 42) ==== 8366+0+0 (secure 0 0 0) 0x7f201c0bb040 con 0x55fbfa7e1af0 2024-03-19T01:17:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20313a1740 10 client.25537 fetch_fsmap finished waiting for FSMap version 42 2024-03-19T01:17:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20313a1740 10 client.25537 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:17:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20313a1740 20 client.25537 populate_metadata read hostname 'smithi116' 2024-03-19T01:17:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20313a1740 1 --2- 172.21.15.116:0/1061777949 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fbfa85dc70 0x55fbfa860090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20313a1740 4 client.25537 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:17:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.809+0000 7f20313a1740 1 -- 172.21.15.116:0/1061777949 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fbfa74d470 con 0x55fbfa85dc70 2024-03-19T01:17:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.810+0000 7f2027fff640 1 --2- 172.21.15.116:0/1061777949 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fbfa85dc70 0x55fbfa860090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.810+0000 7f2027fff640 1 --2- 172.21.15.116:0/1061777949 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fbfa85dc70 0x55fbfa860090 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bc5ab976df2e0e7 server_cookie=e778b61aae17f2ba in_seq=0 out_seq=0 2024-03-19T01:17:36.819 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.810+0000 7f20077fe640 10 client.25537 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:36.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20077fe640 1 -- 172.21.15.116:0/1061777949 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fbfa74d470 con 0x55fbfa85dc70 2024-03-19T01:17:36.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20077fe640 10 client.25537 handle_command_reply: tid=0 2024-03-19T01:17:36.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20313a1740 1 client.25537 shutdown 2024-03-19T01:17:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20313a1740 1 -- 172.21.15.116:0/1061777949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e1af0 msgr2=0x55fbfa7efd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20313a1740 1 --2- 172.21.15.116:0/1061777949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7efd70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f201c09b780 tx=0x7f201c086040 comp rx=0 tx=0).stop 2024-03-19T01:17:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20313a1740 1 -- 172.21.15.116:0/1061777949 shutdown_connections 2024-03-19T01:17:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20313a1740 1 --2- 172.21.15.116:0/1061777949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fbfa7e6c60 0x55fbfa7f0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20313a1740 1 --2- 172.21.15.116:0/1061777949 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fbfa85dc70 0x55fbfa860090 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20077fe640 0 client.25537 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20313a1740 1 --2- 172.21.15.116:0/1061777949 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fbfa7e1af0 0x55fbfa7efd70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20313a1740 1 --2- 172.21.15.116:0/1061777949 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fbfa7e1100 0x55fbfa7ef830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20313a1740 1 -- 172.21.15.116:0/1061777949 >> 172.21.15.116:0/1061777949 conn(0x55fbfa7dc670 msgr2=0x55fbfa7eb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20313a1740 1 -- 172.21.15.116:0/1061777949 shutdown_connections 2024-03-19T01:17:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20313a1740 1 -- 172.21.15.116:0/1061777949 wait complete. 2024-03-19T01:17:36.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.868+0000 7f20313a1740 20 client.25537 trim_cache size 0 max 16384 2024-03-19T01:17:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:17:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:17:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:17:36.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:17:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.869+0000 7f202e6dc640 1 -- 172.21.15.116:0/1804863912 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20100a28a0 msgr2=0x7f20100a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.869+0000 7f202e6dc640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20100a28a0 0x7f20100a4d60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55fbfa7e1ef0 tx=0x7f2018005f90 comp rx=0 tx=0).stop 2024-03-19T01:17:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.869+0000 7f202e6dc640 1 -- 172.21.15.116:0/1804863912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f202810b190 msgr2=0x7f20281a6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.869+0000 7f202e6dc640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f202810b190 0x7f20281a6dc0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f201c00e9f0 tx=0x7f201c00edf0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.869+0000 7f202e6dc640 1 -- 172.21.15.116:0/1804863912 shutdown_connections 2024-03-19T01:17:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.869+0000 7f202e6dc640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f202810db40 0x7f20281adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.869+0000 7f202e6dc640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f202810b190 0x7f20281a6dc0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.869+0000 7f202e6dc640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20100a28a0 0x7f20100a4d60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.869+0000 7f202e6dc640 1 --2- 172.21.15.116:0/1804863912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2028108820 0x7f20281a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.869+0000 7f202e6dc640 1 -- 172.21.15.116:0/1804863912 >> 172.21.15.116:0/1804863912 conn(0x7f202806e860 msgr2=0x7f2028109640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.869+0000 7f202e6dc640 1 -- 172.21.15.116:0/1804863912 shutdown_connections 2024-03-19T01:17:36.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:36.869+0000 7f202e6dc640 1 -- 172.21.15.116:0/1804863912 wait complete. 2024-03-19T01:17:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:37 smithi162 ceph-mon[67178]: pgmap v502: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.9 MiB/s wr, 628 op/s 2024-03-19T01:17:37.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:37 smithi116 ceph-mon[62701]: pgmap v502: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.9 MiB/s wr, 628 op/s 2024-03-19T01:17:37.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:37 smithi156 ceph-mon[66799]: pgmap v502: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.9 MiB/s wr, 628 op/s 2024-03-19T01:17:39.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:39 smithi162 ceph-mon[67178]: pgmap v503: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 696 op/s 2024-03-19T01:17:39.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:39 smithi116 ceph-mon[62701]: pgmap v503: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 696 op/s 2024-03-19T01:17:39.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:39 smithi156 ceph-mon[66799]: pgmap v503: 97 pgs: 97 active+clean; 1.6 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 696 op/s 2024-03-19T01:17:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 295.0 s, 2063.2 tps, lat 48.421 ms stddev 52.023 2024-03-19T01:17:41.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:41 smithi162 ceph-mon[67178]: pgmap v504: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 5.6 MiB/s wr, 619 op/s 2024-03-19T01:17:41.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:41 smithi116 ceph-mon[62701]: pgmap v504: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 5.6 MiB/s wr, 619 op/s 2024-03-19T01:17:41.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:41 smithi156 ceph-mon[66799]: pgmap v504: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 5.6 MiB/s wr, 619 op/s 2024-03-19T01:17:43.527 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:43 smithi156 ceph-mon[66799]: pgmap v505: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.6 MiB/s wr, 602 op/s 2024-03-19T01:17:43.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:43 smithi162 ceph-mon[67178]: pgmap v505: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.6 MiB/s wr, 602 op/s 2024-03-19T01:17:43.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:43 smithi116 ceph-mon[62701]: pgmap v505: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.6 MiB/s wr, 602 op/s 2024-03-19T01:17:45.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:45 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:45.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:45 smithi162 ceph-mon[67178]: pgmap v506: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 722 op/s 2024-03-19T01:17:45.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:45 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:45.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:45 smithi116 ceph-mon[62701]: pgmap v506: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 722 op/s 2024-03-19T01:17:45.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:45 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:17:45.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:45 smithi156 ceph-mon[66799]: pgmap v506: 97 pgs: 97 active+clean; 1.6 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 722 op/s 2024-03-19T01:17:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 300.0 s, 2368.8 tps, lat 42.136 ms stddev 46.392 2024-03-19T01:17:47.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:47 smithi162 ceph-mon[67178]: pgmap v507: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 649 op/s 2024-03-19T01:17:47.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:47 smithi116 ceph-mon[62701]: pgmap v507: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 649 op/s 2024-03-19T01:17:47.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:47 smithi156 ceph-mon[66799]: pgmap v507: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 649 op/s 2024-03-19T01:17:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:49 smithi162 ceph-mon[67178]: pgmap v508: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.3 MiB/s wr, 657 op/s 2024-03-19T01:17:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:49 smithi116 ceph-mon[62701]: pgmap v508: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.3 MiB/s wr, 657 op/s 2024-03-19T01:17:49.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:49 smithi156 ceph-mon[66799]: pgmap v508: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.3 MiB/s wr, 657 op/s 2024-03-19T01:17:50.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.000+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3733323615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6a81009c0 msgr2=0x7fa6a8100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:50.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.000+0000 7fa6af9c6640 1 --2- 172.21.15.116:0/3733323615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6a81009c0 0x7fa6a8100e40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa698009930 tx=0x7fa69802bc30 comp rx=0 tx=0).stop 2024-03-19T01:17:50.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.000+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3733323615 shutdown_connections 2024-03-19T01:17:50.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.000+0000 7fa6af9c6640 1 --2- 172.21.15.116:0/3733323615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6a8101380 0x7fa6a8108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.000+0000 7fa6af9c6640 1 --2- 172.21.15.116:0/3733323615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6a81009c0 0x7fa6a8100e40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.000+0000 7fa6af9c6640 1 --2- 172.21.15.116:0/3733323615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6a8101d20 0x7fa6a8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.000+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3733323615 >> 172.21.15.116:0/3733323615 conn(0x7fa6a80fb470 msgr2=0x7fa6a80fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.000+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3733323615 shutdown_connections 2024-03-19T01:17:50.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.000+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3733323615 wait complete. 2024-03-19T01:17:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.001+0000 7fa6af9c6640 1 Processor -- start 2024-03-19T01:17:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.002+0000 7fa6af9c6640 1 -- start start 2024-03-19T01:17:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.002+0000 7fa6af9c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6a81009c0 0x7fa6a8196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.002+0000 7fa6af9c6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6a8101380 0x7fa6a8196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.002+0000 7fa6af9c6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6a8101d20 0x7fa6a819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.002+0000 7fa6af9c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a810c3c0 con 0x7fa6a81009c0 2024-03-19T01:17:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.002+0000 7fa6af9c6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a810c240 con 0x7fa6a8101380 2024-03-19T01:17:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.003+0000 7fa6adf63640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6a8101d20 0x7fa6a819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.003+0000 7fa6adf63640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6a8101d20 0x7fa6a819dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:33816/0 (socket says 172.21.15.116:33816) 2024-03-19T01:17:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.003+0000 7fa6adf63640 1 -- 172.21.15.116:0/3731584163 learned_addr learned my addr 172.21.15.116:0/3731584163 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.003+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3731584163 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a810c540 con 0x7fa6a8101d20 2024-03-19T01:17:50.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.003+0000 7fa6ad762640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6a81009c0 0x7fa6a8196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.003+0000 7fa6ad762640 1 -- 172.21.15.116:0/3731584163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6a8101d20 msgr2=0x7fa6a819dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.004+0000 7fa6ad762640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6a8101d20 0x7fa6a819dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.004+0000 7fa6ad762640 1 -- 172.21.15.116:0/3731584163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6a8101380 msgr2=0x7fa6a8196f40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T01:17:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.004+0000 7fa6ad762640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6a8101380 0x7fa6a8196f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.004+0000 7fa6ad762640 1 -- 172.21.15.116:0/3731584163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a819e620 con 0x7fa6a81009c0 2024-03-19T01:17:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.004+0000 7fa6ad762640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6a81009c0 0x7fa6a8196a00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4005fd0 tx=0x7fa6a400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:50.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.004+0000 7fa6adf63640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6a8101d20 0x7fa6a819dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:17:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.005+0000 7fa69e7fc640 1 -- 172.21.15.116:0/3731584163 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4003970 con 0x7fa6a81009c0 2024-03-19T01:17:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.005+0000 7fa69e7fc640 1 -- 172.21.15.116:0/3731584163 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6a4004030 con 0x7fa6a81009c0 2024-03-19T01:17:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.006+0000 7fa69e7fc640 1 -- 172.21.15.116:0/3731584163 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4010790 con 0x7fa6a81009c0 2024-03-19T01:17:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.006+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3731584163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6a819e910 con 0x7fa6a81009c0 2024-03-19T01:17:50.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.006+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3731584163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6a819eeb0 con 0x7fa6a81009c0 2024-03-19T01:17:50.010 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.009+0000 7fa69e7fc640 1 -- 172.21.15.116:0/3731584163 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fa6a40041e0 con 0x7fa6a81009c0 2024-03-19T01:17:50.011 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.010+0000 7fa6b25b3740 1 Processor -- start 2024-03-19T01:17:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.010+0000 7fa6b25b3740 1 -- start start 2024-03-19T01:17:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.010+0000 7fa6b25b3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e65480 0x55e279e658a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.010+0000 7fa6b25b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e72eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.010+0000 7fa6b25b3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e73480 0x55e279e758f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.010+0000 7fa6b25b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e279dc9be0 con 0x55e279e65de0 2024-03-19T01:17:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.010+0000 7fa6b25b3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e279d72960 con 0x55e279e65480 2024-03-19T01:17:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.011+0000 7fa6b25b3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e279dc5c80 con 0x55e279e73480 2024-03-19T01:17:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.011+0000 7fa6adf63640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e65480 0x55e279e658a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.011+0000 7fa6acf61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e73480 0x55e279e758f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.011+0000 7fa6acf61640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e73480 0x55e279e758f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:33820/0 (socket says 172.21.15.116:33820) 2024-03-19T01:17:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.011+0000 7fa6acf61640 1 -- 172.21.15.116:0/517686736 learned_addr learned my addr 172.21.15.116:0/517686736 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:50.012 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.011+0000 7fa6ad762640 1 --2- 172.21.15.116:0/517686736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e72eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.011+0000 7fa6acf61640 1 -- 172.21.15.116:0/517686736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e65480 msgr2=0x55e279e658a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.011+0000 7fa6acf61640 1 --2- 172.21.15.116:0/517686736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e65480 0x55e279e658a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.011+0000 7fa6acf61640 1 -- 172.21.15.116:0/517686736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 msgr2=0x55e279e72eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.011+0000 7fa6acf61640 1 --2- 172.21.15.116:0/517686736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e72eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.011+0000 7fa6acf61640 1 -- 172.21.15.116:0/517686736 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e279d69d20 con 0x55e279e73480 2024-03-19T01:17:50.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.011+0000 7fa69e7fc640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa6940a2920 0x7fa6940a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.011+0000 7fa6acf61640 1 --2- 172.21.15.116:0/517686736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e73480 0x55e279e758f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa698009230 tx=0x7fa698005be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35328716bbbc8b84 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.012+0000 7fa6adf63640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa6940a2920 0x7fa6940a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.012+0000 7fa69e7fc640 1 -- 172.21.15.116:0/3731584163 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa6a40cefe0 con 0x7fa6a81009c0 2024-03-19T01:17:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.012+0000 7fa6937fe640 1 -- 172.21.15.116:0/517686736 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa698042020 con 0x55e279e73480 2024-03-19T01:17:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.012+0000 7fa6937fe640 1 -- 172.21.15.116:0/517686736 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa698009520 con 0x55e279e73480 2024-03-19T01:17:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.012+0000 7fa6937fe640 1 -- 172.21.15.116:0/517686736 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa698003b30 con 0x55e279e73480 2024-03-19T01:17:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.013+0000 7fa6adf63640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa6940a2920 0x7fa6940a4de0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000bd20 tx=0x7fa6a000b470 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.013+0000 7fa6b25b3740 1 -- 172.21.15.116:0/517686736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e73480 msgr2=0x55e279e758f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.013+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/517686736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e73480 0x55e279e758f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa698009230 tx=0x7fa698005be0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.013+0000 7fa6b25b3740 1 -- 172.21.15.116:0/517686736 shutdown_connections 2024-03-19T01:17:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.013+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/517686736 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e73480 0x55e279e758f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.013+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/517686736 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e72eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.013+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/517686736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e65480 0x55e279e658a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.013+0000 7fa6b25b3740 1 -- 172.21.15.116:0/517686736 >> 172.21.15.116:0/517686736 conn(0x55e279e61c30 msgr2=0x55e279e60180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.013+0000 7fa6b25b3740 1 -- 172.21.15.116:0/517686736 shutdown_connections 2024-03-19T01:17:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.013+0000 7fa6b25b3740 1 -- 172.21.15.116:0/517686736 wait complete. 2024-03-19T01:17:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.013+0000 7fa6b25b3740 1 -- start start 2024-03-19T01:17:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.013+0000 7fa6b25b3740 1 Processor -- start 2024-03-19T01:17:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6b25b3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e65480 0x55e279e88890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6b25b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e88dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6b25b3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e73480 0x55e279e8fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6b25b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e279e87310 con 0x55e279e65de0 2024-03-19T01:17:50.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6b25b3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e279d85db0 con 0x55e279e73480 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6b25b3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e279e87490 con 0x55e279e65480 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6b25b3740 10 client.0 _ll_register_callbacks cb 0x55e279d8ae70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6acf61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e88dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6ad762640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e65480 0x55e279e88890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6acf61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e88dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52364/0 (socket says 172.21.15.116:52364) 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6acf61640 1 -- 172.21.15.116:0/2401119724 learned_addr learned my addr 172.21.15.116:0/2401119724 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6adf63640 1 --2- 172.21.15.116:0/2401119724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e73480 0x55e279e8fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6acf61640 1 -- 172.21.15.116:0/2401119724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e65480 msgr2=0x55e279e88890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6acf61640 1 --2- 172.21.15.116:0/2401119724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e65480 0x55e279e88890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6acf61640 1 -- 172.21.15.116:0/2401119724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e73480 msgr2=0x55e279e8fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6acf61640 1 --2- 172.21.15.116:0/2401119724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e73480 0x55e279e8fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6acf61640 1 -- 172.21.15.116:0/2401119724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e279d84c60 con 0x55e279e65de0 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6ad762640 1 --2- 172.21.15.116:0/2401119724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e65480 0x55e279e88890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:17:50.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6acf61640 1 --2- 172.21.15.116:0/2401119724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e88dd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa69802c170 tx=0x7fa698004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b77051c58077ac0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa690ff9640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:17:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa690ff9640 1 -- 172.21.15.116:0/2401119724 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa698042030 con 0x55e279e65de0 2024-03-19T01:17:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa690ff9640 1 -- 172.21.15.116:0/2401119724 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa698003cc0 con 0x55e279e65de0 2024-03-19T01:17:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.014+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2401119724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa698009520 con 0x55e279e65de0 2024-03-19T01:17:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.015+0000 7fa690ff9640 1 -- 172.21.15.116:0/2401119724 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fa698005d50 con 0x55e279e65de0 2024-03-19T01:17:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.015+0000 7fa6b25b3740 10 client.15726 fetch_fsmap learned FSMap version 43 2024-03-19T01:17:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.015+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2401119724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55e279ade210 con 0x55e279e65de0 2024-03-19T01:17:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.015+0000 7fa690ff9640 1 -- 172.21.15.116:0/2401119724 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa698034920 con 0x55e279e65de0 2024-03-19T01:17:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.015+0000 7fa690ff9640 1 -- 172.21.15.116:0/2401119724 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7fa698051490 con 0x55e279e65de0 2024-03-19T01:17:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.015+0000 7fa6b25b3740 10 client.15726 fetch_fsmap finished waiting for FSMap version 43 2024-03-19T01:17:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.015+0000 7fa6b25b3740 10 client.15726 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:17:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.015+0000 7fa6b25b3740 20 client.15726 populate_metadata read hostname 'smithi116' 2024-03-19T01:17:50.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.015+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/2401119724 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e279e94300 0x55e279e966f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.015+0000 7fa6b25b3740 4 client.15726 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.015+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2401119724 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa698009520 con 0x55e279e94300 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.015+0000 7fa6ad762640 1 --2- 172.21.15.116:0/2401119724 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e279e94300 0x55e279e966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.016+0000 7fa6ad762640 1 --2- 172.21.15.116:0/2401119724 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e279e94300 0x55e279e966f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54f94edea36e0593 server_cookie=af7690dae468b17a in_seq=0 out_seq=0 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.016+0000 7fa690ff9640 10 client.15726 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.017+0000 7fa690ff9640 1 -- 172.21.15.116:0/2401119724 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fa698009520 con 0x55e279e94300 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.017+0000 7fa690ff9640 10 client.15726 handle_command_reply: tid=0 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.017+0000 7fa6b25b3740 1 client.15726 shutdown 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.017+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2401119724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 msgr2=0x55e279e88dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.017+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/2401119724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e88dd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa69802c170 tx=0x7fa698004270 comp rx=0 tx=0).stop 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.017+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2401119724 shutdown_connections 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.017+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/2401119724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e73480 0x55e279e8fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.017+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/2401119724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e88dd0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.017+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/2401119724 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e279e94300 0x55e279e966f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.017+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/2401119724 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e65480 0x55e279e88890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.017+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2401119724 >> 172.21.15.116:0/2401119724 conn(0x55e279e61c30 msgr2=0x55e279e62050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.017+0000 7fa690ff9640 0 client.15726 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.017+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2401119724 shutdown_connections 2024-03-19T01:17:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.018+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2401119724 wait complete. 2024-03-19T01:17:50.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.018+0000 7fa6b25b3740 20 client.15726 trim_cache size 0 max 16384 2024-03-19T01:17:50.038 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.037+0000 7fa6b25b3740 1 Processor -- start 2024-03-19T01:17:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.037+0000 7fa6b25b3740 1 -- start start 2024-03-19T01:17:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.037+0000 7fa6b25b3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e65480 0x55e279e64020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.037+0000 7fa6b25b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e8fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.037+0000 7fa6b25b3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e73480 0x55e279e64560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.037+0000 7fa6acf61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e8fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.037+0000 7fa6acf61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e8fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52380/0 (socket says 172.21.15.116:52380) 2024-03-19T01:17:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.037+0000 7fa6acf61640 1 -- 172.21.15.116:0/2118382096 learned_addr learned my addr 172.21.15.116:0/2118382096 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.037+0000 7fa6ad762640 1 --2- 172.21.15.116:0/2118382096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e65480 0x55e279e64020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.038+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2118382096 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a8072ba0 con 0x55e279e65de0 2024-03-19T01:17:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.038+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2118382096 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a819fe40 con 0x55e279e73480 2024-03-19T01:17:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.038+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2118382096 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a8073270 con 0x55e279e65480 2024-03-19T01:17:50.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.038+0000 7fa6adf63640 1 --2- 172.21.15.116:0/2118382096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e73480 0x55e279e64560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.038+0000 7fa6acf61640 1 -- 172.21.15.116:0/2118382096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e65480 msgr2=0x55e279e64020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.038+0000 7fa6acf61640 1 --2- 172.21.15.116:0/2118382096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e65480 0x55e279e64020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.038+0000 7fa6acf61640 1 -- 172.21.15.116:0/2118382096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e73480 msgr2=0x55e279e64560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.038+0000 7fa6acf61640 1 --2- 172.21.15.116:0/2118382096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e73480 0x55e279e64560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.038+0000 7fa6acf61640 1 -- 172.21.15.116:0/2118382096 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e279e5bee0 con 0x55e279e65de0 2024-03-19T01:17:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.038+0000 7fa6acf61640 1 --2- 172.21.15.116:0/2118382096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e8fdd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa698052fd0 tx=0x7fa69802c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50239d6a63da5916 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.039+0000 7fa692ffd640 1 -- 172.21.15.116:0/2118382096 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa698042040 con 0x55e279e65de0 2024-03-19T01:17:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.039+0000 7fa692ffd640 1 -- 172.21.15.116:0/2118382096 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa698009520 con 0x55e279e65de0 2024-03-19T01:17:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.039+0000 7fa692ffd640 1 -- 172.21.15.116:0/2118382096 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa698004460 con 0x55e279e65de0 2024-03-19T01:17:50.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.040+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2118382096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 msgr2=0x55e279e8fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.040+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/2118382096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e8fdd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa698052fd0 tx=0x7fa69802c330 comp rx=0 tx=0).stop 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.040+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2118382096 shutdown_connections 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.040+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/2118382096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e73480 0x55e279e64560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.040+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/2118382096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e8fdd0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.040+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/2118382096 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e65480 0x55e279e64020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.040+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2118382096 >> 172.21.15.116:0/2118382096 conn(0x55e279e61c30 msgr2=0x55e279e88890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.040+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2118382096 shutdown_connections 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.040+0000 7fa6b25b3740 1 -- 172.21.15.116:0/2118382096 wait complete. 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.040+0000 7fa6b25b3740 1 -- start start 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.040+0000 7fa6b25b3740 1 Processor -- start 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6b25b3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e65480 0x55e279e8d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6b25b3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e8d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6b25b3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e73480 0x55e279e8e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.045 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6b25b3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a819f770 con 0x55e279e65de0 2024-03-19T01:17:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6ad762640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e65480 0x55e279e8d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6b25b3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a819f140 con 0x55e279e65480 2024-03-19T01:17:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6ad762640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e65480 0x55e279e8d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:48736/0 (socket says 172.21.15.116:48736) 2024-03-19T01:17:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6b25b3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a8070b80 con 0x55e279e73480 2024-03-19T01:17:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6ad762640 1 -- 172.21.15.116:0/3584277264 learned_addr learned my addr 172.21.15.116:0/3584277264 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:17:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6b25b3740 10 client.0 _ll_register_callbacks cb 0x7fa6a000b340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:17:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6acf61640 1 --2- 172.21.15.116:0/3584277264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e8d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6adf63640 1 --2- 172.21.15.116:0/3584277264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e73480 0x55e279e8e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6ad762640 1 -- 172.21.15.116:0/3584277264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e73480 msgr2=0x55e279e8e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6ad762640 1 --2- 172.21.15.116:0/3584277264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e73480 0x55e279e8e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6ad762640 1 -- 172.21.15.116:0/3584277264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 msgr2=0x55e279e8d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:17:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6ad762640 1 --2- 172.21.15.116:0/3584277264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e8d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.041+0000 7fa6ad762640 1 -- 172.21.15.116:0/3584277264 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e279baa620 con 0x55e279e65480 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa6ad762640 1 --2- 172.21.15.116:0/3584277264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e65480 0x55e279e8d150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40ccac0 tx=0x7fa6a40cf300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db3d919de153630c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa690ff9640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa690ff9640 1 -- 172.21.15.116:0/3584277264 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a409fab0 con 0x55e279e65480 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa690ff9640 1 -- 172.21.15.116:0/3584277264 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fa6a4099c60 con 0x55e279e65480 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa6b25b3740 1 -- 172.21.15.116:0/3584277264 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa698009520 con 0x55e279e65480 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa690ff9640 1 -- 172.21.15.116:0/3584277264 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fa6a40a4100 con 0x55e279e65480 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa6b25b3740 10 client.25568 fetch_fsmap learned FSMap version 43 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa6b25b3740 1 -- 172.21.15.116:0/3584277264 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55e279e8fea0 con 0x55e279e65480 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa690ff9640 1 -- 172.21.15.116:0/3584277264 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa698009520 con 0x55e279e65480 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa690ff9640 1 -- 172.21.15.116:0/3584277264 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7fa6a4020020 con 0x55e279e65480 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa6b25b3740 10 client.25568 fetch_fsmap finished waiting for FSMap version 43 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa6b25b3740 10 client.25568 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa6b25b3740 20 client.25568 populate_metadata read hostname 'smithi116' 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/3584277264 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e279efa9e0 0x55e279efce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:17:50.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa6b25b3740 4 client.25568 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:17:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.042+0000 7fa6b25b3740 1 -- 172.21.15.116:0/3584277264 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e279dd7120 con 0x55e279efa9e0 2024-03-19T01:17:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.043+0000 7fa6acf61640 1 --2- 172.21.15.116:0/3584277264 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e279efa9e0 0x55e279efce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:17:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.043+0000 7fa690ff9640 10 client.25568 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:50.048 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.043+0000 7fa6acf61640 1 --2- 172.21.15.116:0/3584277264 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e279efa9e0 0x55e279efce00 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff876d842a3761e server_cookie=d8d4561c5574e8a4 in_seq=0 out_seq=0 2024-03-19T01:17:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.051+0000 7fa690ff9640 1 -- 172.21.15.116:0/3584277264 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e279dd7120 con 0x55e279efa9e0 2024-03-19T01:17:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.051+0000 7fa690ff9640 10 client.25568 handle_command_reply: tid=0 2024-03-19T01:17:50.052 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.051+0000 7fa6b25b3740 1 client.25568 shutdown 2024-03-19T01:17:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.051+0000 7fa6b25b3740 1 -- 172.21.15.116:0/3584277264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e65480 msgr2=0x55e279e8d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.051+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/3584277264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e65480 0x55e279e8d150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40ccac0 tx=0x7fa6a40cf300 comp rx=0 tx=0).stop 2024-03-19T01:17:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.052+0000 7fa6b25b3740 1 -- 172.21.15.116:0/3584277264 shutdown_connections 2024-03-19T01:17:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.052+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/3584277264 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e279efa9e0 0x55e279efce00 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.052+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/3584277264 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e279e73480 0x55e279e8e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.052+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/3584277264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e279e65de0 0x55e279e8d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.052+0000 7fa6b25b3740 1 --2- 172.21.15.116:0/3584277264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e279e65480 0x55e279e8d150 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.052+0000 7fa690ff9640 0 client.25568 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:17:50.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.052+0000 7fa6b25b3740 1 -- 172.21.15.116:0/3584277264 >> 172.21.15.116:0/3584277264 conn(0x55e279e61c30 msgr2=0x55e279e5f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.052+0000 7fa6b25b3740 1 -- 172.21.15.116:0/3584277264 shutdown_connections 2024-03-19T01:17:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.052+0000 7fa6b25b3740 1 -- 172.21.15.116:0/3584277264 wait complete. 2024-03-19T01:17:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.052+0000 7fa6b25b3740 20 client.25568 trim_cache size 0 max 16384 2024-03-19T01:17:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:17:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:17:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:17:50.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:17:50.055 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.053+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3731584163 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa6940a2920 msgr2=0x7fa6940a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:50.055 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.053+0000 7fa6af9c6640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa6940a2920 0x7fa6940a4de0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000bd20 tx=0x7fa6a000b470 comp rx=0 tx=0).stop 2024-03-19T01:17:50.055 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.053+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3731584163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6a81009c0 msgr2=0x7fa6a8196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:17:50.055 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.053+0000 7fa6af9c6640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6a81009c0 0x7fa6a8196a00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4005fd0 tx=0x7fa6a400ef90 comp rx=0 tx=0).stop 2024-03-19T01:17:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.053+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3731584163 shutdown_connections 2024-03-19T01:17:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.053+0000 7fa6af9c6640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fa6a8101d20 0x7fa6a819dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.053+0000 7fa6af9c6640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fa6940a2920 0x7fa6940a4de0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.053+0000 7fa6af9c6640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa6a8101380 0x7fa6a8196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.053+0000 7fa6af9c6640 1 --2- 172.21.15.116:0/3731584163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6a81009c0 0x7fa6a8196a00 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:17:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.053+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3731584163 >> 172.21.15.116:0/3731584163 conn(0x7fa6a80fb470 msgr2=0x7fa6a8104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:17:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.053+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3731584163 shutdown_connections 2024-03-19T01:17:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:17:50.053+0000 7fa6af9c6640 1 -- 172.21.15.116:0/3731584163 wait complete. 2024-03-19T01:17:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 305.0 s, 2162.8 tps, lat 46.215 ms stddev 48.599 2024-03-19T01:17:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:51 smithi162 ceph-mon[67178]: pgmap v509: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.7 MiB/s wr, 674 op/s 2024-03-19T01:17:51.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:51 smithi116 ceph-mon[62701]: pgmap v509: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.7 MiB/s wr, 674 op/s 2024-03-19T01:17:51.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:51 smithi156 ceph-mon[66799]: pgmap v509: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.7 MiB/s wr, 674 op/s 2024-03-19T01:17:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:53 smithi162 ceph-mon[67178]: pgmap v510: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.3 MiB/s wr, 621 op/s 2024-03-19T01:17:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:53 smithi116 ceph-mon[62701]: pgmap v510: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.3 MiB/s wr, 621 op/s 2024-03-19T01:17:53.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:53 smithi156 ceph-mon[66799]: pgmap v510: 97 pgs: 97 active+clean; 1.6 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.3 MiB/s wr, 621 op/s 2024-03-19T01:17:55.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:55 smithi162 ceph-mon[67178]: pgmap v511: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.7 MiB/s wr, 750 op/s 2024-03-19T01:17:55.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:55 smithi116 ceph-mon[62701]: pgmap v511: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.7 MiB/s wr, 750 op/s 2024-03-19T01:17:55.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:55 smithi156 ceph-mon[66799]: pgmap v511: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.7 MiB/s wr, 750 op/s 2024-03-19T01:17:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 310.0 s, 2491.6 tps, lat 40.250 ms stddev 46.450 2024-03-19T01:17:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:57 smithi162 ceph-mon[67178]: pgmap v512: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 653 op/s 2024-03-19T01:17:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:57 smithi116 ceph-mon[62701]: pgmap v512: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 653 op/s 2024-03-19T01:17:57.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:57 smithi156 ceph-mon[66799]: pgmap v512: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.4 MiB/s wr, 653 op/s 2024-03-19T01:17:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:17:59 smithi162 ceph-mon[67178]: pgmap v513: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 710 op/s 2024-03-19T01:17:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:17:59 smithi116 ceph-mon[62701]: pgmap v513: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 710 op/s 2024-03-19T01:17:59.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:17:59 smithi156 ceph-mon[66799]: pgmap v513: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 710 op/s 2024-03-19T01:18:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 315.0 s, 2179.8 tps, lat 45.763 ms stddev 56.574 2024-03-19T01:18:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:01 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:18:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:01 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:18:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:01 smithi162 ceph-mon[67178]: pgmap v514: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.1 MiB/s wr, 698 op/s 2024-03-19T01:18:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:01 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:18:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:01 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:18:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:01 smithi116 ceph-mon[62701]: pgmap v514: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.1 MiB/s wr, 698 op/s 2024-03-19T01:18:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:01 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:18:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:01 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:18:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:01 smithi156 ceph-mon[66799]: pgmap v514: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.1 MiB/s wr, 698 op/s 2024-03-19T01:18:02.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:02 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:18:02.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:02 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:18:02.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:02 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:18:02.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:02 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:18:02.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:02 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:18:02.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:02 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:18:02.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:02 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:18:02.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:02 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:18:02.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:02 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:18:03.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.176+0000 7fd69e075640 1 -- 172.21.15.116:0/1540333165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd698102a70 msgr2=0x7fd698102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.176+0000 7fd69e075640 1 --2- 172.21.15.116:0/1540333165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd698102a70 0x7fd698102ef0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd688009900 tx=0x7fd68802bc50 comp rx=0 tx=0).stop 2024-03-19T01:18:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.176+0000 7fd69e075640 1 -- 172.21.15.116:0/1540333165 shutdown_connections 2024-03-19T01:18:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.176+0000 7fd69e075640 1 --2- 172.21.15.116:0/1540333165 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd698103430 0x7fd698109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.176+0000 7fd69e075640 1 --2- 172.21.15.116:0/1540333165 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd698102a70 0x7fd698102ef0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.176+0000 7fd69e075640 1 --2- 172.21.15.116:0/1540333165 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd698101880 0x7fd698101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.176+0000 7fd69e075640 1 -- 172.21.15.116:0/1540333165 >> 172.21.15.116:0/1540333165 conn(0x7fd6980fcdf0 msgr2=0x7fd6980ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.176+0000 7fd69e075640 1 -- 172.21.15.116:0/1540333165 shutdown_connections 2024-03-19T01:18:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.176+0000 7fd69e075640 1 -- 172.21.15.116:0/1540333165 wait complete. 2024-03-19T01:18:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.177+0000 7fd69e075640 1 Processor -- start 2024-03-19T01:18:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.177+0000 7fd69e075640 1 -- start start 2024-03-19T01:18:03.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.177+0000 7fd69e075640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd698101880 0x7fd69819e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.177+0000 7fd69e075640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd698102a70 0x7fd69819e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.177+0000 7fd69e075640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd698103430 0x7fd6981a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.177+0000 7fd69e075640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd698075180 con 0x7fd698101880 2024-03-19T01:18:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.177+0000 7fd69e075640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd698075000 con 0x7fd698103430 2024-03-19T01:18:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.177+0000 7fd69e075640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd698075300 con 0x7fd698102a70 2024-03-19T01:18:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.177+0000 7fd6977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd698101880 0x7fd69819e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.177+0000 7fd697fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd698103430 0x7fd6981a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.177+0000 7fd696ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd698102a70 0x7fd69819e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.177+0000 7fd6977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd698101880 0x7fd69819e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60086/0 (socket says 172.21.15.116:60086) 2024-03-19T01:18:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.177+0000 7fd6977fe640 1 -- 172.21.15.116:0/2707026452 learned_addr learned my addr 172.21.15.116:0/2707026452 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.179+0000 7fd6977fe640 1 -- 172.21.15.116:0/2707026452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd698102a70 msgr2=0x7fd69819e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.179+0000 7fd6977fe640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd698102a70 0x7fd69819e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.179+0000 7fd6977fe640 1 -- 172.21.15.116:0/2707026452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd698103430 msgr2=0x7fd6981a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.179+0000 7fd6977fe640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd698103430 0x7fd6981a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.179+0000 7fd6977fe640 1 -- 172.21.15.116:0/2707026452 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6981a5ce0 con 0x7fd698101880 2024-03-19T01:18:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.179+0000 7fd697fff640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd698103430 0x7fd6981a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:18:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.179+0000 7fd696ffd640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd698102a70 0x7fd69819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:18:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.179+0000 7fd6977fe640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd698101880 0x7fd69819e0c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00e9c0 tx=0x7fd68c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.179+0000 7fd694ff9640 1 -- 172.21.15.116:0/2707026452 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68c00cd80 con 0x7fd698101880 2024-03-19T01:18:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.179+0000 7fd694ff9640 1 -- 172.21.15.116:0/2707026452 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd68c0035f0 con 0x7fd698101880 2024-03-19T01:18:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.179+0000 7fd694ff9640 1 -- 172.21.15.116:0/2707026452 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68c010690 con 0x7fd698101880 2024-03-19T01:18:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.180+0000 7fd69e075640 1 -- 172.21.15.116:0/2707026452 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6981a5fd0 con 0x7fd698101880 2024-03-19T01:18:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.180+0000 7fd69e075640 1 -- 172.21.15.116:0/2707026452 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6981a63e0 con 0x7fd698101880 2024-03-19T01:18:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd694ff9640 1 -- 172.21.15.116:0/2707026452 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fd68c0037a0 con 0x7fd698101880 2024-03-19T01:18:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd6a0c5b740 1 Processor -- start 2024-03-19T01:18:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd6a0c5b740 1 -- start start 2024-03-19T01:18:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd6a0c5b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5e4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd6a0c5b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5e95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd6a0c5b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e9c60 0x56369d5ec0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd6a0c5b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56369d548be0 con 0x56369d5e4af0 2024-03-19T01:18:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd6a0c5b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56369d4f1960 con 0x56369d5e4100 2024-03-19T01:18:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd6a0c5b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56369d544c80 con 0x56369d5e9c60 2024-03-19T01:18:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd6977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5e95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd696ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e9c60 0x56369d5ec0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd694ff9640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd6800a2970 0x7fd6800a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd697fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5e4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd6977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5e95f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60098/0 (socket says 172.21.15.116:60098) 2024-03-19T01:18:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.183+0000 7fd6977fe640 1 -- 172.21.15.116:0/760813644 learned_addr learned my addr 172.21.15.116:0/760813644 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.184+0000 7fd697fff640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd6800a2970 0x7fd6800a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.184+0000 7fd696ffd640 1 -- 172.21.15.116:0/760813644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 msgr2=0x56369d5e4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.184+0000 7fd694ff9640 1 -- 172.21.15.116:0/2707026452 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd68c0cef10 con 0x7fd698101880 2024-03-19T01:18:03.201 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.184+0000 7fd696ffd640 1 --2- 172.21.15.116:0/760813644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5e4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.184+0000 7fd696ffd640 1 -- 172.21.15.116:0/760813644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 msgr2=0x56369d5e95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.184+0000 7fd696ffd640 1 --2- 172.21.15.116:0/760813644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5e95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.184+0000 7fd696ffd640 1 -- 172.21.15.116:0/760813644 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56369d4e8d20 con 0x56369d5e9c60 2024-03-19T01:18:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.184+0000 7fd697fff640 1 --2- 172.21.15.116:0/760813644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5e4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:18:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.184+0000 7fd696ffd640 1 --2- 172.21.15.116:0/760813644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e9c60 0x56369d5ec0f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd688036040 tx=0x7fd688009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a07e4d5e061c28fd server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.185+0000 7fd67dffb640 1 -- 172.21.15.116:0/760813644 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd688043020 con 0x56369d5e9c60 2024-03-19T01:18:03.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.185+0000 7fd697fff640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd6800a2970 0x7fd6800a4e30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x56369d5e5350 tx=0x7fd690004ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.185+0000 7fd67dffb640 1 -- 172.21.15.116:0/760813644 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd688004040 con 0x56369d5e9c60 2024-03-19T01:18:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.185+0000 7fd67dffb640 1 -- 172.21.15.116:0/760813644 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd688005630 con 0x56369d5e9c60 2024-03-19T01:18:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.186+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/760813644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e9c60 msgr2=0x56369d5ec0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.186+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/760813644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e9c60 0x56369d5ec0f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd688036040 tx=0x7fd688009400 comp rx=0 tx=0).stop 2024-03-19T01:18:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.186+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/760813644 shutdown_connections 2024-03-19T01:18:03.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.186+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/760813644 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e9c60 0x56369d5ec0f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.186+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/760813644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5e95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.186+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/760813644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5e4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.186+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/760813644 >> 172.21.15.116:0/760813644 conn(0x56369d5df670 msgr2=0x56369d5dfa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.186+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/760813644 shutdown_connections 2024-03-19T01:18:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.186+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/760813644 wait complete. 2024-03-19T01:18:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.186+0000 7fd6a0c5b740 1 -- start start 2024-03-19T01:18:03.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd6a0c5b740 1 Processor -- start 2024-03-19T01:18:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd6a0c5b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e4100 0x56369d5ee0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd6a0c5b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5ee630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd6a0c5b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e9c60 0x56369d5f5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd6a0c5b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56369d5ecbb0 con 0x56369d5e4af0 2024-03-19T01:18:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd6a0c5b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56369d504db0 con 0x56369d5e9c60 2024-03-19T01:18:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd6a0c5b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56369d5ecd30 con 0x56369d5e4100 2024-03-19T01:18:03.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd696ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd6977fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e4100 0x56369d5ee0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd697fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e9c60 0x56369d5f5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd6977fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e4100 0x56369d5ee0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51650/0 (socket says 172.21.15.116:51650) 2024-03-19T01:18:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd6977fe640 1 -- 172.21.15.116:0/105106627 learned_addr learned my addr 172.21.15.116:0/105106627 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd6a0c5b740 10 client.0 _ll_register_callbacks cb 0x56369d509e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:18:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd697fff640 1 -- 172.21.15.116:0/105106627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e4100 msgr2=0x56369d5ee0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:03.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd697fff640 1 --2- 172.21.15.116:0/105106627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e4100 0x56369d5ee0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd697fff640 1 -- 172.21.15.116:0/105106627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 msgr2=0x56369d5ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd697fff640 1 --2- 172.21.15.116:0/105106627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5ee630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd697fff640 1 -- 172.21.15.116:0/105106627 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56369d503c60 con 0x56369d5e9c60 2024-03-19T01:18:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.187+0000 7fd6977fe640 1 --2- 172.21.15.116:0/105106627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e4100 0x56369d5ee0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:18:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd697fff640 1 --2- 172.21.15.116:0/105106627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e9c60 0x56369d5f5670 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd690011bd0 tx=0x7fd690004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ca9b2317f3e2c0b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd676ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:18:03.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd676ffd640 1 -- 172.21.15.116:0/105106627 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd690019070 con 0x56369d5e9c60 2024-03-19T01:18:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/105106627 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd688004040 con 0x56369d5e9c60 2024-03-19T01:18:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd676ffd640 1 -- 172.21.15.116:0/105106627 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd690003970 con 0x56369d5e9c60 2024-03-19T01:18:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd676ffd640 1 -- 172.21.15.116:0/105106627 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fd690003b70 con 0x56369d5e9c60 2024-03-19T01:18:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd6a0c5b740 10 client.25577 fetch_fsmap learned FSMap version 44 2024-03-19T01:18:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/105106627 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x56369d25d210 con 0x56369d5e9c60 2024-03-19T01:18:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd676ffd640 1 -- 172.21.15.116:0/105106627 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd688004040 con 0x56369d5e9c60 2024-03-19T01:18:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd676ffd640 1 -- 172.21.15.116:0/105106627 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7fd690020430 con 0x56369d5e9c60 2024-03-19T01:18:03.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd6a0c5b740 10 client.25577 fetch_fsmap finished waiting for FSMap version 44 2024-03-19T01:18:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd6a0c5b740 10 client.25577 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:18:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd6a0c5b740 20 client.25577 populate_metadata read hostname 'smithi116' 2024-03-19T01:18:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/105106627 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56369d5f9fd0 0x56369d5fc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd6a0c5b740 4 client.25577 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:18:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/105106627 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56369d556120 con 0x56369d5f9fd0 2024-03-19T01:18:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.188+0000 7fd6977fe640 1 --2- 172.21.15.116:0/105106627 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56369d5f9fd0 0x56369d5fc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.189+0000 7fd6977fe640 1 --2- 172.21.15.116:0/105106627 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56369d5f9fd0 0x56369d5fc460 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a8fe6ca90b41bff server_cookie=fba469585f729157 in_seq=0 out_seq=0 2024-03-19T01:18:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.189+0000 7fd676ffd640 10 client.25577 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd676ffd640 1 -- 172.21.15.116:0/105106627 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56369d556120 con 0x56369d5f9fd0 2024-03-19T01:18:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd676ffd640 10 client.25577 handle_command_reply: tid=0 2024-03-19T01:18:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd6a0c5b740 1 client.25577 shutdown 2024-03-19T01:18:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/105106627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e9c60 msgr2=0x56369d5f5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:03.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/105106627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e9c60 0x56369d5f5670 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd690011bd0 tx=0x7fd690004a40 comp rx=0 tx=0).stop 2024-03-19T01:18:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/105106627 shutdown_connections 2024-03-19T01:18:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/105106627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e9c60 0x56369d5f5670 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/105106627 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/105106627 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56369d5f9fd0 0x56369d5fc460 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd676ffd640 0 client.25577 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/105106627 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e4100 0x56369d5ee0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/105106627 >> 172.21.15.116:0/105106627 conn(0x56369d5df670 msgr2=0x56369d5e08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:03.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/105106627 shutdown_connections 2024-03-19T01:18:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.190+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/105106627 wait complete. 2024-03-19T01:18:03.212 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.191+0000 7fd6a0c5b740 20 client.25577 trim_cache size 0 max 16384 2024-03-19T01:18:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.213+0000 7fd6a0c5b740 1 Processor -- start 2024-03-19T01:18:03.217 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.213+0000 7fd6a0c5b740 1 -- start start 2024-03-19T01:18:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6a0c5b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5de220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6a0c5b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e4af0 0x56369d5f5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6a0c5b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e9c60 0x56369d5de760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6a0c5b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd6981af440 con 0x56369d5e9c60 2024-03-19T01:18:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6a0c5b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd6981b0fb0 con 0x56369d5e4100 2024-03-19T01:18:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6a0c5b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd6981afb10 con 0x56369d5e4af0 2024-03-19T01:18:03.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6977fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5de220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd697fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e9c60 0x56369d5de760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6977fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5de220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:46404/0 (socket says 172.21.15.116:46404) 2024-03-19T01:18:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6977fe640 1 -- 172.21.15.116:0/532484756 learned_addr learned my addr 172.21.15.116:0/532484756 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd696ffd640 1 --2- 172.21.15.116:0/532484756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e4af0 0x56369d5f5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6977fe640 1 -- 172.21.15.116:0/532484756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e4af0 msgr2=0x56369d5f5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6977fe640 1 --2- 172.21.15.116:0/532484756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e4af0 0x56369d5f5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6977fe640 1 -- 172.21.15.116:0/532484756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e9c60 msgr2=0x56369d5de760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:03.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6977fe640 1 --2- 172.21.15.116:0/532484756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e9c60 0x56369d5de760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd6977fe640 1 -- 172.21.15.116:0/532484756 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56369d329620 con 0x56369d5e4100 2024-03-19T01:18:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.214+0000 7fd697fff640 1 --2- 172.21.15.116:0/532484756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e9c60 0x56369d5de760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:18:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.215+0000 7fd6977fe640 1 --2- 172.21.15.116:0/532484756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5de220 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd68c097c90 tx=0x7fd68c086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50793279c7de5e69 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.215+0000 7fd67d7fa640 1 -- 172.21.15.116:0/532484756 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68c0a3fd0 con 0x56369d5e4100 2024-03-19T01:18:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.215+0000 7fd67d7fa640 1 -- 172.21.15.116:0/532484756 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd68c0b3030 con 0x56369d5e4100 2024-03-19T01:18:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.215+0000 7fd67d7fa640 1 -- 172.21.15.116:0/532484756 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68c0860b0 con 0x56369d5e4100 2024-03-19T01:18:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.215+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/532484756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 msgr2=0x56369d5de220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:03.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.215+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/532484756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5de220 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd68c097c90 tx=0x7fd68c086040 comp rx=0 tx=0).stop 2024-03-19T01:18:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.215+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/532484756 shutdown_connections 2024-03-19T01:18:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.215+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/532484756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e9c60 0x56369d5de760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.215+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/532484756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e4af0 0x56369d5f5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.215+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/532484756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5de220 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.215+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/532484756 >> 172.21.15.116:0/532484756 conn(0x56369d5df670 msgr2=0x56369d5dfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.216+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/532484756 shutdown_connections 2024-03-19T01:18:03.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.216+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/532484756 wait complete. 2024-03-19T01:18:03.222 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.216+0000 7fd6a0c5b740 1 -- start start 2024-03-19T01:18:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.216+0000 7fd6a0c5b740 1 Processor -- start 2024-03-19T01:18:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.216+0000 7fd6a0c5b740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.216+0000 7fd6a0c5b740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5f35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.216+0000 7fd6a0c5b740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e9c60 0x56369d5f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.216+0000 7fd6a0c5b740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd6981b08e0 con 0x56369d5e4af0 2024-03-19T01:18:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.216+0000 7fd6a0c5b740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd6981b01e0 con 0x56369d5e4100 2024-03-19T01:18:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.216+0000 7fd6a0c5b740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd6981ad420 con 0x56369d5e9c60 2024-03-19T01:18:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.216+0000 7fd6a0c5b740 10 client.0 _ll_register_callbacks cb 0x7fd6900135c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:18:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.217+0000 7fd697fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e9c60 0x56369d5f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.217+0000 7fd696ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5f35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.217+0000 7fd697fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e9c60 0x56369d5f45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51672/0 (socket says 172.21.15.116:51672) 2024-03-19T01:18:03.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.217+0000 7fd697fff640 1 -- 172.21.15.116:0/188316542 learned_addr learned my addr 172.21.15.116:0/188316542 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.217+0000 7fd6977fe640 1 --2- 172.21.15.116:0/188316542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.217+0000 7fd696ffd640 1 -- 172.21.15.116:0/188316542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e9c60 msgr2=0x56369d5f45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.217+0000 7fd696ffd640 1 --2- 172.21.15.116:0/188316542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e9c60 0x56369d5f45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.217+0000 7fd696ffd640 1 -- 172.21.15.116:0/188316542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 msgr2=0x56369d5f30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.217+0000 7fd696ffd640 1 --2- 172.21.15.116:0/188316542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5f30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.217+0000 7fd696ffd640 1 -- 172.21.15.116:0/188316542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56369d5f4b90 con 0x56369d5e4af0 2024-03-19T01:18:03.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.217+0000 7fd696ffd640 1 --2- 172.21.15.116:0/188316542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5f35e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd688000c00 tx=0x7fd68802ce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7bd4fbadc8834a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.217+0000 7fd676ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:18:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.218+0000 7fd676ffd640 1 -- 172.21.15.116:0/188316542 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd688043040 con 0x56369d5e4af0 2024-03-19T01:18:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.218+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/188316542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd68c0b3030 con 0x56369d5e4af0 2024-03-19T01:18:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.218+0000 7fd676ffd640 1 -- 172.21.15.116:0/188316542 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd688005040 con 0x56369d5e4af0 2024-03-19T01:18:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.218+0000 7fd676ffd640 1 -- 172.21.15.116:0/188316542 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fd688005260 con 0x56369d5e4af0 2024-03-19T01:18:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.218+0000 7fd6a0c5b740 10 client.15750 fetch_fsmap learned FSMap version 44 2024-03-19T01:18:03.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.218+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/188316542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x56369d6603f0 con 0x56369d5e4af0 2024-03-19T01:18:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.218+0000 7fd676ffd640 1 -- 172.21.15.116:0/188316542 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd68c0b3030 con 0x56369d5e4af0 2024-03-19T01:18:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.218+0000 7fd676ffd640 1 -- 172.21.15.116:0/188316542 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7fd6880035f0 con 0x56369d5e4af0 2024-03-19T01:18:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.218+0000 7fd6a0c5b740 10 client.15750 fetch_fsmap finished waiting for FSMap version 44 2024-03-19T01:18:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.218+0000 7fd6a0c5b740 10 client.15750 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:18:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.218+0000 7fd6a0c5b740 20 client.15750 populate_metadata read hostname 'smithi116' 2024-03-19T01:18:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.219+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/188316542 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56369d660970 0x56369d662d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:03.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.219+0000 7fd6a0c5b740 4 client.15750 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:18:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.219+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/188316542 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56369d550470 con 0x56369d660970 2024-03-19T01:18:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.219+0000 7fd6977fe640 1 --2- 172.21.15.116:0/188316542 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56369d660970 0x56369d662d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.219+0000 7fd676ffd640 10 client.15750 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:03.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.219+0000 7fd6977fe640 1 --2- 172.21.15.116:0/188316542 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56369d660970 0x56369d662d90 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2293e00f2a67181a server_cookie=53d7ad7d1a2079bb in_seq=0 out_seq=0 2024-03-19T01:18:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.226+0000 7fd676ffd640 1 -- 172.21.15.116:0/188316542 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56369d550470 con 0x56369d660970 2024-03-19T01:18:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.226+0000 7fd676ffd640 10 client.15750 handle_command_reply: tid=0 2024-03-19T01:18:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.226+0000 7fd6a0c5b740 1 client.15750 shutdown 2024-03-19T01:18:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.226+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/188316542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 msgr2=0x56369d5f35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.227+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/188316542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5f35e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd688000c00 tx=0x7fd68802ce10 comp rx=0 tx=0).stop 2024-03-19T01:18:03.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.227+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/188316542 shutdown_connections 2024-03-19T01:18:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.227+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/188316542 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56369d660970 0x56369d662d90 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.227+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/188316542 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56369d5e9c60 0x56369d5f45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.227+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/188316542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56369d5e4af0 0x56369d5f35e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.227+0000 7fd6a0c5b740 1 --2- 172.21.15.116:0/188316542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56369d5e4100 0x56369d5f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.227+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/188316542 >> 172.21.15.116:0/188316542 conn(0x56369d5df670 msgr2=0x56369d5de220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.227+0000 7fd676ffd640 0 client.15750 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.227+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/188316542 shutdown_connections 2024-03-19T01:18:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.227+0000 7fd6a0c5b740 1 -- 172.21.15.116:0/188316542 wait complete. 2024-03-19T01:18:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.227+0000 7fd6a0c5b740 20 client.15750 trim_cache size 0 max 16384 2024-03-19T01:18:03.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:18:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:18:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:18:03.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:18:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.228+0000 7fd69e075640 1 -- 172.21.15.116:0/2707026452 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd6800a2970 msgr2=0x7fd6800a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.228+0000 7fd69e075640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd6800a2970 0x7fd6800a4e30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x56369d5e5350 tx=0x7fd690004ab0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.228+0000 7fd69e075640 1 -- 172.21.15.116:0/2707026452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd698101880 msgr2=0x7fd69819e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:03.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.228+0000 7fd69e075640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd698101880 0x7fd69819e0c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00e9c0 tx=0x7fd68c00ee80 comp rx=0 tx=0).stop 2024-03-19T01:18:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.229+0000 7fd69e075640 1 -- 172.21.15.116:0/2707026452 shutdown_connections 2024-03-19T01:18:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.229+0000 7fd69e075640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd6800a2970 0x7fd6800a4e30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.229+0000 7fd69e075640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd698103430 0x7fd6981a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.229+0000 7fd69e075640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd698102a70 0x7fd69819e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.229+0000 7fd69e075640 1 --2- 172.21.15.116:0/2707026452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd698101880 0x7fd69819e0c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.229+0000 7fd69e075640 1 -- 172.21.15.116:0/2707026452 >> 172.21.15.116:0/2707026452 conn(0x7fd6980fcdf0 msgr2=0x7fd69810dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.229+0000 7fd69e075640 1 -- 172.21.15.116:0/2707026452 shutdown_connections 2024-03-19T01:18:03.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:03.229+0000 7fd69e075640 1 -- 172.21.15.116:0/2707026452 wait complete. 2024-03-19T01:18:03.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:03 smithi162 ceph-mon[67178]: pgmap v515: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.0 MiB/s wr, 599 op/s 2024-03-19T01:18:03.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:03 smithi116 ceph-mon[62701]: pgmap v515: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.0 MiB/s wr, 599 op/s 2024-03-19T01:18:03.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:03 smithi156 ceph-mon[66799]: pgmap v515: 97 pgs: 97 active+clean; 1.6 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.0 MiB/s wr, 599 op/s 2024-03-19T01:18:05.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:05 smithi162 ceph-mon[67178]: pgmap v516: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 790 op/s 2024-03-19T01:18:05.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:05 smithi116 ceph-mon[62701]: pgmap v516: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 790 op/s 2024-03-19T01:18:05.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:05 smithi156 ceph-mon[66799]: pgmap v516: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.0 MiB/s wr, 790 op/s 2024-03-19T01:18:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 320.0 s, 3111.6 tps, lat 32.211 ms stddev 36.137 2024-03-19T01:18:07.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:07 smithi162 ceph-mon[67178]: pgmap v517: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.3 MiB/s wr, 654 op/s 2024-03-19T01:18:07.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:07 smithi116 ceph-mon[62701]: pgmap v517: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.3 MiB/s wr, 654 op/s 2024-03-19T01:18:07.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:07 smithi156 ceph-mon[66799]: pgmap v517: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 6.3 MiB/s wr, 654 op/s 2024-03-19T01:18:09.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:09 smithi162 ceph-mon[67178]: pgmap v518: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.5 MiB/s wr, 688 op/s 2024-03-19T01:18:09.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:09 smithi116 ceph-mon[62701]: pgmap v518: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.5 MiB/s wr, 688 op/s 2024-03-19T01:18:09.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:09 smithi156 ceph-mon[66799]: pgmap v518: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.5 MiB/s wr, 688 op/s 2024-03-19T01:18:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 325.0 s, 2852.6 tps, lat 35.095 ms stddev 40.033 2024-03-19T01:18:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:11 smithi162 ceph-mon[67178]: pgmap v519: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.8 MiB/s wr, 699 op/s 2024-03-19T01:18:11.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:11 smithi116 ceph-mon[62701]: pgmap v519: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.8 MiB/s wr, 699 op/s 2024-03-19T01:18:11.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:11 smithi156 ceph-mon[66799]: pgmap v519: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.8 MiB/s wr, 699 op/s 2024-03-19T01:18:13.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:13 smithi162 ceph-mon[67178]: pgmap v520: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.2 MiB/s wr, 644 op/s 2024-03-19T01:18:13.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:13 smithi116 ceph-mon[62701]: pgmap v520: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.2 MiB/s wr, 644 op/s 2024-03-19T01:18:13.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:13 smithi156 ceph-mon[66799]: pgmap v520: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.2 MiB/s wr, 644 op/s 2024-03-19T01:18:15.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:15 smithi162 ceph-mon[67178]: pgmap v521: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.1 MiB/s wr, 800 op/s 2024-03-19T01:18:15.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:15 smithi116 ceph-mon[62701]: pgmap v521: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.1 MiB/s wr, 800 op/s 2024-03-19T01:18:15.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:15 smithi156 ceph-mon[66799]: pgmap v521: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.1 MiB/s wr, 800 op/s 2024-03-19T01:18:16.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.348+0000 7fc705d77640 1 -- 172.21.15.116:0/2630655716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc70010def0 msgr2=0x7fc7001103b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:16.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.348+0000 7fc705d77640 1 --2- 172.21.15.116:0/2630655716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc70010def0 0x7fc7001103b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8005fd0 tx=0x7fc6f800aec0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.348+0000 7fc705d77640 1 -- 172.21.15.116:0/2630655716 shutdown_connections 2024-03-19T01:18:16.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.348+0000 7fc705d77640 1 --2- 172.21.15.116:0/2630655716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc70010def0 0x7fc7001103b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.349+0000 7fc705d77640 1 --2- 172.21.15.116:0/2630655716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc70010b450 0x7fc70010d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.349+0000 7fc705d77640 1 --2- 172.21.15.116:0/2630655716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc700108a90 0x7fc70010ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.350 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.349+0000 7fc705d77640 1 -- 172.21.15.116:0/2630655716 >> 172.21.15.116:0/2630655716 conn(0x7fc70006ea20 msgr2=0x7fc700070e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:16.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.349+0000 7fc705d77640 1 -- 172.21.15.116:0/2630655716 shutdown_connections 2024-03-19T01:18:16.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.349+0000 7fc705d77640 1 -- 172.21.15.116:0/2630655716 wait complete. 2024-03-19T01:18:16.351 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.349+0000 7fc705d77640 1 Processor -- start 2024-03-19T01:18:16.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc705d77640 1 -- start start 2024-03-19T01:18:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc705d77640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc700108a90 0x7fc70019e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc705d77640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc70010b450 0x7fc70019e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc705d77640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc70010def0 0x7fc7001a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc705d77640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc700113ad0 con 0x7fc70010b450 2024-03-19T01:18:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc705d77640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc700113950 con 0x7fc700108a90 2024-03-19T01:18:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc705d77640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc700113c50 con 0x7fc70010def0 2024-03-19T01:18:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc6feffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc70010b450 0x7fc70019e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc6feffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc70010b450 0x7fc70019e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57252/0 (socket says 172.21.15.116:57252) 2024-03-19T01:18:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc6feffd640 1 -- 172.21.15.116:0/2826802748 learned_addr learned my addr 172.21.15.116:0/2826802748 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc6ff7fe640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc700108a90 0x7fc70019e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc6fffff640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc70010def0 0x7fc7001a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc6feffd640 1 -- 172.21.15.116:0/2826802748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc70010def0 msgr2=0x7fc7001a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc6feffd640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc70010def0 0x7fc7001a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc6feffd640 1 -- 172.21.15.116:0/2826802748 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc700108a90 msgr2=0x7fc70019e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc6feffd640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc700108a90 0x7fc70019e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc6feffd640 1 -- 172.21.15.116:0/2826802748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7001a5c80 con 0x7fc70010b450 2024-03-19T01:18:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.350+0000 7fc6fffff640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc70010def0 0x7fc7001a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:18:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.351+0000 7fc6feffd640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc70010b450 0x7fc70019e5a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400bc20 tx=0x7fc6f4008600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.351+0000 7fc6fcff9640 1 -- 172.21.15.116:0/2826802748 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4008c90 con 0x7fc70010b450 2024-03-19T01:18:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.351+0000 7fc6fcff9640 1 -- 172.21.15.116:0/2826802748 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6f4008e30 con 0x7fc70010b450 2024-03-19T01:18:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.351+0000 7fc6fcff9640 1 -- 172.21.15.116:0/2826802748 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4004d90 con 0x7fc70010b450 2024-03-19T01:18:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.351+0000 7fc705d77640 1 -- 172.21.15.116:0/2826802748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7001a5f70 con 0x7fc70010b450 2024-03-19T01:18:16.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.352+0000 7fc705d77640 1 -- 172.21.15.116:0/2826802748 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7001a63e0 con 0x7fc70010b450 2024-03-19T01:18:16.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.354+0000 7fc708967740 1 Processor -- start 2024-03-19T01:18:16.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.354+0000 7fc708967740 1 -- start start 2024-03-19T01:18:16.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.354+0000 7fc708967740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86100 0x558282d86520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.354+0000 7fc708967740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86af0 0x558282d8b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.354+0000 7fc708967740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d8e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.354+0000 7fc708967740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558282ceabe0 con 0x558282d86100 2024-03-19T01:18:16.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.354+0000 7fc708967740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558282c93960 con 0x558282d8bc60 2024-03-19T01:18:16.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.354+0000 7fc708967740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558282ce6c80 con 0x558282d86af0 2024-03-19T01:18:16.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6fcff9640 1 -- 172.21.15.116:0/2826802748 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fc6f401c020 con 0x7fc70010b450 2024-03-19T01:18:16.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6fffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86100 0x558282d86520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6ff7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86af0 0x558282d8b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6ff7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86af0 0x558282d8b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51216/0 (socket says 172.21.15.116:51216) 2024-03-19T01:18:16.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6ff7fe640 1 -- 172.21.15.116:0/3643200947 learned_addr learned my addr 172.21.15.116:0/3643200947 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:16.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6feffd640 1 --2- 172.21.15.116:0/3643200947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d8e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6feffd640 1 -- 172.21.15.116:0/3643200947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86af0 msgr2=0x558282d8b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6feffd640 1 --2- 172.21.15.116:0/3643200947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86af0 0x558282d8b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6feffd640 1 -- 172.21.15.116:0/3643200947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86100 msgr2=0x558282d86520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6feffd640 1 --2- 172.21.15.116:0/3643200947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86100 0x558282d86520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6feffd640 1 -- 172.21.15.116:0/3643200947 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558282c8ad20 con 0x558282d8bc60 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6fcff9640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6e80a2a10 0x7fc6e80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6feffd640 1 --2- 172.21.15.116:0/3643200947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d8e0f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40cba50 tx=0x7fc6f40d6640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24eb3174064b50b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6fcff9640 1 -- 172.21.15.116:0/2826802748 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc6f40cdff0 con 0x7fc70010b450 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.355+0000 7fc6fffff640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6e80a2a10 0x7fc6e80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.356+0000 7fc6e5ffb640 1 -- 172.21.15.116:0/3643200947 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4096bf0 con 0x558282d8bc60 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.356+0000 7fc6e5ffb640 1 -- 172.21.15.116:0/3643200947 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6f40d6e40 con 0x558282d8bc60 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.356+0000 7fc6e5ffb640 1 -- 172.21.15.116:0/3643200947 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f409a2b0 con 0x558282d8bc60 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.356+0000 7fc708967740 1 -- 172.21.15.116:0/3643200947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 msgr2=0x558282d8e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.356+0000 7fc708967740 1 --2- 172.21.15.116:0/3643200947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d8e0f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40cba50 tx=0x7fc6f40d6640 comp rx=0 tx=0).stop 2024-03-19T01:18:16.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.356+0000 7fc6fffff640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6e80a2a10 0x7fc6e80a4ed0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc6f802dab0 tx=0x7fc6f8038040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:16.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.357+0000 7fc708967740 1 -- 172.21.15.116:0/3643200947 shutdown_connections 2024-03-19T01:18:16.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.357+0000 7fc708967740 1 --2- 172.21.15.116:0/3643200947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d8e0f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.357+0000 7fc708967740 1 --2- 172.21.15.116:0/3643200947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86af0 0x558282d8b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.357+0000 7fc708967740 1 --2- 172.21.15.116:0/3643200947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86100 0x558282d86520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.357+0000 7fc708967740 1 -- 172.21.15.116:0/3643200947 >> 172.21.15.116:0/3643200947 conn(0x558282d81670 msgr2=0x558282d81a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:16.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.357+0000 7fc708967740 1 -- 172.21.15.116:0/3643200947 shutdown_connections 2024-03-19T01:18:16.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.357+0000 7fc708967740 1 -- 172.21.15.116:0/3643200947 wait complete. 2024-03-19T01:18:16.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.357+0000 7fc708967740 1 -- start start 2024-03-19T01:18:16.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.358+0000 7fc708967740 1 Processor -- start 2024-03-19T01:18:16.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.358+0000 7fc708967740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86100 0x558282d90110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.358+0000 7fc708967740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86af0 0x558282d90650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.358+0000 7fc708967740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d97690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.358+0000 7fc708967740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558282d8ec10 con 0x558282d86100 2024-03-19T01:18:16.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.358+0000 7fc708967740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558282ca6db0 con 0x558282d8bc60 2024-03-19T01:18:16.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.358+0000 7fc708967740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558282d8ed90 con 0x558282d86af0 2024-03-19T01:18:16.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.358+0000 7fc708967740 10 client.0 _ll_register_callbacks cb 0x558282cabe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:18:16.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.358+0000 7fc6fffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d97690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.358+0000 7fc6fffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d97690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:34658/0 (socket says 172.21.15.116:34658) 2024-03-19T01:18:16.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.358+0000 7fc6fffff640 1 -- 172.21.15.116:0/96862987 learned_addr learned my addr 172.21.15.116:0/96862987 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:16.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.358+0000 7fc6feffd640 1 --2- 172.21.15.116:0/96862987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86af0 0x558282d90650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.359+0000 7fc6fffff640 1 -- 172.21.15.116:0/96862987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86af0 msgr2=0x558282d90650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:16.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.359+0000 7fc6fffff640 1 --2- 172.21.15.116:0/96862987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86af0 0x558282d90650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.359+0000 7fc6fffff640 1 -- 172.21.15.116:0/96862987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86100 msgr2=0x558282d90110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.359+0000 7fc6fffff640 1 --2- 172.21.15.116:0/96862987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86100 0x558282d90110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.359+0000 7fc6fffff640 1 -- 172.21.15.116:0/96862987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558282ca5c60 con 0x558282d8bc60 2024-03-19T01:18:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.360+0000 7fc6fffff640 1 --2- 172.21.15.116:0/96862987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d97690 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8005e30 tx=0x7fc6f800aec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=848b81dc33c97118 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.360+0000 7fc6deffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:18:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.360+0000 7fc6deffd640 1 -- 172.21.15.116:0/96862987 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8004450 con 0x558282d8bc60 2024-03-19T01:18:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.360+0000 7fc708967740 1 -- 172.21.15.116:0/96862987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558282cf8120 con 0x558282d8bc60 2024-03-19T01:18:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.360+0000 7fc6deffd640 1 -- 172.21.15.116:0/96862987 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc6f80045f0 con 0x558282d8bc60 2024-03-19T01:18:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.361+0000 7fc6deffd640 1 -- 172.21.15.116:0/96862987 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fc6f80047f0 con 0x558282d8bc60 2024-03-19T01:18:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.361+0000 7fc708967740 10 client.25595 fetch_fsmap learned FSMap version 44 2024-03-19T01:18:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.361+0000 7fc708967740 1 -- 172.21.15.116:0/96862987 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5582829ff210 con 0x558282d8bc60 2024-03-19T01:18:16.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.361+0000 7fc6deffd640 1 -- 172.21.15.116:0/96862987 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558282cf8120 con 0x558282d8bc60 2024-03-19T01:18:16.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.361+0000 7fc6deffd640 1 -- 172.21.15.116:0/96862987 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7fc6f8056030 con 0x558282d8bc60 2024-03-19T01:18:16.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.362+0000 7fc708967740 10 client.25595 fetch_fsmap finished waiting for FSMap version 44 2024-03-19T01:18:16.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.362+0000 7fc708967740 10 client.25595 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:18:16.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.362+0000 7fc708967740 20 client.25595 populate_metadata read hostname 'smithi116' 2024-03-19T01:18:16.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.362+0000 7fc708967740 1 --2- 172.21.15.116:0/96862987 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558282d9c3c0 0x558282d9e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.362+0000 7fc708967740 4 client.25595 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:18:16.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.362+0000 7fc708967740 1 -- 172.21.15.116:0/96862987 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558282cf2470 con 0x558282d9c3c0 2024-03-19T01:18:16.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.362+0000 7fc6ff7fe640 1 --2- 172.21.15.116:0/96862987 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558282d9c3c0 0x558282d9e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.362+0000 7fc6deffd640 10 client.25595 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.362+0000 7fc6ff7fe640 1 --2- 172.21.15.116:0/96862987 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558282d9c3c0 0x558282d9e880 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d79e10ba364f3d4 server_cookie=5f867655eb6d9f71 in_seq=0 out_seq=0 2024-03-19T01:18:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.363+0000 7fc6deffd640 1 -- 172.21.15.116:0/96862987 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558282cf2470 con 0x558282d9c3c0 2024-03-19T01:18:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.363+0000 7fc6deffd640 10 client.25595 handle_command_reply: tid=0 2024-03-19T01:18:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.363+0000 7fc708967740 1 client.25595 shutdown 2024-03-19T01:18:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.363+0000 7fc708967740 1 -- 172.21.15.116:0/96862987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 msgr2=0x558282d97690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.363+0000 7fc708967740 1 --2- 172.21.15.116:0/96862987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d97690 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc6f8005e30 tx=0x7fc6f800aec0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.363+0000 7fc708967740 1 -- 172.21.15.116:0/96862987 shutdown_connections 2024-03-19T01:18:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.363+0000 7fc708967740 1 --2- 172.21.15.116:0/96862987 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d97690 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.363+0000 7fc708967740 1 --2- 172.21.15.116:0/96862987 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558282d9c3c0 0x558282d9e880 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.363+0000 7fc708967740 1 --2- 172.21.15.116:0/96862987 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86af0 0x558282d90650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.363+0000 7fc708967740 1 --2- 172.21.15.116:0/96862987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86100 0x558282d90110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.363+0000 7fc708967740 1 -- 172.21.15.116:0/96862987 >> 172.21.15.116:0/96862987 conn(0x558282d81670 msgr2=0x558282d828c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:16.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.363+0000 7fc6deffd640 0 client.25595 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:16.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.363+0000 7fc708967740 1 -- 172.21.15.116:0/96862987 shutdown_connections 2024-03-19T01:18:16.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.364+0000 7fc708967740 1 -- 172.21.15.116:0/96862987 wait complete. 2024-03-19T01:18:16.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.364+0000 7fc708967740 20 client.25595 trim_cache size 0 max 16384 2024-03-19T01:18:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.390+0000 7fc708967740 1 Processor -- start 2024-03-19T01:18:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.390+0000 7fc708967740 1 -- start start 2024-03-19T01:18:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.390+0000 7fc708967740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86100 0x558282d80930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.390+0000 7fc708967740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86af0 0x558282d97650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.390+0000 7fc708967740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d87590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.390+0000 7fc708967740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc7001af490 con 0x558282d86af0 2024-03-19T01:18:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.390+0000 7fc708967740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc7001b1000 con 0x558282d8bc60 2024-03-19T01:18:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.390+0000 7fc708967740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc7001afb60 con 0x558282d86100 2024-03-19T01:18:16.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.392+0000 7fc6fffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d87590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.392+0000 7fc6fffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d87590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:34662/0 (socket says 172.21.15.116:34662) 2024-03-19T01:18:16.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.392+0000 7fc6fffff640 1 -- 172.21.15.116:0/3651710936 learned_addr learned my addr 172.21.15.116:0/3651710936 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:16.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.394+0000 7fc6feffd640 1 --2- 172.21.15.116:0/3651710936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86af0 0x558282d97650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.397+0000 7fc6feffd640 1 -- 172.21.15.116:0/3651710936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86100 msgr2=0x558282d80930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:18:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.397+0000 7fc6feffd640 1 --2- 172.21.15.116:0/3651710936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86100 0x558282d80930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.397+0000 7fc6feffd640 1 -- 172.21.15.116:0/3651710936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 msgr2=0x558282d87590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.397+0000 7fc6feffd640 1 --2- 172.21.15.116:0/3651710936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d87590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.397+0000 7fc6feffd640 1 -- 172.21.15.116:0/3651710936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558282acb620 con 0x558282d86af0 2024-03-19T01:18:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.397+0000 7fc6feffd640 1 --2- 172.21.15.116:0/3651710936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86af0 0x558282d97650 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4004850 tx=0x7fc6f40ce270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad143f6b6461c18 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.399+0000 7fc6e57fa640 1 -- 172.21.15.116:0/3651710936 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f409a490 con 0x558282d86af0 2024-03-19T01:18:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.399+0000 7fc6e57fa640 1 -- 172.21.15.116:0/3651710936 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6f409b050 con 0x558282d86af0 2024-03-19T01:18:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.399+0000 7fc6e57fa640 1 -- 172.21.15.116:0/3651710936 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f409a720 con 0x558282d86af0 2024-03-19T01:18:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.400+0000 7fc708967740 1 -- 172.21.15.116:0/3651710936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86af0 msgr2=0x558282d97650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.400+0000 7fc708967740 1 --2- 172.21.15.116:0/3651710936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86af0 0x558282d97650 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4004850 tx=0x7fc6f40ce270 comp rx=0 tx=0).stop 2024-03-19T01:18:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.403+0000 7fc708967740 1 -- 172.21.15.116:0/3651710936 shutdown_connections 2024-03-19T01:18:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.404+0000 7fc708967740 1 --2- 172.21.15.116:0/3651710936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d8bc60 0x558282d87590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.404+0000 7fc708967740 1 --2- 172.21.15.116:0/3651710936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d86af0 0x558282d97650 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.404+0000 7fc708967740 1 --2- 172.21.15.116:0/3651710936 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86100 0x558282d80930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.404+0000 7fc708967740 1 -- 172.21.15.116:0/3651710936 >> 172.21.15.116:0/3651710936 conn(0x558282d81670 msgr2=0x558282d80220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.412+0000 7fc708967740 1 -- 172.21.15.116:0/3651710936 shutdown_connections 2024-03-19T01:18:16.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.412+0000 7fc708967740 1 -- 172.21.15.116:0/3651710936 wait complete. 2024-03-19T01:18:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.413+0000 7fc708967740 1 -- start start 2024-03-19T01:18:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.413+0000 7fc708967740 1 Processor -- start 2024-03-19T01:18:16.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.413+0000 7fc708967740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86100 0x558282d94a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.413+0000 7fc708967740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d86af0 0x558282d94fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.413+0000 7fc708967740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d8bc60 0x558282d95fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.413+0000 7fc708967740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc7001b0930 con 0x558282d8bc60 2024-03-19T01:18:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.413+0000 7fc708967740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc7001b0230 con 0x558282d86af0 2024-03-19T01:18:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.413+0000 7fc708967740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc7001ad470 con 0x558282d86100 2024-03-19T01:18:16.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.413+0000 7fc708967740 10 client.0 _ll_register_callbacks cb 0x7fc6f80086c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:18:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.414+0000 7fc6feffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d86af0 0x558282d94fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.414+0000 7fc6feffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d86af0 0x558282d94fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:34664/0 (socket says 172.21.15.116:34664) 2024-03-19T01:18:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.414+0000 7fc6feffd640 1 -- 172.21.15.116:0/3968970080 learned_addr learned my addr 172.21.15.116:0/3968970080 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.414+0000 7fc6fffff640 1 --2- 172.21.15.116:0/3968970080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d8bc60 0x558282d95fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.414+0000 7fc6ff7fe640 1 --2- 172.21.15.116:0/3968970080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86100 0x558282d94a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.414+0000 7fc6fffff640 1 -- 172.21.15.116:0/3968970080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86100 msgr2=0x558282d94a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.414+0000 7fc6fffff640 1 --2- 172.21.15.116:0/3968970080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86100 0x558282d94a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.414+0000 7fc6fffff640 1 -- 172.21.15.116:0/3968970080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d86af0 msgr2=0x558282d94fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.414+0000 7fc6fffff640 1 --2- 172.21.15.116:0/3968970080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d86af0 0x558282d94fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.414+0000 7fc6fffff640 1 -- 172.21.15.116:0/3968970080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558282d965a0 con 0x558282d8bc60 2024-03-19T01:18:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.415+0000 7fc6fffff640 1 --2- 172.21.15.116:0/3968970080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d8bc60 0x558282d95fb0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x558282d97c50 tx=0x7fc6f8036ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0dd15c6620ff38d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.415+0000 7fc6deffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:18:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.416+0000 7fc6deffd640 1 -- 172.21.15.116:0/3968970080 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8005bd0 con 0x558282d8bc60 2024-03-19T01:18:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.416+0000 7fc6deffd640 1 -- 172.21.15.116:0/3968970080 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc6f8036dc0 con 0x558282d8bc60 2024-03-19T01:18:16.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.416+0000 7fc708967740 1 -- 172.21.15.116:0/3968970080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6f409b050 con 0x558282d8bc60 2024-03-19T01:18:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.418+0000 7fc6deffd640 1 -- 172.21.15.116:0/3968970080 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fc6f800a050 con 0x558282d8bc60 2024-03-19T01:18:16.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.418+0000 7fc708967740 10 client.15765 fetch_fsmap learned FSMap version 44 2024-03-19T01:18:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.418+0000 7fc708967740 1 -- 172.21.15.116:0/3968970080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x558282d97920 con 0x558282d8bc60 2024-03-19T01:18:16.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.418+0000 7fc6deffd640 1 -- 172.21.15.116:0/3968970080 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6f409b050 con 0x558282d8bc60 2024-03-19T01:18:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.419+0000 7fc6deffd640 1 -- 172.21.15.116:0/3968970080 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 44) ==== 8358+0+0 (secure 0 0 0) 0x7fc6f8056040 con 0x558282d8bc60 2024-03-19T01:18:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.419+0000 7fc708967740 10 client.15765 fetch_fsmap finished waiting for FSMap version 44 2024-03-19T01:18:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.419+0000 7fc708967740 10 client.15765 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:18:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.419+0000 7fc708967740 20 client.15765 populate_metadata read hostname 'smithi116' 2024-03-19T01:18:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.419+0000 7fc708967740 1 --2- 172.21.15.116:0/3968970080 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558282e02e60 0x558282e05280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.419+0000 7fc708967740 4 client.15765 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:18:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.419+0000 7fc708967740 1 -- 172.21.15.116:0/3968970080 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558282c70770 con 0x558282e02e60 2024-03-19T01:18:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.419+0000 7fc6ff7fe640 1 --2- 172.21.15.116:0/3968970080 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558282e02e60 0x558282e05280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.420+0000 7fc6deffd640 10 client.15765 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:16.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.420+0000 7fc6ff7fe640 1 --2- 172.21.15.116:0/3968970080 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558282e02e60 0x558282e05280 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95f6fa57bb81cacf server_cookie=b1935c90bcf9b841 in_seq=0 out_seq=0 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.430+0000 7fc6deffd640 1 -- 172.21.15.116:0/3968970080 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558282c70770 con 0x558282e02e60 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.430+0000 7fc6deffd640 10 client.15765 handle_command_reply: tid=0 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.430+0000 7fc708967740 1 client.15765 shutdown 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.430+0000 7fc708967740 1 -- 172.21.15.116:0/3968970080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d8bc60 msgr2=0x558282d95fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.430+0000 7fc708967740 1 --2- 172.21.15.116:0/3968970080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d8bc60 0x558282d95fb0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x558282d97c50 tx=0x7fc6f8036ac0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.431+0000 7fc708967740 1 -- 172.21.15.116:0/3968970080 shutdown_connections 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.431+0000 7fc708967740 1 --2- 172.21.15.116:0/3968970080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558282d8bc60 0x558282d95fb0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.431+0000 7fc708967740 1 --2- 172.21.15.116:0/3968970080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558282d86af0 0x558282d94fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.431+0000 7fc708967740 1 --2- 172.21.15.116:0/3968970080 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558282e02e60 0x558282e05280 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.431+0000 7fc708967740 1 --2- 172.21.15.116:0/3968970080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558282d86100 0x558282d94a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.431+0000 7fc708967740 1 -- 172.21.15.116:0/3968970080 >> 172.21.15.116:0/3968970080 conn(0x558282d81670 msgr2=0x558282d848d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.431+0000 7fc6deffd640 0 client.15765 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.431+0000 7fc708967740 1 -- 172.21.15.116:0/3968970080 shutdown_connections 2024-03-19T01:18:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.431+0000 7fc708967740 1 -- 172.21.15.116:0/3968970080 wait complete. 2024-03-19T01:18:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.431+0000 7fc708967740 20 client.15765 trim_cache size 0 max 16384 2024-03-19T01:18:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:18:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:18:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:18:16.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:18:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.432+0000 7fc705d77640 1 -- 172.21.15.116:0/2826802748 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6e80a2a10 msgr2=0x7fc6e80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.432+0000 7fc705d77640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6e80a2a10 0x7fc6e80a4ed0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc6f802dab0 tx=0x7fc6f8038040 comp rx=0 tx=0).stop 2024-03-19T01:18:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.432+0000 7fc705d77640 1 -- 172.21.15.116:0/2826802748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc70010b450 msgr2=0x7fc70019e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.432+0000 7fc705d77640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc70010b450 0x7fc70019e5a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400bc20 tx=0x7fc6f4008600 comp rx=0 tx=0).stop 2024-03-19T01:18:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.432+0000 7fc705d77640 1 -- 172.21.15.116:0/2826802748 shutdown_connections 2024-03-19T01:18:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.432+0000 7fc705d77640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc70010def0 0x7fc7001a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.432+0000 7fc705d77640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc70010b450 0x7fc70019e5a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.433+0000 7fc705d77640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc6e80a2a10 0x7fc6e80a4ed0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.433+0000 7fc705d77640 1 --2- 172.21.15.116:0/2826802748 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc700108a90 0x7fc70019e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.433+0000 7fc705d77640 1 -- 172.21.15.116:0/2826802748 >> 172.21.15.116:0/2826802748 conn(0x7fc70006ea20 msgr2=0x7fc7001093d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.433+0000 7fc705d77640 1 -- 172.21.15.116:0/2826802748 shutdown_connections 2024-03-19T01:18:16.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:16.433+0000 7fc705d77640 1 -- 172.21.15.116:0/2826802748 wait complete. 2024-03-19T01:18:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 330.0 s, 1898.8 tps, lat 52.470 ms stddev 54.687 2024-03-19T01:18:17.651 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:17 smithi162 ceph-mon[67178]: pgmap v522: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.8 MiB/s wr, 673 op/s 2024-03-19T01:18:17.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:17 smithi116 ceph-mon[62701]: pgmap v522: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.8 MiB/s wr, 673 op/s 2024-03-19T01:18:17.727 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:17 smithi156 ceph-mon[66799]: pgmap v522: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.8 MiB/s wr, 673 op/s 2024-03-19T01:18:19.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:19 smithi162 ceph-mon[67178]: pgmap v523: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.1 MiB/s wr, 709 op/s 2024-03-19T01:18:19.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:19 smithi116 ceph-mon[62701]: pgmap v523: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.1 MiB/s wr, 709 op/s 2024-03-19T01:18:19.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:19 smithi156 ceph-mon[66799]: pgmap v523: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.1 MiB/s wr, 709 op/s 2024-03-19T01:18:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 335.0 s, 1681.6 tps, lat 59.468 ms stddev 60.655 2024-03-19T01:18:21.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:21 smithi162 ceph-mon[67178]: pgmap v524: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.5 MiB/s wr, 661 op/s 2024-03-19T01:18:21.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:21 smithi116 ceph-mon[62701]: pgmap v524: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.5 MiB/s wr, 661 op/s 2024-03-19T01:18:21.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:21 smithi156 ceph-mon[66799]: pgmap v524: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.5 MiB/s wr, 661 op/s 2024-03-19T01:18:23.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:23 smithi162 ceph-mon[67178]: pgmap v525: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.7 MiB/s wr, 585 op/s 2024-03-19T01:18:23.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:23 smithi116 ceph-mon[62701]: pgmap v525: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.7 MiB/s wr, 585 op/s 2024-03-19T01:18:23.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:23 smithi156 ceph-mon[66799]: pgmap v525: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.7 MiB/s wr, 585 op/s 2024-03-19T01:18:25.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:25 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:18:25.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:25 smithi162 ceph-mon[67178]: pgmap v526: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 909 op/s 2024-03-19T01:18:25.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:25 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:18:25.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:25 smithi116 ceph-mon[62701]: pgmap v526: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 909 op/s 2024-03-19T01:18:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:25 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:18:25.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:25 smithi156 ceph-mon[66799]: pgmap v526: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 909 op/s 2024-03-19T01:18:26.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 340.0 s, 1735.2 tps, lat 57.457 ms stddev 70.977 2024-03-19T01:18:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:27 smithi162 ceph-mon[67178]: pgmap v527: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 13 MiB/s wr, 867 op/s 2024-03-19T01:18:27.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:27 smithi116 ceph-mon[62701]: pgmap v527: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 13 MiB/s wr, 867 op/s 2024-03-19T01:18:27.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:27 smithi156 ceph-mon[66799]: pgmap v527: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 13 MiB/s wr, 867 op/s 2024-03-19T01:18:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.567+0000 7fab1f5f3640 1 -- 172.21.15.116:0/1014588607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab18108820 msgr2=0x7fab1810ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.567+0000 7fab1f5f3640 1 --2- 172.21.15.116:0/1014588607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab18108820 0x7fab1810ac50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fab0801a910 tx=0x7fab0803cc70 comp rx=0 tx=0).stop 2024-03-19T01:18:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.567+0000 7fab1f5f3640 1 -- 172.21.15.116:0/1014588607 shutdown_connections 2024-03-19T01:18:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.567+0000 7fab1f5f3640 1 --2- 172.21.15.116:0/1014588607 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab1810db40 0x7fab1810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.567+0000 7fab1f5f3640 1 --2- 172.21.15.116:0/1014588607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1810b190 0x7fab1810d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.567+0000 7fab1f5f3640 1 --2- 172.21.15.116:0/1014588607 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab18108820 0x7fab1810ac50 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fab0801a910 tx=0x7fab0803cc70 comp rx=0 tx=0).stop 2024-03-19T01:18:29.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.567+0000 7fab1f5f3640 1 -- 172.21.15.116:0/1014588607 >> 172.21.15.116:0/1014588607 conn(0x7fab1806e860 msgr2=0x7fab18070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.568+0000 7fab1f5f3640 1 -- 172.21.15.116:0/1014588607 shutdown_connections 2024-03-19T01:18:29.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.568+0000 7fab1f5f3640 1 -- 172.21.15.116:0/1014588607 wait complete. 2024-03-19T01:18:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.569+0000 7fab1f5f3640 1 Processor -- start 2024-03-19T01:18:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.569+0000 7fab1f5f3640 1 -- start start 2024-03-19T01:18:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.569+0000 7fab1f5f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1810b190 0x7fab1819e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.569+0000 7fab1f5f3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab1810db40 0x7fab1819e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.569+0000 7fab1f5f3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab181a37d0 0x7fab181a5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.569+0000 7fab1f5f3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fab18111af0 con 0x7fab1810b190 2024-03-19T01:18:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.569+0000 7fab1f5f3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fab18111970 con 0x7fab181a37d0 2024-03-19T01:18:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.569+0000 7fab1f5f3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fab18111c70 con 0x7fab1810db40 2024-03-19T01:18:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.569+0000 7fab1d38f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1810b190 0x7fab1819e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.569+0000 7fab1db90640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab181a37d0 0x7fab181a5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.569+0000 7fab1d38f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1810b190 0x7fab1819e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55710/0 (socket says 172.21.15.116:55710) 2024-03-19T01:18:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.569+0000 7fab1d38f640 1 -- 172.21.15.116:0/2095914840 learned_addr learned my addr 172.21.15.116:0/2095914840 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:29.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.569+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/2095914840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab1810db40 0x7fab1819e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.570+0000 7fab1cb8e640 1 -- 172.21.15.116:0/2095914840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab181a37d0 msgr2=0x7fab181a5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.570+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/2095914840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab181a37d0 0x7fab181a5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.570+0000 7fab1cb8e640 1 -- 172.21.15.116:0/2095914840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1810b190 msgr2=0x7fab1819e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:29.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.570+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/2095914840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1810b190 0x7fab1819e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.570+0000 7fab1cb8e640 1 -- 172.21.15.116:0/2095914840 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab181a6230 con 0x7fab1810db40 2024-03-19T01:18:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.570+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/2095914840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab1810db40 0x7fab1819e770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fab1400e9f0 tx=0x7fab1400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.571+0000 7fab0e7fc640 1 -- 172.21.15.116:0/2095914840 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1400ccc0 con 0x7fab1810db40 2024-03-19T01:18:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.571+0000 7fab1f5f3640 1 -- 172.21.15.116:0/2095914840 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab181a6520 con 0x7fab1810db40 2024-03-19T01:18:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.571+0000 7fab1f5f3640 1 -- 172.21.15.116:0/2095914840 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab181a6ac0 con 0x7fab1810db40 2024-03-19T01:18:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.571+0000 7fab0e7fc640 1 -- 172.21.15.116:0/2095914840 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab14010910 con 0x7fab1810db40 2024-03-19T01:18:29.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.572+0000 7fab0e7fc640 1 -- 172.21.15.116:0/2095914840 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1400ccc0 con 0x7fab1810db40 2024-03-19T01:18:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.574+0000 7fab221d6740 1 Processor -- start 2024-03-19T01:18:29.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.574+0000 7fab221d6740 1 -- start start 2024-03-19T01:18:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.574+0000 7fab221d6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf042520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.574+0000 7fab221d6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf042af0 0x55a2bf0475f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.574+0000 7fab221d6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf047c60 0x55a2bf04a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.574+0000 7fab221d6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a2befa6be0 con 0x55a2bf042af0 2024-03-19T01:18:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.574+0000 7fab221d6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a2bef4f960 con 0x55a2bf042100 2024-03-19T01:18:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.574+0000 7fab221d6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a2befa2c80 con 0x55a2bf047c60 2024-03-19T01:18:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.574+0000 7fab1d38f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf042af0 0x55a2bf0475f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.575+0000 7fab1d38f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf042af0 0x55a2bf0475f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55716/0 (socket says 172.21.15.116:55716) 2024-03-19T01:18:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.575+0000 7fab1d38f640 1 -- 172.21.15.116:0/1348284057 learned_addr learned my addr 172.21.15.116:0/1348284057 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.575+0000 7fab1db90640 1 --2- 172.21.15.116:0/1348284057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf042520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.575+0000 7fab1d38f640 1 -- 172.21.15.116:0/1348284057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf047c60 msgr2=0x55a2bf04a0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:18:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.575+0000 7fab1d38f640 1 --2- 172.21.15.116:0/1348284057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf047c60 0x55a2bf04a0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.575+0000 7fab1d38f640 1 -- 172.21.15.116:0/1348284057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 msgr2=0x55a2bf042520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.575+0000 7fab1d38f640 1 --2- 172.21.15.116:0/1348284057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf042520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.575+0000 7fab1d38f640 1 -- 172.21.15.116:0/1348284057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2bef46d20 con 0x55a2bf042af0 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.575+0000 7fab1db90640 1 --2- 172.21.15.116:0/1348284057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf042520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.575+0000 7fab0e7fc640 1 -- 172.21.15.116:0/2095914840 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fab14010ac0 con 0x7fab1810db40 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.575+0000 7fab1d38f640 1 --2- 172.21.15.116:0/1348284057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf042af0 0x55a2bf0475f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fab08004880 tx=0x7fab08005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e2d56a3ac9fa19 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.575+0000 7fab037fe640 1 -- 172.21.15.116:0/1348284057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0801a0a0 con 0x55a2bf042af0 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.575+0000 7fab037fe640 1 -- 172.21.15.116:0/1348284057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab08004040 con 0x55a2bf042af0 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.576+0000 7fab0e7fc640 1 --2- 172.21.15.116:0/2095914840 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fab040a2cf0 0x7fab040a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.576+0000 7fab0e7fc640 1 -- 172.21.15.116:0/2095914840 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fab1400cac0 con 0x7fab1810db40 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.576+0000 7fab037fe640 1 -- 172.21.15.116:0/1348284057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0803d820 con 0x55a2bf042af0 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.576+0000 7fab1db90640 1 --2- 172.21.15.116:0/2095914840 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fab040a2cf0 0x7fab040a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.576+0000 7fab221d6740 1 -- 172.21.15.116:0/1348284057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf042af0 msgr2=0x55a2bf0475f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.576+0000 7fab221d6740 1 --2- 172.21.15.116:0/1348284057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf042af0 0x55a2bf0475f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fab08004880 tx=0x7fab08005be0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.576+0000 7fab221d6740 1 -- 172.21.15.116:0/1348284057 shutdown_connections 2024-03-19T01:18:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.576+0000 7fab221d6740 1 --2- 172.21.15.116:0/1348284057 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf047c60 0x55a2bf04a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.576+0000 7fab221d6740 1 --2- 172.21.15.116:0/1348284057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf042af0 0x55a2bf0475f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab221d6740 1 --2- 172.21.15.116:0/1348284057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf042520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab221d6740 1 -- 172.21.15.116:0/1348284057 >> 172.21.15.116:0/1348284057 conn(0x55a2bf03d670 msgr2=0x55a2bf03da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab221d6740 1 -- 172.21.15.116:0/1348284057 shutdown_connections 2024-03-19T01:18:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab1db90640 1 --2- 172.21.15.116:0/2095914840 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fab040a2cf0 0x7fab040a51b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55a2bf043350 tx=0x7fab10009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab221d6740 1 -- 172.21.15.116:0/1348284057 wait complete. 2024-03-19T01:18:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab221d6740 1 -- start start 2024-03-19T01:18:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab221d6740 1 Processor -- start 2024-03-19T01:18:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab221d6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf04c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab221d6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042af0 0x55a2bf04c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab221d6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 0x55a2bf053630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab221d6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a2bf04ac00 con 0x55a2bf047c60 2024-03-19T01:18:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab221d6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a2bef62db0 con 0x55a2bf042100 2024-03-19T01:18:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab221d6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a2bf04ad80 con 0x55a2bf042af0 2024-03-19T01:18:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab221d6740 10 client.0 _ll_register_callbacks cb 0x55a2bef67e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:18:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab1d38f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf04c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab1db90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 0x55a2bf053630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.577+0000 7fab1d38f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf04c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:56624/0 (socket says 172.21.15.116:56624) 2024-03-19T01:18:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.578+0000 7fab1d38f640 1 -- 172.21.15.116:0/3744907242 learned_addr learned my addr 172.21.15.116:0/3744907242 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.578+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/3744907242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042af0 0x55a2bf04c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.578+0000 7fab1d38f640 1 -- 172.21.15.116:0/3744907242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042af0 msgr2=0x55a2bf04c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:29.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.578+0000 7fab1d38f640 1 --2- 172.21.15.116:0/3744907242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042af0 0x55a2bf04c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.578+0000 7fab1d38f640 1 -- 172.21.15.116:0/3744907242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 msgr2=0x55a2bf053630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:29.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.578+0000 7fab1d38f640 1 --2- 172.21.15.116:0/3744907242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 0x55a2bf053630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.578+0000 7fab1d38f640 1 -- 172.21.15.116:0/3744907242 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2bef61c60 con 0x55a2bf042100 2024-03-19T01:18:29.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.578+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/3744907242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042af0 0x55a2bf04c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:18:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.579+0000 7fab1d38f640 1 --2- 172.21.15.116:0/3744907242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf04c0b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fab080047e0 tx=0x7fab080044a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91d14eef41e7c0d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.579+0000 7fab00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:18:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.579+0000 7fab00ff9640 1 -- 172.21.15.116:0/3744907242 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0801a0a0 con 0x55a2bf042100 2024-03-19T01:18:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.579+0000 7fab00ff9640 1 -- 172.21.15.116:0/3744907242 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fab08005670 con 0x55a2bf042100 2024-03-19T01:18:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.580+0000 7fab221d6740 1 -- 172.21.15.116:0/3744907242 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab08004040 con 0x55a2bf042100 2024-03-19T01:18:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.580+0000 7fab00ff9640 1 -- 172.21.15.116:0/3744907242 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fab0803d820 con 0x55a2bf042100 2024-03-19T01:18:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.580+0000 7fab221d6740 10 client.25613 fetch_fsmap learned FSMap version 45 2024-03-19T01:18:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.580+0000 7fab221d6740 1 -- 172.21.15.116:0/3744907242 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55a2becbb210 con 0x55a2bf042100 2024-03-19T01:18:29.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.580+0000 7fab00ff9640 1 -- 172.21.15.116:0/3744907242 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab08004040 con 0x55a2bf042100 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.580+0000 7fab00ff9640 1 -- 172.21.15.116:0/3744907242 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fab08003cf0 con 0x55a2bf042100 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.580+0000 7fab221d6740 10 client.25613 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.580+0000 7fab221d6740 10 client.25613 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.580+0000 7fab221d6740 20 client.25613 populate_metadata read hostname 'smithi116' 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.580+0000 7fab221d6740 1 --2- 172.21.15.116:0/3744907242 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a2bf057b20 0x55a2bf059f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.580+0000 7fab221d6740 4 client.25613 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.580+0000 7fab221d6740 1 -- 172.21.15.116:0/3744907242 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2befb4120 con 0x55a2bf057b20 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.581+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/3744907242 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a2bf057b20 0x55a2bf059f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.581+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/3744907242 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a2bf057b20 0x55a2bf059f10 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e436c93400f4b05 server_cookie=9ccdec3233668a7c in_seq=0 out_seq=0 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.581+0000 7fab00ff9640 10 client.25613 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab00ff9640 1 -- 172.21.15.116:0/3744907242 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a2befb4120 con 0x55a2bf057b20 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab00ff9640 10 client.25613 handle_command_reply: tid=0 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab221d6740 1 client.25613 shutdown 2024-03-19T01:18:29.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab221d6740 1 -- 172.21.15.116:0/3744907242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 msgr2=0x55a2bf04c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab221d6740 1 --2- 172.21.15.116:0/3744907242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf04c0b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fab080047e0 tx=0x7fab080044a0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab221d6740 1 -- 172.21.15.116:0/3744907242 shutdown_connections 2024-03-19T01:18:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab221d6740 1 --2- 172.21.15.116:0/3744907242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 0x55a2bf053630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab221d6740 1 --2- 172.21.15.116:0/3744907242 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a2bf057b20 0x55a2bf059f10 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab221d6740 1 --2- 172.21.15.116:0/3744907242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042af0 0x55a2bf04c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab00ff9640 0 client.25613 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab221d6740 1 --2- 172.21.15.116:0/3744907242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf04c0b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab221d6740 1 -- 172.21.15.116:0/3744907242 >> 172.21.15.116:0/3744907242 conn(0x55a2bf03d670 msgr2=0x55a2bf03e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab221d6740 1 -- 172.21.15.116:0/3744907242 shutdown_connections 2024-03-19T01:18:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab221d6740 1 -- 172.21.15.116:0/3744907242 wait complete. 2024-03-19T01:18:29.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.582+0000 7fab221d6740 20 client.25613 trim_cache size 0 max 16384 2024-03-19T01:18:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.605+0000 7fab221d6740 1 Processor -- start 2024-03-19T01:18:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.605+0000 7fab221d6740 1 -- start start 2024-03-19T01:18:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.605+0000 7fab221d6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042100 0x55a2bf043c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.605+0000 7fab221d6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042af0 0x55a2bf04b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.605+0000 7fab221d6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 0x55a2bf04b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.605+0000 7fab221d6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faafc00e5f0 con 0x55a2bf047c60 2024-03-19T01:18:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.605+0000 7fab221d6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faafc010130 con 0x55a2bf042af0 2024-03-19T01:18:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.605+0000 7fab221d6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faafc00ecc0 con 0x55a2bf042100 2024-03-19T01:18:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.605+0000 7fab1d38f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042100 0x55a2bf043c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.605+0000 7fab1cb8e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042af0 0x55a2bf04b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.605+0000 7fab1d38f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042100 0x55a2bf043c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54078/0 (socket says 172.21.15.116:54078) 2024-03-19T01:18:29.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.605+0000 7fab1d38f640 1 -- 172.21.15.116:0/1868879276 learned_addr learned my addr 172.21.15.116:0/1868879276 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.605+0000 7fab1db90640 1 --2- 172.21.15.116:0/1868879276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 0x55a2bf04b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.606+0000 7fab1cb8e640 1 -- 172.21.15.116:0/1868879276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042100 msgr2=0x55a2bf043c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.606+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/1868879276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042100 0x55a2bf043c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.606+0000 7fab1cb8e640 1 -- 172.21.15.116:0/1868879276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 msgr2=0x55a2bf04b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.606+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/1868879276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 0x55a2bf04b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.606+0000 7fab1cb8e640 1 -- 172.21.15.116:0/1868879276 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2bed87620 con 0x55a2bf042af0 2024-03-19T01:18:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.606+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/1868879276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042af0 0x55a2bf04b200 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c6a0 tx=0x7fab140cc1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=156a56439ae3dee server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.606+0000 7fab02ffd640 1 -- 172.21.15.116:0/1868879276 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab140a3950 con 0x55a2bf042af0 2024-03-19T01:18:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.606+0000 7fab02ffd640 1 -- 172.21.15.116:0/1868879276 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab14099250 con 0x55a2bf042af0 2024-03-19T01:18:29.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.606+0000 7fab02ffd640 1 -- 172.21.15.116:0/1868879276 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab140979b0 con 0x55a2bf042af0 2024-03-19T01:18:29.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.607+0000 7fab221d6740 1 -- 172.21.15.116:0/1868879276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042af0 msgr2=0x55a2bf04b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:29.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.607+0000 7fab221d6740 1 --2- 172.21.15.116:0/1868879276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042af0 0x55a2bf04b200 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c6a0 tx=0x7fab140cc1e0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.607+0000 7fab221d6740 1 -- 172.21.15.116:0/1868879276 shutdown_connections 2024-03-19T01:18:29.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.607+0000 7fab221d6740 1 --2- 172.21.15.116:0/1868879276 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 0x55a2bf04b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.607+0000 7fab221d6740 1 --2- 172.21.15.116:0/1868879276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042af0 0x55a2bf04b200 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.607+0000 7fab221d6740 1 --2- 172.21.15.116:0/1868879276 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042100 0x55a2bf043c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.607+0000 7fab221d6740 1 -- 172.21.15.116:0/1868879276 >> 172.21.15.116:0/1868879276 conn(0x55a2bf03d670 msgr2=0x55a2bf043560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.607+0000 7fab221d6740 1 -- 172.21.15.116:0/1868879276 shutdown_connections 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.607+0000 7fab221d6740 1 -- 172.21.15.116:0/1868879276 wait complete. 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.607+0000 7fab221d6740 1 -- start start 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.607+0000 7fab221d6740 1 Processor -- start 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab221d6740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf050870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab221d6740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042af0 0x55a2bf050db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab221d6740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 0x55a2bf051dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab221d6740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faafc00fa60 con 0x55a2bf047c60 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab221d6740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faafc00f390 con 0x55a2bf042100 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab221d6740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7faafc00c5d0 con 0x55a2bf042af0 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab221d6740 10 client.0 _ll_register_callbacks cb 0x7fab10009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab1cb8e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042af0 0x55a2bf050db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab1d38f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf050870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab1cb8e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042af0 0x55a2bf050db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:54082/0 (socket says 172.21.15.116:54082) 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab1cb8e640 1 -- 172.21.15.116:0/373537362 learned_addr learned my addr 172.21.15.116:0/373537362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab1db90640 1 --2- 172.21.15.116:0/373537362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 0x55a2bf051dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab1cb8e640 1 -- 172.21.15.116:0/373537362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 msgr2=0x55a2bf050870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/373537362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf050870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab1cb8e640 1 -- 172.21.15.116:0/373537362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 msgr2=0x55a2bf051dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/373537362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 0x55a2bf051dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab1cb8e640 1 -- 172.21.15.116:0/373537362 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2bf0523b0 con 0x55a2bf042af0 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab1d38f640 1 --2- 172.21.15.116:0/373537362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf050870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab1cb8e640 1 --2- 172.21.15.116:0/373537362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042af0 0x55a2bf050db0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fab140cc360 tx=0x7fab1409f1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1697be742e530401 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab00ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.608+0000 7fab00ff9640 1 -- 172.21.15.116:0/373537362 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab140994b0 con 0x55a2bf042af0 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.609+0000 7fab00ff9640 1 -- 172.21.15.116:0/373537362 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fab1409a600 con 0x55a2bf042af0 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.609+0000 7fab221d6740 1 -- 172.21.15.116:0/373537362 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab14099250 con 0x55a2bf042af0 2024-03-19T01:18:29.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.609+0000 7fab00ff9640 1 -- 172.21.15.116:0/373537362 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fab140a3950 con 0x55a2bf042af0 2024-03-19T01:18:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.609+0000 7fab221d6740 10 client.25597 fetch_fsmap learned FSMap version 45 2024-03-19T01:18:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.609+0000 7fab221d6740 1 -- 172.21.15.116:0/373537362 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55a2bf0bdd40 con 0x55a2bf042af0 2024-03-19T01:18:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.609+0000 7fab00ff9640 1 -- 172.21.15.116:0/373537362 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab14099250 con 0x55a2bf042af0 2024-03-19T01:18:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.609+0000 7fab00ff9640 1 -- 172.21.15.116:0/373537362 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fab1409a160 con 0x55a2bf042af0 2024-03-19T01:18:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.609+0000 7fab221d6740 10 client.25597 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:18:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.609+0000 7fab221d6740 10 client.25597 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:18:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.609+0000 7fab221d6740 20 client.25597 populate_metadata read hostname 'smithi116' 2024-03-19T01:18:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.609+0000 7fab221d6740 1 --2- 172.21.15.116:0/373537362 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a2bf0be4e0 0x55a2bf0c0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.609+0000 7fab221d6740 4 client.25597 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:18:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.609+0000 7fab221d6740 1 -- 172.21.15.116:0/373537362 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a2befae470 con 0x55a2bf0be4e0 2024-03-19T01:18:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.610+0000 7fab1d38f640 1 --2- 172.21.15.116:0/373537362 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a2bf0be4e0 0x55a2bf0c0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:29.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.610+0000 7fab00ff9640 10 client.25597 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:29.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.611+0000 7fab1d38f640 1 --2- 172.21.15.116:0/373537362 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a2bf0be4e0 0x55a2bf0c0900 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=def50649d50f2494 server_cookie=d656f50ef48cefaf in_seq=0 out_seq=0 2024-03-19T01:18:29.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.618+0000 7fab00ff9640 1 -- 172.21.15.116:0/373537362 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a2befae470 con 0x55a2bf0be4e0 2024-03-19T01:18:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.618+0000 7fab00ff9640 10 client.25597 handle_command_reply: tid=0 2024-03-19T01:18:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.618+0000 7fab221d6740 1 client.25597 shutdown 2024-03-19T01:18:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.618+0000 7fab221d6740 1 -- 172.21.15.116:0/373537362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042af0 msgr2=0x55a2bf050db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.618+0000 7fab221d6740 1 --2- 172.21.15.116:0/373537362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042af0 0x55a2bf050db0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fab140cc360 tx=0x7fab1409f1f0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.619+0000 7fab221d6740 1 -- 172.21.15.116:0/373537362 shutdown_connections 2024-03-19T01:18:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.619+0000 7fab221d6740 1 --2- 172.21.15.116:0/373537362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a2bf047c60 0x55a2bf051dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.619+0000 7fab221d6740 1 --2- 172.21.15.116:0/373537362 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a2bf0be4e0 0x55a2bf0c0900 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.619+0000 7fab221d6740 1 --2- 172.21.15.116:0/373537362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a2bf042af0 0x55a2bf050db0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.619+0000 7fab221d6740 1 --2- 172.21.15.116:0/373537362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a2bf042100 0x55a2bf050870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.619+0000 7fab00ff9640 0 client.25597 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.619+0000 7fab221d6740 1 -- 172.21.15.116:0/373537362 >> 172.21.15.116:0/373537362 conn(0x55a2bf03d670 msgr2=0x55a2bf041160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.619+0000 7fab221d6740 1 -- 172.21.15.116:0/373537362 shutdown_connections 2024-03-19T01:18:29.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.619+0000 7fab221d6740 1 -- 172.21.15.116:0/373537362 wait complete. 2024-03-19T01:18:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.619+0000 7fab221d6740 20 client.25597 trim_cache size 0 max 16384 2024-03-19T01:18:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:18:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:18:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:18:29.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:18:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.620+0000 7fab1f5f3640 1 -- 172.21.15.116:0/2095914840 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fab040a2cf0 msgr2=0x7fab040a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.620+0000 7fab1f5f3640 1 --2- 172.21.15.116:0/2095914840 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fab040a2cf0 0x7fab040a51b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55a2bf043350 tx=0x7fab10009500 comp rx=0 tx=0).stop 2024-03-19T01:18:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.620+0000 7fab1f5f3640 1 -- 172.21.15.116:0/2095914840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab1810db40 msgr2=0x7fab1819e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.620+0000 7fab1f5f3640 1 --2- 172.21.15.116:0/2095914840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab1810db40 0x7fab1819e770 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fab1400e9f0 tx=0x7fab1400eeb0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.620+0000 7fab1f5f3640 1 -- 172.21.15.116:0/2095914840 shutdown_connections 2024-03-19T01:18:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.620+0000 7fab1f5f3640 1 --2- 172.21.15.116:0/2095914840 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fab040a2cf0 0x7fab040a51b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.620+0000 7fab1f5f3640 1 --2- 172.21.15.116:0/2095914840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab181a37d0 0x7fab181a5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.620+0000 7fab1f5f3640 1 --2- 172.21.15.116:0/2095914840 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fab1810db40 0x7fab1819e770 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.620+0000 7fab1f5f3640 1 --2- 172.21.15.116:0/2095914840 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fab1810b190 0x7fab1819e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.620+0000 7fab1f5f3640 1 -- 172.21.15.116:0/2095914840 >> 172.21.15.116:0/2095914840 conn(0x7fab1806e860 msgr2=0x7fab18108f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.620+0000 7fab1f5f3640 1 -- 172.21.15.116:0/2095914840 shutdown_connections 2024-03-19T01:18:29.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:29.620+0000 7fab1f5f3640 1 -- 172.21.15.116:0/2095914840 wait complete. 2024-03-19T01:18:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:29 smithi162 ceph-mon[67178]: pgmap v528: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 MiB/s wr, 935 op/s 2024-03-19T01:18:29.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:29 smithi116 ceph-mon[62701]: pgmap v528: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 MiB/s wr, 935 op/s 2024-03-19T01:18:29.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:29 smithi156 ceph-mon[66799]: pgmap v528: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 14 MiB/s wr, 935 op/s 2024-03-19T01:18:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 345.0 s, 1964.6 tps, lat 51.158 ms stddev 55.739 2024-03-19T01:18:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:31 smithi162 ceph-mon[67178]: pgmap v529: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 14 MiB/s wr, 913 op/s 2024-03-19T01:18:31.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:31 smithi116 ceph-mon[62701]: pgmap v529: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 14 MiB/s wr, 913 op/s 2024-03-19T01:18:31.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:31 smithi156 ceph-mon[66799]: pgmap v529: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 14 MiB/s wr, 913 op/s 2024-03-19T01:18:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:33 smithi162 ceph-mon[67178]: pgmap v530: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 13 MiB/s wr, 877 op/s 2024-03-19T01:18:33.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:33 smithi116 ceph-mon[62701]: pgmap v530: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 13 MiB/s wr, 877 op/s 2024-03-19T01:18:33.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:33 smithi156 ceph-mon[66799]: pgmap v530: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 13 MiB/s wr, 877 op/s 2024-03-19T01:18:36.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:36 smithi162 ceph-mon[67178]: pgmap v531: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 MiB/s wr, 1.04k op/s 2024-03-19T01:18:36.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:36 smithi116 ceph-mon[62701]: pgmap v531: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 MiB/s wr, 1.04k op/s 2024-03-19T01:18:36.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:36 smithi156 ceph-mon[66799]: pgmap v531: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 MiB/s wr, 1.04k op/s 2024-03-19T01:18:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 350.0 s, 2115.6 tps, lat 46.947 ms stddev 55.848 2024-03-19T01:18:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:37 smithi162 ceph-mon[67178]: pgmap v532: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 MiB/s wr, 795 op/s 2024-03-19T01:18:37.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:37 smithi116 ceph-mon[62701]: pgmap v532: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 MiB/s wr, 795 op/s 2024-03-19T01:18:37.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:37 smithi156 ceph-mon[66799]: pgmap v532: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 13 MiB/s wr, 795 op/s 2024-03-19T01:18:39.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:39 smithi162 ceph-mon[67178]: pgmap v533: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 791 op/s 2024-03-19T01:18:39.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:39 smithi116 ceph-mon[62701]: pgmap v533: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 791 op/s 2024-03-19T01:18:39.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:39 smithi156 ceph-mon[66799]: pgmap v533: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 791 op/s 2024-03-19T01:18:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 355.0 s, 2613.8 tps, lat 38.557 ms stddev 46.510 2024-03-19T01:18:41.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:41 smithi162 ceph-mon[67178]: pgmap v534: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 15 MiB/s wr, 748 op/s 2024-03-19T01:18:41.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:41 smithi116 ceph-mon[62701]: pgmap v534: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 15 MiB/s wr, 748 op/s 2024-03-19T01:18:41.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:41 smithi156 ceph-mon[66799]: pgmap v534: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 15 MiB/s wr, 748 op/s 2024-03-19T01:18:42.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.746+0000 7f6a37eac640 1 -- 172.21.15.116:0/2660798152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30101930 msgr2=0x7f6a30101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:42.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.746+0000 7f6a37eac640 1 --2- 172.21.15.116:0/2660798152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30101930 0x7f6a30101db0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009900 tx=0x7f6a2002bc30 comp rx=0 tx=0).stop 2024-03-19T01:18:42.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.746+0000 7f6a37eac640 1 -- 172.21.15.116:0/2660798152 shutdown_connections 2024-03-19T01:18:42.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.746+0000 7f6a37eac640 1 --2- 172.21.15.116:0/2660798152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a301022f0 0x7f6a30108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.746+0000 7f6a37eac640 1 --2- 172.21.15.116:0/2660798152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30101930 0x7f6a30101db0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.746+0000 7f6a37eac640 1 --2- 172.21.15.116:0/2660798152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a30100740 0x7f6a30100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.746+0000 7f6a37eac640 1 -- 172.21.15.116:0/2660798152 >> 172.21.15.116:0/2660798152 conn(0x7f6a300fbcb0 msgr2=0x7f6a300fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:42.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.747+0000 7f6a37eac640 1 -- 172.21.15.116:0/2660798152 shutdown_connections 2024-03-19T01:18:42.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.747+0000 7f6a37eac640 1 -- 172.21.15.116:0/2660798152 wait complete. 2024-03-19T01:18:42.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.748+0000 7f6a37eac640 1 Processor -- start 2024-03-19T01:18:42.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.748+0000 7f6a37eac640 1 -- start start 2024-03-19T01:18:42.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.748+0000 7f6a37eac640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a30100740 0x7f6a3019e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.748+0000 7f6a37eac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30101930 0x7f6a3019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.748+0000 7f6a37eac640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a301022f0 0x7f6a301a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.748+0000 7f6a37eac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3010b2a0 con 0x7f6a30101930 2024-03-19T01:18:42.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.748+0000 7f6a37eac640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3010b120 con 0x7f6a30100740 2024-03-19T01:18:42.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.748+0000 7f6a35c48640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a30100740 0x7f6a3019e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.748+0000 7f6a35447640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30101930 0x7f6a3019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.748+0000 7f6a35447640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30101930 0x7f6a3019e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57772/0 (socket says 172.21.15.116:57772) 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.748+0000 7f6a35447640 1 -- 172.21.15.116:0/3588644965 learned_addr learned my addr 172.21.15.116:0/3588644965 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.748+0000 7f6a35c48640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a30100740 0x7f6a3019e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:52148/0 (socket says 172.21.15.116:52148) 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.748+0000 7f6a36449640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a301022f0 0x7f6a301a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.749+0000 7f6a37eac640 1 -- 172.21.15.116:0/3588644965 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3010b420 con 0x7f6a301022f0 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.749+0000 7f6a35447640 1 -- 172.21.15.116:0/3588644965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a301022f0 msgr2=0x7f6a301a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.749+0000 7f6a35447640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a301022f0 0x7f6a301a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.749+0000 7f6a35447640 1 -- 172.21.15.116:0/3588644965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a30100740 msgr2=0x7f6a3019e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.749+0000 7f6a35447640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a30100740 0x7f6a3019e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.749+0000 7f6a35447640 1 -- 172.21.15.116:0/3588644965 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a301a5d30 con 0x7f6a30101930 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.749+0000 7f6a35c48640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a30100740 0x7f6a3019e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.749+0000 7f6a35447640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30101930 0x7f6a3019e650 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009230 tx=0x7f6a2002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.749+0000 7f6a26ffd640 1 -- 172.21.15.116:0/3588644965 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a20041070 con 0x7f6a30101930 2024-03-19T01:18:42.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.749+0000 7f6a37eac640 1 -- 172.21.15.116:0/3588644965 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a301a5fc0 con 0x7f6a30101930 2024-03-19T01:18:42.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.749+0000 7f6a26ffd640 1 -- 172.21.15.116:0/3588644965 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a20004450 con 0x7f6a30101930 2024-03-19T01:18:42.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.749+0000 7f6a26ffd640 1 -- 172.21.15.116:0/3588644965 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a20003e20 con 0x7f6a30101930 2024-03-19T01:18:42.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.749+0000 7f6a37eac640 1 -- 172.21.15.116:0/3588644965 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a301a6430 con 0x7f6a30101930 2024-03-19T01:18:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.753+0000 7f6a3ab75740 1 Processor -- start 2024-03-19T01:18:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.753+0000 7f6a3ab75740 1 -- start start 2024-03-19T01:18:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.753+0000 7f6a3ab75740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f716100 0x56473f716520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.753+0000 7f6a3ab75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716af0 0x56473f71b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.753+0000 7f6a3ab75740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f71bc60 0x56473f71e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.753+0000 7f6a3ab75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56473f67abe0 con 0x56473f716af0 2024-03-19T01:18:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.753+0000 7f6a3ab75740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56473f623960 con 0x56473f716100 2024-03-19T01:18:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.753+0000 7f6a3ab75740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56473f676c80 con 0x56473f71bc60 2024-03-19T01:18:42.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.753+0000 7f6a35c48640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716af0 0x56473f71b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.753+0000 7f6a35c48640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716af0 0x56473f71b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57774/0 (socket says 172.21.15.116:57774) 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.753+0000 7f6a35c48640 1 -- 172.21.15.116:0/2820583109 learned_addr learned my addr 172.21.15.116:0/2820583109 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.753+0000 7f6a36449640 1 --2- 172.21.15.116:0/2820583109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f716100 0x56473f716520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.753+0000 7f6a26ffd640 1 -- 172.21.15.116:0/3588644965 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6a20044070 con 0x7f6a30101930 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.754+0000 7f6a35c48640 1 -- 172.21.15.116:0/2820583109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f71bc60 msgr2=0x56473f71e0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.754+0000 7f6a35c48640 1 --2- 172.21.15.116:0/2820583109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f71bc60 0x56473f71e0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.754+0000 7f6a35c48640 1 -- 172.21.15.116:0/2820583109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f716100 msgr2=0x56473f716520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.754+0000 7f6a35c48640 1 --2- 172.21.15.116:0/2820583109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f716100 0x56473f716520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.754+0000 7f6a35c48640 1 -- 172.21.15.116:0/2820583109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56473f61ad20 con 0x56473f716af0 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.754+0000 7f6a36449640 1 --2- 172.21.15.116:0/2820583109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f716100 0x56473f716520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.754+0000 7f6a35c48640 1 --2- 172.21.15.116:0/2820583109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716af0 0x56473f71b5f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0114f0 tx=0x7f6a2c0119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba2f661c1327e0d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.754+0000 7f6a1bfff640 1 -- 172.21.15.116:0/2820583109 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c01e070 con 0x56473f716af0 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.754+0000 7f6a1bfff640 1 -- 172.21.15.116:0/2820583109 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a2c0035f0 con 0x56473f716af0 2024-03-19T01:18:42.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.754+0000 7f6a1bfff640 1 -- 172.21.15.116:0/2820583109 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c00f800 con 0x56473f716af0 2024-03-19T01:18:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.754+0000 7f6a26ffd640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6a1c0a2ab0 0x7f6a1c0a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.754+0000 7f6a36449640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6a1c0a2ab0 0x7f6a1c0a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.754+0000 7f6a26ffd640 1 -- 172.21.15.116:0/3588644965 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6a20003bc0 con 0x7f6a30101930 2024-03-19T01:18:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.755+0000 7f6a3ab75740 1 -- 172.21.15.116:0/2820583109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716af0 msgr2=0x56473f71b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.755+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/2820583109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716af0 0x56473f71b5f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0114f0 tx=0x7f6a2c0119b0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.755+0000 7f6a36449640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6a1c0a2ab0 0x7f6a1c0a4f70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x56473f717350 tx=0x7f6a28009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.756+0000 7f6a3ab75740 1 -- 172.21.15.116:0/2820583109 shutdown_connections 2024-03-19T01:18:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.756+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/2820583109 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f71bc60 0x56473f71e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.756+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/2820583109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716af0 0x56473f71b5f0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.756+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/2820583109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f716100 0x56473f716520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.756+0000 7f6a3ab75740 1 -- 172.21.15.116:0/2820583109 >> 172.21.15.116:0/2820583109 conn(0x56473f711670 msgr2=0x56473f711a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.757+0000 7f6a3ab75740 1 -- 172.21.15.116:0/2820583109 shutdown_connections 2024-03-19T01:18:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.757+0000 7f6a3ab75740 1 -- 172.21.15.116:0/2820583109 wait complete. 2024-03-19T01:18:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.757+0000 7f6a3ab75740 1 -- start start 2024-03-19T01:18:42.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.757+0000 7f6a3ab75740 1 Processor -- start 2024-03-19T01:18:42.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a3ab75740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716100 0x56473f720130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a3ab75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716af0 0x56473f720670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a3ab75740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f71bc60 0x56473f7276b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a3ab75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56473f71ec10 con 0x56473f716af0 2024-03-19T01:18:42.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a3ab75740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56473f636db0 con 0x56473f71bc60 2024-03-19T01:18:42.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a3ab75740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56473f71ed90 con 0x56473f716100 2024-03-19T01:18:42.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a3ab75740 10 client.0 _ll_register_callbacks cb 0x56473f63be70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:18:42.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a35c48640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716100 0x56473f720130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a35c48640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716100 0x56473f720130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38662/0 (socket says 172.21.15.116:38662) 2024-03-19T01:18:42.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a35c48640 1 -- 172.21.15.116:0/3311752603 learned_addr learned my addr 172.21.15.116:0/3311752603 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:42.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a35447640 1 --2- 172.21.15.116:0/3311752603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716af0 0x56473f720670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a36449640 1 --2- 172.21.15.116:0/3311752603 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f71bc60 0x56473f7276b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a35c48640 1 -- 172.21.15.116:0/3311752603 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f71bc60 msgr2=0x56473f7276b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a35c48640 1 --2- 172.21.15.116:0/3311752603 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f71bc60 0x56473f7276b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a35c48640 1 -- 172.21.15.116:0/3311752603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716af0 msgr2=0x56473f720670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a35c48640 1 --2- 172.21.15.116:0/3311752603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716af0 0x56473f720670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a35c48640 1 -- 172.21.15.116:0/3311752603 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56473f635c60 con 0x56473f716100 2024-03-19T01:18:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a35c48640 1 --2- 172.21.15.116:0/3311752603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716100 0x56473f720130 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c011eb0 tx=0x7f6a2c004290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=495e2789e3745dbf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.758+0000 7f6a197fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:18:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.759+0000 7f6a197fa640 1 -- 172.21.15.116:0/3311752603 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c026020 con 0x56473f716100 2024-03-19T01:18:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.759+0000 7f6a197fa640 1 -- 172.21.15.116:0/3311752603 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6a2c005040 con 0x56473f716100 2024-03-19T01:18:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.759+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3311752603 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a2c0035f0 con 0x56473f716100 2024-03-19T01:18:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.759+0000 7f6a197fa640 1 -- 172.21.15.116:0/3311752603 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f6a2c031050 con 0x56473f716100 2024-03-19T01:18:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.759+0000 7f6a3ab75740 10 client.25606 fetch_fsmap learned FSMap version 45 2024-03-19T01:18:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.759+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3311752603 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56473f38f210 con 0x56473f716100 2024-03-19T01:18:42.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.759+0000 7f6a197fa640 1 -- 172.21.15.116:0/3311752603 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a2c0035f0 con 0x56473f716100 2024-03-19T01:18:42.765 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.760+0000 7f6a197fa640 1 -- 172.21.15.116:0/3311752603 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f6a2c021030 con 0x56473f716100 2024-03-19T01:18:42.765 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.760+0000 7f6a3ab75740 10 client.25606 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:18:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.760+0000 7f6a3ab75740 10 client.25606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:18:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.760+0000 7f6a3ab75740 20 client.25606 populate_metadata read hostname 'smithi116' 2024-03-19T01:18:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.760+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/3311752603 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56473f72bd60 0x56473f72e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.760+0000 7f6a3ab75740 4 client.25606 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:18:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.760+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3311752603 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56473f688120 con 0x56473f72bd60 2024-03-19T01:18:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.760+0000 7f6a35447640 1 --2- 172.21.15.116:0/3311752603 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56473f72bd60 0x56473f72e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.761+0000 7f6a197fa640 10 client.25606 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.761+0000 7f6a35447640 1 --2- 172.21.15.116:0/3311752603 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56473f72bd60 0x56473f72e150 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b43e2ee7ec0f83e server_cookie=ec5f32e80a9533a8 in_seq=0 out_seq=0 2024-03-19T01:18:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a197fa640 1 -- 172.21.15.116:0/3311752603 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56473f688120 con 0x56473f72bd60 2024-03-19T01:18:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a197fa640 10 client.25606 handle_command_reply: tid=0 2024-03-19T01:18:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a3ab75740 1 client.25606 shutdown 2024-03-19T01:18:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3311752603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716100 msgr2=0x56473f720130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/3311752603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716100 0x56473f720130 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c011eb0 tx=0x7f6a2c004290 comp rx=0 tx=0).stop 2024-03-19T01:18:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3311752603 shutdown_connections 2024-03-19T01:18:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/3311752603 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f71bc60 0x56473f7276b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/3311752603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716af0 0x56473f720670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/3311752603 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56473f72bd60 0x56473f72e150 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/3311752603 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716100 0x56473f720130 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a197fa640 0 client.25606 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3311752603 >> 172.21.15.116:0/3311752603 conn(0x56473f711670 msgr2=0x56473f7128c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3311752603 shutdown_connections 2024-03-19T01:18:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3311752603 wait complete. 2024-03-19T01:18:42.767 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.762+0000 7f6a3ab75740 20 client.25606 trim_cache size 0 max 16384 2024-03-19T01:18:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a3ab75740 1 Processor -- start 2024-03-19T01:18:42.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a3ab75740 1 -- start start 2024-03-19T01:18:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a3ab75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716100 0x56473f71f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a3ab75740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716af0 0x56473f727670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a3ab75740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f71bc60 0x56473f71fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a3ab75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a301af300 con 0x56473f716100 2024-03-19T01:18:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a3ab75740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6a301b0e40 con 0x56473f71bc60 2024-03-19T01:18:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a3ab75740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a301af9d0 con 0x56473f716af0 2024-03-19T01:18:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a35c48640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716100 0x56473f71f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a35447640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716af0 0x56473f727670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a35447640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716af0 0x56473f727670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38672/0 (socket says 172.21.15.116:38672) 2024-03-19T01:18:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a35c48640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716100 0x56473f71f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57784/0 (socket says 172.21.15.116:57784) 2024-03-19T01:18:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a35447640 1 -- 172.21.15.116:0/2381225053 learned_addr learned my addr 172.21.15.116:0/2381225053 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.784+0000 7f6a36449640 1 --2- 172.21.15.116:0/2381225053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f71bc60 0x56473f71fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.785+0000 7f6a35447640 1 -- 172.21.15.116:0/2381225053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f71bc60 msgr2=0x56473f71fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.785+0000 7f6a35447640 1 --2- 172.21.15.116:0/2381225053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f71bc60 0x56473f71fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.785+0000 7f6a35447640 1 -- 172.21.15.116:0/2381225053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716100 msgr2=0x56473f71f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.785+0000 7f6a35447640 1 --2- 172.21.15.116:0/2381225053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716100 0x56473f71f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.785+0000 7f6a35447640 1 -- 172.21.15.116:0/2381225053 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56473f45b620 con 0x56473f716af0 2024-03-19T01:18:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.785+0000 7f6a35c48640 1 --2- 172.21.15.116:0/2381225053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716100 0x56473f71f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:18:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.785+0000 7f6a35447640 1 --2- 172.21.15.116:0/2381225053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716af0 0x56473f727670 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6a200edea0 tx=0x7f6a20003920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66e292d1aea156d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.785+0000 7f6a1b7fe640 1 -- 172.21.15.116:0/2381225053 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2004b040 con 0x56473f716af0 2024-03-19T01:18:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.785+0000 7f6a1b7fe640 1 -- 172.21.15.116:0/2381225053 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a200bcda0 con 0x56473f716af0 2024-03-19T01:18:42.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.785+0000 7f6a1b7fe640 1 -- 172.21.15.116:0/2381225053 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a200b9410 con 0x56473f716af0 2024-03-19T01:18:42.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.786+0000 7f6a3ab75740 1 -- 172.21.15.116:0/2381225053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716af0 msgr2=0x56473f727670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:42.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.786+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/2381225053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716af0 0x56473f727670 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6a200edea0 tx=0x7f6a20003920 comp rx=0 tx=0).stop 2024-03-19T01:18:42.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.786+0000 7f6a3ab75740 1 -- 172.21.15.116:0/2381225053 shutdown_connections 2024-03-19T01:18:42.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.786+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/2381225053 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f71bc60 0x56473f71fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.786+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/2381225053 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716af0 0x56473f727670 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.786+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/2381225053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f716100 0x56473f71f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.786+0000 7f6a3ab75740 1 -- 172.21.15.116:0/2381225053 >> 172.21.15.116:0/2381225053 conn(0x56473f711670 msgr2=0x56473f715160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:42.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.786+0000 7f6a3ab75740 1 -- 172.21.15.116:0/2381225053 shutdown_connections 2024-03-19T01:18:42.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.786+0000 7f6a3ab75740 1 -- 172.21.15.116:0/2381225053 wait complete. 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.786+0000 7f6a3ab75740 1 -- start start 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.786+0000 7f6a3ab75740 1 Processor -- start 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a3ab75740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716100 0x56473f724ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a3ab75740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f716af0 0x56473f725400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a3ab75740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f71bc60 0x56473f726410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a3ab75740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6a301b0770 con 0x56473f71bc60 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a3ab75740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6a301b00a0 con 0x56473f716af0 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a3ab75740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6a301ad2e0 con 0x56473f716100 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a3ab75740 10 client.0 _ll_register_callbacks cb 0x7f6a2c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a36449640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f71bc60 0x56473f726410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a36449640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f71bc60 0x56473f726410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57792/0 (socket says 172.21.15.116:57792) 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a36449640 1 -- 172.21.15.116:0/3940410989 learned_addr learned my addr 172.21.15.116:0/3940410989 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a35c48640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716100 0x56473f724ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a35447640 1 --2- 172.21.15.116:0/3940410989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f716af0 0x56473f725400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a36449640 1 -- 172.21.15.116:0/3940410989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716100 msgr2=0x56473f724ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a36449640 1 --2- 172.21.15.116:0/3940410989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716100 0x56473f724ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a36449640 1 -- 172.21.15.116:0/3940410989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f716af0 msgr2=0x56473f725400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a36449640 1 --2- 172.21.15.116:0/3940410989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f716af0 0x56473f725400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.787+0000 7f6a36449640 1 -- 172.21.15.116:0/3940410989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56473f726a00 con 0x56473f71bc60 2024-03-19T01:18:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.788+0000 7f6a36449640 1 --2- 172.21.15.116:0/3940410989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f71bc60 0x56473f726410 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6a28005fd0 tx=0x7f6a28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd053e4ef28db97 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.788+0000 7f6a197fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:18:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.788+0000 7f6a197fa640 1 -- 172.21.15.116:0/3940410989 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a280045c0 con 0x56473f71bc60 2024-03-19T01:18:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.788+0000 7f6a197fa640 1 -- 172.21.15.116:0/3940410989 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6a28004760 con 0x56473f71bc60 2024-03-19T01:18:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.788+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3940410989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a200bcda0 con 0x56473f71bc60 2024-03-19T01:18:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.789+0000 7f6a197fa640 1 -- 172.21.15.116:0/3940410989 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f6a28004b20 con 0x56473f71bc60 2024-03-19T01:18:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.789+0000 7f6a3ab75740 10 client.15801 fetch_fsmap learned FSMap version 45 2024-03-19T01:18:42.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.789+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3940410989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56473f7921e0 con 0x56473f71bc60 2024-03-19T01:18:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.789+0000 7f6a197fa640 1 -- 172.21.15.116:0/3940410989 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a200bcda0 con 0x56473f71bc60 2024-03-19T01:18:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.789+0000 7f6a197fa640 1 -- 172.21.15.116:0/3940410989 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f6a2801e430 con 0x56473f71bc60 2024-03-19T01:18:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.789+0000 7f6a3ab75740 10 client.15801 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:18:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.789+0000 7f6a3ab75740 10 client.15801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:18:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.789+0000 7f6a3ab75740 20 client.15801 populate_metadata read hostname 'smithi116' 2024-03-19T01:18:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.789+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/3940410989 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56473f792c90 0x56473f7950b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.790+0000 7f6a3ab75740 4 client.15801 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:18:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.790+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3940410989 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56473f682470 con 0x56473f792c90 2024-03-19T01:18:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.790+0000 7f6a35c48640 1 --2- 172.21.15.116:0/3940410989 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56473f792c90 0x56473f7950b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.790+0000 7f6a35c48640 1 --2- 172.21.15.116:0/3940410989 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56473f792c90 0x56473f7950b0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5118ca5820b6df38 server_cookie=bd4ad2cb01819f09 in_seq=0 out_seq=0 2024-03-19T01:18:42.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.790+0000 7f6a197fa640 10 client.15801 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.798+0000 7f6a197fa640 1 -- 172.21.15.116:0/3940410989 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56473f682470 con 0x56473f792c90 2024-03-19T01:18:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.798+0000 7f6a197fa640 10 client.15801 handle_command_reply: tid=0 2024-03-19T01:18:42.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.798+0000 7f6a3ab75740 1 client.15801 shutdown 2024-03-19T01:18:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.798+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3940410989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f71bc60 msgr2=0x56473f726410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.798+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/3940410989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f71bc60 0x56473f726410 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6a28005fd0 tx=0x7f6a28004a40 comp rx=0 tx=0).stop 2024-03-19T01:18:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3940410989 shutdown_connections 2024-03-19T01:18:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/3940410989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56473f71bc60 0x56473f726410 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/3940410989 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56473f716af0 0x56473f725400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/3940410989 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56473f792c90 0x56473f7950b0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a3ab75740 1 --2- 172.21.15.116:0/3940410989 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56473f716100 0x56473f724ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a197fa640 0 client.15801 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3940410989 >> 172.21.15.116:0/3940410989 conn(0x56473f711670 msgr2=0x56473f717380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3940410989 shutdown_connections 2024-03-19T01:18:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a3ab75740 1 -- 172.21.15.116:0/3940410989 wait complete. 2024-03-19T01:18:42.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a3ab75740 20 client.15801 trim_cache size 0 max 16384 2024-03-19T01:18:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:18:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:18:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:18:42.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:18:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a37eac640 1 -- 172.21.15.116:0/3588644965 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6a1c0a2ab0 msgr2=0x7f6a1c0a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a37eac640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6a1c0a2ab0 0x7f6a1c0a4f70 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x56473f717350 tx=0x7f6a28009500 comp rx=0 tx=0).stop 2024-03-19T01:18:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a37eac640 1 -- 172.21.15.116:0/3588644965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30101930 msgr2=0x7f6a3019e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a37eac640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30101930 0x7f6a3019e650 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009230 tx=0x7f6a2002c650 comp rx=0 tx=0).stop 2024-03-19T01:18:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a37eac640 1 -- 172.21.15.116:0/3588644965 shutdown_connections 2024-03-19T01:18:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a37eac640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6a301022f0 0x7f6a301a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a37eac640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6a30101930 0x7f6a3019e650 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a37eac640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6a1c0a2ab0 0x7f6a1c0a4f70 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.799+0000 7f6a37eac640 1 --2- 172.21.15.116:0/3588644965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a30100740 0x7f6a3019e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.800+0000 7f6a37eac640 1 -- 172.21.15.116:0/3588644965 >> 172.21.15.116:0/3588644965 conn(0x7f6a300fbcb0 msgr2=0x7f6a3010dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.800+0000 7f6a37eac640 1 -- 172.21.15.116:0/3588644965 shutdown_connections 2024-03-19T01:18:42.803 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:42.800+0000 7f6a37eac640 1 -- 172.21.15.116:0/3588644965 wait complete. 2024-03-19T01:18:43.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:43 smithi162 ceph-mon[67178]: pgmap v535: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 16 MiB/s wr, 717 op/s 2024-03-19T01:18:43.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:43 smithi116 ceph-mon[62701]: pgmap v535: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 16 MiB/s wr, 717 op/s 2024-03-19T01:18:43.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:43 smithi156 ceph-mon[66799]: pgmap v535: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 16 MiB/s wr, 717 op/s 2024-03-19T01:18:45.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:45 smithi162 ceph-mon[67178]: pgmap v536: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 21 MiB/s wr, 863 op/s 2024-03-19T01:18:45.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:45 smithi116 ceph-mon[62701]: pgmap v536: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 21 MiB/s wr, 863 op/s 2024-03-19T01:18:45.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:45 smithi156 ceph-mon[66799]: pgmap v536: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 21 MiB/s wr, 863 op/s 2024-03-19T01:18:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 360.0 s, 2352.2 tps, lat 42.477 ms stddev 47.882 2024-03-19T01:18:47.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:47 smithi162 ceph-mon[67178]: pgmap v537: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 20 MiB/s wr, 732 op/s 2024-03-19T01:18:47.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:47 smithi116 ceph-mon[62701]: pgmap v537: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 20 MiB/s wr, 732 op/s 2024-03-19T01:18:47.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:47 smithi156 ceph-mon[66799]: pgmap v537: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 20 MiB/s wr, 732 op/s 2024-03-19T01:18:49.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:49 smithi162 ceph-mon[67178]: pgmap v538: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 21 MiB/s wr, 758 op/s 2024-03-19T01:18:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:49 smithi116 ceph-mon[62701]: pgmap v538: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 21 MiB/s wr, 758 op/s 2024-03-19T01:18:49.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:49 smithi156 ceph-mon[66799]: pgmap v538: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 21 MiB/s wr, 758 op/s 2024-03-19T01:18:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 365.0 s, 2462.8 tps, lat 40.302 ms stddev 47.247 2024-03-19T01:18:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:51 smithi162 ceph-mon[67178]: pgmap v539: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 696 op/s 2024-03-19T01:18:51.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:51 smithi116 ceph-mon[62701]: pgmap v539: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 696 op/s 2024-03-19T01:18:51.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:51 smithi156 ceph-mon[66799]: pgmap v539: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 18 MiB/s wr, 696 op/s 2024-03-19T01:18:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:53 smithi162 ceph-mon[67178]: pgmap v540: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 MiB/s wr, 629 op/s 2024-03-19T01:18:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:53 smithi116 ceph-mon[62701]: pgmap v540: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 MiB/s wr, 629 op/s 2024-03-19T01:18:53.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:53 smithi156 ceph-mon[66799]: pgmap v540: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 17 MiB/s wr, 629 op/s 2024-03-19T01:18:55.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:55 smithi162 ceph-mon[67178]: pgmap v541: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 MiB/s wr, 831 op/s 2024-03-19T01:18:55.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:55 smithi116 ceph-mon[62701]: pgmap v541: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 MiB/s wr, 831 op/s 2024-03-19T01:18:55.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:55 smithi156 ceph-mon[66799]: pgmap v541: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 20 MiB/s wr, 831 op/s 2024-03-19T01:18:55.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.927+0000 7f6356eae640 1 -- 172.21.15.116:0/3932621020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6350101850 msgr2=0x7f6350101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.927+0000 7f6356eae640 1 --2- 172.21.15.116:0/3932621020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6350101850 0x7f6350101c70 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6344009900 tx=0x7f634402bc30 comp rx=0 tx=0).stop 2024-03-19T01:18:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.927+0000 7f6356eae640 1 -- 172.21.15.116:0/3932621020 shutdown_connections 2024-03-19T01:18:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.927+0000 7f6356eae640 1 --2- 172.21.15.116:0/3932621020 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6350103400 0x7f6350109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.927+0000 7f6356eae640 1 --2- 172.21.15.116:0/3932621020 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6350102a40 0x7f6350102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.927+0000 7f6356eae640 1 --2- 172.21.15.116:0/3932621020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6350101850 0x7f6350101c70 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.927+0000 7f6356eae640 1 -- 172.21.15.116:0/3932621020 >> 172.21.15.116:0/3932621020 conn(0x7f63500fcde0 msgr2=0x7f63500ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.927+0000 7f6356eae640 1 -- 172.21.15.116:0/3932621020 shutdown_connections 2024-03-19T01:18:55.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.927+0000 7f6356eae640 1 -- 172.21.15.116:0/3932621020 wait complete. 2024-03-19T01:18:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.927+0000 7f6356eae640 1 Processor -- start 2024-03-19T01:18:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f6356eae640 1 -- start start 2024-03-19T01:18:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f6356eae640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6350101850 0x7f6350197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f6356eae640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6350102a40 0x7f6350197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f6356eae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6350103400 0x7f635019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f6356eae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f635010d4f0 con 0x7f6350103400 2024-03-19T01:18:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f6356eae640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f635010d370 con 0x7f6350101850 2024-03-19T01:18:55.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f6356eae640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f635010d670 con 0x7f6350102a40 2024-03-19T01:18:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f634ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6350102a40 0x7f6350197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f634ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6350102a40 0x7f6350197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44434/0 (socket says 172.21.15.116:44434) 2024-03-19T01:18:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f634ffff640 1 -- 172.21.15.116:0/4006429083 learned_addr learned my addr 172.21.15.116:0/4006429083 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f6354c4a640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6350101850 0x7f6350197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f635544b640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6350103400 0x7f635019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f634ffff640 1 -- 172.21.15.116:0/4006429083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6350101850 msgr2=0x7f6350197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f634ffff640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6350101850 0x7f6350197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f634ffff640 1 -- 172.21.15.116:0/4006429083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6350103400 msgr2=0x7f635019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f634ffff640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6350103400 0x7f635019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.928+0000 7f634ffff640 1 -- 172.21.15.116:0/4006429083 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f635019f6c0 con 0x7f6350102a40 2024-03-19T01:18:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.929+0000 7f634ffff640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6350102a40 0x7f6350197fe0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f634000e9f0 tx=0x7f634000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.929+0000 7f634dffb640 1 -- 172.21.15.116:0/4006429083 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634000cd80 con 0x7f6350102a40 2024-03-19T01:18:55.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.929+0000 7f634dffb640 1 -- 172.21.15.116:0/4006429083 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6340010910 con 0x7f6350102a40 2024-03-19T01:18:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.929+0000 7f634dffb640 1 -- 172.21.15.116:0/4006429083 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340018990 con 0x7f6350102a40 2024-03-19T01:18:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.930+0000 7f6356eae640 1 -- 172.21.15.116:0/4006429083 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f635019f9b0 con 0x7f6350102a40 2024-03-19T01:18:55.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.930+0000 7f6356eae640 1 -- 172.21.15.116:0/4006429083 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f635019feb0 con 0x7f6350102a40 2024-03-19T01:18:55.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.933+0000 7f6359b70740 1 Processor -- start 2024-03-19T01:18:55.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.933+0000 7f6359b70740 1 -- start start 2024-03-19T01:18:55.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.933+0000 7f6359b70740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56278398c100 0x56278398c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.933+0000 7f6359b70740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x5627839915f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.934+0000 7f6359b70740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562783996070 0x562783998500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.934+0000 7f6359b70740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5627838f0be0 con 0x56278398c100 2024-03-19T01:18:55.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.934+0000 7f6359b70740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562783899960 con 0x56278398caf0 2024-03-19T01:18:55.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.934+0000 7f6359b70740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5627838ecc80 con 0x562783996070 2024-03-19T01:18:55.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.934+0000 7f634dffb640 1 -- 172.21.15.116:0/4006429083 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6340010ac0 con 0x7f6350102a40 2024-03-19T01:18:55.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.934+0000 7f6354c4a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x5627839915f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.934+0000 7f6354c4a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x5627839915f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:38116/0 (socket says 172.21.15.116:38116) 2024-03-19T01:18:55.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.934+0000 7f6354c4a640 1 -- 172.21.15.116:0/3416322484 learned_addr learned my addr 172.21.15.116:0/3416322484 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:55.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.934+0000 7f635544b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56278398c100 0x56278398c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.935+0000 7f634ffff640 1 --2- 172.21.15.116:0/3416322484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562783996070 0x562783998500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.935+0000 7f6354c4a640 1 -- 172.21.15.116:0/3416322484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562783996070 msgr2=0x562783998500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:55.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.935+0000 7f634dffb640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f63380a2a90 0x7f63380a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.935+0000 7f6354c4a640 1 --2- 172.21.15.116:0/3416322484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562783996070 0x562783998500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.935+0000 7f6354c4a640 1 -- 172.21.15.116:0/3416322484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56278398c100 msgr2=0x56278398c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:55.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.935+0000 7f6354c4a640 1 --2- 172.21.15.116:0/3416322484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56278398c100 0x56278398c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.935+0000 7f6354c4a640 1 -- 172.21.15.116:0/3416322484 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562783890d20 con 0x56278398caf0 2024-03-19T01:18:55.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.935+0000 7f635544b640 1 --2- 172.21.15.116:0/3416322484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56278398c100 0x56278398c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:18:55.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.935+0000 7f635544b640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f63380a2a90 0x7f63380a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.935+0000 7f634dffb640 1 -- 172.21.15.116:0/4006429083 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f63400cee40 con 0x7f6350102a40 2024-03-19T01:18:55.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.935+0000 7f634ffff640 1 --2- 172.21.15.116:0/3416322484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562783996070 0x562783998500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:18:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.936+0000 7f6354c4a640 1 --2- 172.21.15.116:0/3416322484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x5627839915f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f634402c8e0 tx=0x7f6344009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a130d78c966de6a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:55.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.936+0000 7f6336ffd640 1 -- 172.21.15.116:0/3416322484 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6344044020 con 0x56278398caf0 2024-03-19T01:18:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.936+0000 7f635544b640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f63380a2a90 0x7f63380a4f50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6348009fd0 tx=0x7f6348004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.936+0000 7f6336ffd640 1 -- 172.21.15.116:0/3416322484 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f634402cb80 con 0x56278398caf0 2024-03-19T01:18:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.936+0000 7f6336ffd640 1 -- 172.21.15.116:0/3416322484 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6344044020 con 0x56278398caf0 2024-03-19T01:18:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.937+0000 7f6359b70740 1 -- 172.21.15.116:0/3416322484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 msgr2=0x5627839915f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.937+0000 7f6359b70740 1 --2- 172.21.15.116:0/3416322484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x5627839915f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f634402c8e0 tx=0x7f6344009560 comp rx=0 tx=0).stop 2024-03-19T01:18:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.937+0000 7f6359b70740 1 -- 172.21.15.116:0/3416322484 shutdown_connections 2024-03-19T01:18:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.937+0000 7f6359b70740 1 --2- 172.21.15.116:0/3416322484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562783996070 0x562783998500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.937+0000 7f6359b70740 1 --2- 172.21.15.116:0/3416322484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x5627839915f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.937+0000 7f6359b70740 1 --2- 172.21.15.116:0/3416322484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56278398c100 0x56278398c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.937+0000 7f6359b70740 1 -- 172.21.15.116:0/3416322484 >> 172.21.15.116:0/3416322484 conn(0x562783987670 msgr2=0x562783987a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.938+0000 7f6359b70740 1 -- 172.21.15.116:0/3416322484 shutdown_connections 2024-03-19T01:18:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.938+0000 7f6359b70740 1 -- 172.21.15.116:0/3416322484 wait complete. 2024-03-19T01:18:55.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.938+0000 7f6359b70740 1 -- start start 2024-03-19T01:18:55.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.938+0000 7f6359b70740 1 Processor -- start 2024-03-19T01:18:55.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.939+0000 7f6359b70740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56278398c100 0x56278398ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.939+0000 7f6359b70740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x56278398f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.939+0000 7f6359b70740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562783996070 0x562783990480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.939+0000 7f6359b70740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562783991f30 con 0x562783996070 2024-03-19T01:18:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.939+0000 7f6359b70740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5627838acdb0 con 0x56278398caf0 2024-03-19T01:18:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.939+0000 7f6359b70740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5627839920b0 con 0x56278398c100 2024-03-19T01:18:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.939+0000 7f6359b70740 10 client.0 _ll_register_callbacks cb 0x5627838b1e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:18:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.940+0000 7f6354c4a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56278398c100 0x56278398ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.940+0000 7f6354c4a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56278398c100 0x56278398ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44452/0 (socket says 172.21.15.116:44452) 2024-03-19T01:18:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.940+0000 7f6354c4a640 1 -- 172.21.15.116:0/333442132 learned_addr learned my addr 172.21.15.116:0/333442132 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.940+0000 7f634ffff640 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x56278398f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.940+0000 7f635544b640 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562783996070 0x562783990480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.940+0000 7f634ffff640 1 -- 172.21.15.116:0/333442132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56278398c100 msgr2=0x56278398ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:55.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.940+0000 7f634ffff640 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56278398c100 0x56278398ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.940+0000 7f634ffff640 1 -- 172.21.15.116:0/333442132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562783996070 msgr2=0x562783990480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:55.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.940+0000 7f634ffff640 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562783996070 0x562783990480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.940+0000 7f634ffff640 1 -- 172.21.15.116:0/333442132 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627838abc60 con 0x56278398caf0 2024-03-19T01:18:55.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.940+0000 7f635544b640 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562783996070 0x562783990480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:18:55.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.940+0000 7f634ffff640 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x56278398f470 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6340098130 tx=0x7f634009be60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=621d59b9e1678901 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:55.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.941+0000 7f632ffff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:18:55.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.941+0000 7f632ffff640 1 -- 172.21.15.116:0/333442132 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63400afd40 con 0x56278398caf0 2024-03-19T01:18:55.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.941+0000 7f632ffff640 1 -- 172.21.15.116:0/333442132 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6340018b30 con 0x56278398caf0 2024-03-19T01:18:55.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.941+0000 7f6359b70740 1 -- 172.21.15.116:0/333442132 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f634402cb80 con 0x56278398caf0 2024-03-19T01:18:55.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.942+0000 7f632ffff640 1 -- 172.21.15.116:0/333442132 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f63400a3f00 con 0x56278398caf0 2024-03-19T01:18:55.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.942+0000 7f6359b70740 10 client.25649 fetch_fsmap learned FSMap version 45 2024-03-19T01:18:55.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.942+0000 7f6359b70740 1 -- 172.21.15.116:0/333442132 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x562783605210 con 0x56278398caf0 2024-03-19T01:18:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.942+0000 7f632ffff640 1 -- 172.21.15.116:0/333442132 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f634402cb80 con 0x56278398caf0 2024-03-19T01:18:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.942+0000 7f632ffff640 1 -- 172.21.15.116:0/333442132 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f63400975c0 con 0x56278398caf0 2024-03-19T01:18:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.942+0000 7f6359b70740 10 client.25649 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:18:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.942+0000 7f6359b70740 10 client.25649 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:18:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.942+0000 7f6359b70740 20 client.25649 populate_metadata read hostname 'smithi116' 2024-03-19T01:18:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.942+0000 7f6359b70740 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5627839aab40 0x5627839acfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.942+0000 7f6359b70740 4 client.25649 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:18:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.942+0000 7f6359b70740 1 -- 172.21.15.116:0/333442132 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627838fe120 con 0x5627839aab40 2024-03-19T01:18:55.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.942+0000 7f6354c4a640 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5627839aab40 0x5627839acfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.944+0000 7f6354c4a640 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5627839aab40 0x5627839acfc0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6751366a6c8191f server_cookie=db1084234667cdf5 in_seq=0 out_seq=0 2024-03-19T01:18:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.944+0000 7f632ffff640 10 client.25649 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:55.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.945+0000 7f632ffff640 1 -- 172.21.15.116:0/333442132 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5627838fe120 con 0x5627839aab40 2024-03-19T01:18:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.945+0000 7f632ffff640 10 client.25649 handle_command_reply: tid=0 2024-03-19T01:18:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.945+0000 7f6359b70740 1 client.25649 shutdown 2024-03-19T01:18:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.945+0000 7f6359b70740 1 -- 172.21.15.116:0/333442132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 msgr2=0x56278398f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.945+0000 7f6359b70740 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x56278398f470 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6340098130 tx=0x7f634009be60 comp rx=0 tx=0).stop 2024-03-19T01:18:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.945+0000 7f6359b70740 1 -- 172.21.15.116:0/333442132 shutdown_connections 2024-03-19T01:18:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.945+0000 7f6359b70740 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562783996070 0x562783990480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.945+0000 7f6359b70740 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x56278398f470 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.945+0000 7f6359b70740 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5627839aab40 0x5627839acfc0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.945+0000 7f6359b70740 1 --2- 172.21.15.116:0/333442132 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56278398c100 0x56278398ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.945+0000 7f632ffff640 0 client.25649 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:55.946 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.945+0000 7f6359b70740 1 -- 172.21.15.116:0/333442132 >> 172.21.15.116:0/333442132 conn(0x562783987670 msgr2=0x5627839888c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.946+0000 7f6359b70740 1 -- 172.21.15.116:0/333442132 shutdown_connections 2024-03-19T01:18:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.946+0000 7f6359b70740 1 -- 172.21.15.116:0/333442132 wait complete. 2024-03-19T01:18:55.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.946+0000 7f6359b70740 20 client.25649 trim_cache size 0 max 16384 2024-03-19T01:18:55.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.968+0000 7f6359b70740 1 Processor -- start 2024-03-19T01:18:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.968+0000 7f6359b70740 1 -- start start 2024-03-19T01:18:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.968+0000 7f6359b70740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398c100 0x562783986ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.968+0000 7f6359b70740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56278398caf0 0x562783992450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.968+0000 7f6359b70740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562783996070 0x562783992990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.968+0000 7f6359b70740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f63501a6bc0 con 0x56278398caf0 2024-03-19T01:18:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.968+0000 7f6359b70740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f63501a8700 con 0x56278398c100 2024-03-19T01:18:55.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.968+0000 7f6359b70740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f63501a7290 con 0x562783996070 2024-03-19T01:18:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.969+0000 7f635544b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562783996070 0x562783992990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.969+0000 7f635544b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562783996070 0x562783992990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:44468/0 (socket says 172.21.15.116:44468) 2024-03-19T01:18:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.969+0000 7f635544b640 1 -- 172.21.15.116:0/3924855970 learned_addr learned my addr 172.21.15.116:0/3924855970 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.969+0000 7f6354c4a640 1 --2- 172.21.15.116:0/3924855970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398c100 0x562783986ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.969+0000 7f634ffff640 1 --2- 172.21.15.116:0/3924855970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56278398caf0 0x562783992450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.969+0000 7f635544b640 1 -- 172.21.15.116:0/3924855970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398c100 msgr2=0x562783986ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.969+0000 7f635544b640 1 --2- 172.21.15.116:0/3924855970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398c100 0x562783986ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.969+0000 7f635544b640 1 -- 172.21.15.116:0/3924855970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56278398caf0 msgr2=0x562783992450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.969+0000 7f635544b640 1 --2- 172.21.15.116:0/3924855970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56278398caf0 0x562783992450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.969+0000 7f635544b640 1 -- 172.21.15.116:0/3924855970 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562783982ee0 con 0x562783996070 2024-03-19T01:18:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.969+0000 7f634ffff640 1 --2- 172.21.15.116:0/3924855970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56278398caf0 0x562783992450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:18:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.969+0000 7f635544b640 1 --2- 172.21.15.116:0/3924855970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562783996070 0x562783992990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6348013280 tx=0x7f6348016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b12e0a2e46877fe0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.970+0000 7f63367fc640 1 -- 172.21.15.116:0/3924855970 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6348003b70 con 0x562783996070 2024-03-19T01:18:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.970+0000 7f63367fc640 1 -- 172.21.15.116:0/3924855970 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6348003d10 con 0x562783996070 2024-03-19T01:18:55.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.971+0000 7f63367fc640 1 -- 172.21.15.116:0/3924855970 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6348011ad0 con 0x562783996070 2024-03-19T01:18:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.972+0000 7f6359b70740 1 -- 172.21.15.116:0/3924855970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562783996070 msgr2=0x562783992990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:55.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.972+0000 7f6359b70740 1 --2- 172.21.15.116:0/3924855970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562783996070 0x562783992990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6348013280 tx=0x7f6348016000 comp rx=0 tx=0).stop 2024-03-19T01:18:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.976+0000 7f6359b70740 1 -- 172.21.15.116:0/3924855970 shutdown_connections 2024-03-19T01:18:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.976+0000 7f6359b70740 1 --2- 172.21.15.116:0/3924855970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562783996070 0x562783992990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.976+0000 7f6359b70740 1 --2- 172.21.15.116:0/3924855970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56278398caf0 0x562783992450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.976+0000 7f6359b70740 1 --2- 172.21.15.116:0/3924855970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398c100 0x562783986ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.976+0000 7f6359b70740 1 -- 172.21.15.116:0/3924855970 >> 172.21.15.116:0/3924855970 conn(0x562783987670 msgr2=0x56278398d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.977+0000 7f6359b70740 1 -- 172.21.15.116:0/3924855970 shutdown_connections 2024-03-19T01:18:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.978+0000 7f6359b70740 1 -- 172.21.15.116:0/3924855970 wait complete. 2024-03-19T01:18:55.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.978+0000 7f6359b70740 1 -- start start 2024-03-19T01:18:55.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.978+0000 7f6359b70740 1 Processor -- start 2024-03-19T01:18:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.979+0000 7f6359b70740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56278398c100 0x5627839a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.979+0000 7f6359b70740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x5627839a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.979+0000 7f6359b70740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562783996070 0x5627839a63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.979+0000 7f6359b70740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f63501a8030 con 0x562783996070 2024-03-19T01:18:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.979+0000 7f6359b70740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f63501a7960 con 0x56278398caf0 2024-03-19T01:18:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.979+0000 7f6359b70740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f63501a4ba0 con 0x56278398c100 2024-03-19T01:18:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.979+0000 7f6359b70740 10 client.0 _ll_register_callbacks cb 0x7f634000cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:18:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.979+0000 7f635544b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562783996070 0x5627839a63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.979+0000 7f635544b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562783996070 0x5627839a63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55546/0 (socket says 172.21.15.116:55546) 2024-03-19T01:18:55.980 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.979+0000 7f635544b640 1 -- 172.21.15.116:0/3979896909 learned_addr learned my addr 172.21.15.116:0/3979896909 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:18:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.982+0000 7f634ffff640 1 --2- 172.21.15.116:0/3979896909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x5627839a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.982+0000 7f6354c4a640 1 --2- 172.21.15.116:0/3979896909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56278398c100 0x5627839a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.983+0000 7f635544b640 1 -- 172.21.15.116:0/3979896909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56278398c100 msgr2=0x5627839a4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.983+0000 7f635544b640 1 --2- 172.21.15.116:0/3979896909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56278398c100 0x5627839a4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.983+0000 7f635544b640 1 -- 172.21.15.116:0/3979896909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 msgr2=0x5627839a53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:18:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.983+0000 7f635544b640 1 --2- 172.21.15.116:0/3979896909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x5627839a53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.983+0000 7f635544b640 1 -- 172.21.15.116:0/3979896909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627836d1620 con 0x562783996070 2024-03-19T01:18:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.984+0000 7f635544b640 1 --2- 172.21.15.116:0/3979896909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562783996070 0x5627839a63d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6348009fa0 tx=0x7f6348004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa187aabb1fd087 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:18:55.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.984+0000 7f632ffff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:18:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.987+0000 7f632ffff640 1 -- 172.21.15.116:0/3979896909 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6348032060 con 0x562783996070 2024-03-19T01:18:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.987+0000 7f632ffff640 1 -- 172.21.15.116:0/3979896909 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f63480042d0 con 0x562783996070 2024-03-19T01:18:55.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.987+0000 7f6359b70740 1 -- 172.21.15.116:0/3979896909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6348003d10 con 0x562783996070 2024-03-19T01:18:55.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.991+0000 7f632ffff640 1 -- 172.21.15.116:0/3979896909 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f63480044d0 con 0x562783996070 2024-03-19T01:18:55.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.992+0000 7f6359b70740 10 client.15819 fetch_fsmap learned FSMap version 45 2024-03-19T01:18:55.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.992+0000 7f6359b70740 1 -- 172.21.15.116:0/3979896909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x562783999300 con 0x562783996070 2024-03-19T01:18:55.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.993+0000 7f632ffff640 1 -- 172.21.15.116:0/3979896909 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6348003d10 con 0x562783996070 2024-03-19T01:18:55.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.993+0000 7f632ffff640 1 -- 172.21.15.116:0/3979896909 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f6348027020 con 0x562783996070 2024-03-19T01:18:55.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.993+0000 7f6359b70740 10 client.15819 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:18:55.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.993+0000 7f6359b70740 10 client.15819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:18:55.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.993+0000 7f6359b70740 20 client.15819 populate_metadata read hostname 'smithi116' 2024-03-19T01:18:55.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.993+0000 7f6359b70740 1 --2- 172.21.15.116:0/3979896909 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562783999b60 0x56278399bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:18:55.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.993+0000 7f6359b70740 4 client.15819 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:18:55.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.993+0000 7f6359b70740 1 -- 172.21.15.116:0/3979896909 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5627838f8470 con 0x562783999b60 2024-03-19T01:18:55.995 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:55.995+0000 7f6354c4a640 1 --2- 172.21.15.116:0/3979896909 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562783999b60 0x56278399bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:18:56.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.012+0000 7f6354c4a640 1 --2- 172.21.15.116:0/3979896909 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562783999b60 0x56278399bf80 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e75ab2c649d9e09 server_cookie=584578e010f0f1e4 in_seq=0 out_seq=0 2024-03-19T01:18:56.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.013+0000 7f632ffff640 10 client.15819 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:56.038 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.038+0000 7f632ffff640 1 -- 172.21.15.116:0/3979896909 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5627838f8470 con 0x562783999b60 2024-03-19T01:18:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.038+0000 7f632ffff640 10 client.15819 handle_command_reply: tid=0 2024-03-19T01:18:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.038+0000 7f6359b70740 1 client.15819 shutdown 2024-03-19T01:18:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.038+0000 7f6359b70740 1 -- 172.21.15.116:0/3979896909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562783996070 msgr2=0x5627839a63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.038+0000 7f6359b70740 1 --2- 172.21.15.116:0/3979896909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562783996070 0x5627839a63d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6348009fa0 tx=0x7f6348004ab0 comp rx=0 tx=0).stop 2024-03-19T01:18:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.038+0000 7f6359b70740 1 -- 172.21.15.116:0/3979896909 shutdown_connections 2024-03-19T01:18:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.038+0000 7f6359b70740 1 --2- 172.21.15.116:0/3979896909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562783996070 0x5627839a63d0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.038+0000 7f6359b70740 1 --2- 172.21.15.116:0/3979896909 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56278398caf0 0x5627839a53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.038+0000 7f6359b70740 1 --2- 172.21.15.116:0/3979896909 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562783999b60 0x56278399bf80 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.038+0000 7f6359b70740 1 --2- 172.21.15.116:0/3979896909 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56278398c100 0x5627839a4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.038+0000 7f6359b70740 1 -- 172.21.15.116:0/3979896909 >> 172.21.15.116:0/3979896909 conn(0x562783987670 msgr2=0x56278398a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.038+0000 7f632ffff640 0 client.15819 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:18:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.039+0000 7f6359b70740 1 -- 172.21.15.116:0/3979896909 shutdown_connections 2024-03-19T01:18:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.039+0000 7f6359b70740 1 -- 172.21.15.116:0/3979896909 wait complete. 2024-03-19T01:18:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.039+0000 7f6359b70740 20 client.15819 trim_cache size 0 max 16384 2024-03-19T01:18:56.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:18:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:18:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:18:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:18:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.040+0000 7f6356eae640 1 -- 172.21.15.116:0/4006429083 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f63380a2a90 msgr2=0x7f63380a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.040+0000 7f6356eae640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f63380a2a90 0x7f63380a4f50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6348009fd0 tx=0x7f6348004a40 comp rx=0 tx=0).stop 2024-03-19T01:18:56.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.040+0000 7f6356eae640 1 -- 172.21.15.116:0/4006429083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6350102a40 msgr2=0x7f6350197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:18:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.040+0000 7f6356eae640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6350102a40 0x7f6350197fe0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f634000e9f0 tx=0x7f634000eeb0 comp rx=0 tx=0).stop 2024-03-19T01:18:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.040+0000 7f6356eae640 1 -- 172.21.15.116:0/4006429083 shutdown_connections 2024-03-19T01:18:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.040+0000 7f6356eae640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6350103400 0x7f635019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.040+0000 7f6356eae640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6350102a40 0x7f6350197fe0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.040+0000 7f6356eae640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f63380a2a90 0x7f63380a4f50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.040+0000 7f6356eae640 1 --2- 172.21.15.116:0/4006429083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6350101850 0x7f6350197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:18:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.040+0000 7f6356eae640 1 -- 172.21.15.116:0/4006429083 >> 172.21.15.116:0/4006429083 conn(0x7f63500fcde0 msgr2=0x7f6350105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:18:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.040+0000 7f6356eae640 1 -- 172.21.15.116:0/4006429083 shutdown_connections 2024-03-19T01:18:56.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:18:56.040+0000 7f6356eae640 1 -- 172.21.15.116:0/4006429083 wait complete. 2024-03-19T01:18:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 370.0 s, 2524.4 tps, lat 40.024 ms stddev 49.433 2024-03-19T01:18:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:57 smithi162 ceph-mon[67178]: pgmap v542: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 16 MiB/s wr, 753 op/s 2024-03-19T01:18:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:57 smithi116 ceph-mon[62701]: pgmap v542: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 16 MiB/s wr, 753 op/s 2024-03-19T01:18:57.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:57 smithi156 ceph-mon[66799]: pgmap v542: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 16 MiB/s wr, 753 op/s 2024-03-19T01:18:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:18:59 smithi162 ceph-mon[67178]: pgmap v543: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 MiB/s wr, 818 op/s 2024-03-19T01:18:59.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:18:59 smithi116 ceph-mon[62701]: pgmap v543: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 MiB/s wr, 818 op/s 2024-03-19T01:18:59.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:18:59 smithi156 ceph-mon[66799]: pgmap v543: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 MiB/s wr, 818 op/s 2024-03-19T01:19:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 375.0 s, 2315.6 tps, lat 42.294 ms stddev 47.374 2024-03-19T01:19:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:01 smithi162 ceph-mon[67178]: pgmap v544: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 16 MiB/s wr, 793 op/s 2024-03-19T01:19:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:01 smithi116 ceph-mon[62701]: pgmap v544: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 16 MiB/s wr, 793 op/s 2024-03-19T01:19:01.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:01 smithi156 ceph-mon[66799]: pgmap v544: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 16 MiB/s wr, 793 op/s 2024-03-19T01:19:02.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:02 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:19:02.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:02 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:19:02.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:02 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:19:03.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:03 smithi116 ceph-mon[62701]: pgmap v545: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 18 MiB/s wr, 838 op/s 2024-03-19T01:19:03.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:03 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:19:03.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:03 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:19:03.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:03 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:19:03.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:03 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:19:03.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:03 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:19:03.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:03 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:19:03.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:03 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:19:03.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:03 smithi156 ceph-mon[66799]: pgmap v545: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 18 MiB/s wr, 838 op/s 2024-03-19T01:19:03.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:03 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:19:03.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:03 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:19:03.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:03 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:19:03.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:03 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:19:03.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:03 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:19:03.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:03 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:19:03.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:03 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:19:03.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:03 smithi162 ceph-mon[67178]: pgmap v545: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 18 MiB/s wr, 838 op/s 2024-03-19T01:19:03.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:03 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:19:03.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:03 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:19:03.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:03 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:19:03.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:03 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:19:03.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:03 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:19:03.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:03 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:19:03.900 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:03 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:19:04.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:04 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-19T01:19:04.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:04 smithi156 ceph-mon[66799]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-19T01:19:04.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:04 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-19T01:19:04.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:04 smithi162 ceph-mon[67178]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-19T01:19:04.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:04 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi116 to 176.2M 2024-03-19T01:19:04.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:04 smithi116 ceph-mon[62701]: Unable to set osd_memory_target on smithi116 to 184784588: error parsing value: Value '184784588' is below minimum 939524096 2024-03-19T01:19:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:05 smithi156 ceph-mon[66799]: pgmap v546: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 35 MiB/s wr, 1.48k op/s 2024-03-19T01:19:05.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:05 smithi162 ceph-mon[67178]: pgmap v546: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 35 MiB/s wr, 1.48k op/s 2024-03-19T01:19:05.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:05 smithi116 ceph-mon[62701]: pgmap v546: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 35 MiB/s wr, 1.48k op/s 2024-03-19T01:19:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 380.0 s, 2183.0 tps, lat 46.752 ms stddev 109.797 2024-03-19T01:19:07.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:07 smithi162 ceph-mon[67178]: pgmap v547: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 MiB/s wr, 1.57k op/s 2024-03-19T01:19:07.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:07 smithi116 ceph-mon[62701]: pgmap v547: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 MiB/s wr, 1.57k op/s 2024-03-19T01:19:07.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:07 smithi156 ceph-mon[66799]: pgmap v547: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 MiB/s wr, 1.57k op/s 2024-03-19T01:19:09.180 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.178+0000 7f94c67fc640 1 -- 172.21.15.116:0/1499850721 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c0032990 con 0x7f94c81011f0 2024-03-19T01:19:09.180 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.179+0000 7f94ce044640 1 -- 172.21.15.116:0/1499850721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94c81011f0 msgr2=0x7f94c8105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:09.180 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.179+0000 7f94ce044640 1 --2- 172.21.15.116:0/1499850721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94c81011f0 0x7f94c8105a90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f94c0009ca0 tx=0x7f94c002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.180 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.179+0000 7f94ce044640 1 -- 172.21.15.116:0/1499850721 shutdown_connections 2024-03-19T01:19:09.180 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.179+0000 7f94ce044640 1 --2- 172.21.15.116:0/1499850721 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94c81011f0 0x7f94c8105a90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.180 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.179+0000 7f94ce044640 1 --2- 172.21.15.116:0/1499850721 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94c8100830 0x7f94c8100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.180 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.179+0000 7f94ce044640 1 --2- 172.21.15.116:0/1499850721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c80ffad0 0x7f94c80ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.179+0000 7f94ce044640 1 -- 172.21.15.116:0/1499850721 >> 172.21.15.116:0/1499850721 conn(0x7f94c80fb440 msgr2=0x7f94c80fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.179+0000 7f94ce044640 1 -- 172.21.15.116:0/1499850721 shutdown_connections 2024-03-19T01:19:09.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.179+0000 7f94ce044640 1 -- 172.21.15.116:0/1499850721 wait complete. 2024-03-19T01:19:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.180+0000 7f94ce044640 1 Processor -- start 2024-03-19T01:19:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.180+0000 7f94ce044640 1 -- start start 2024-03-19T01:19:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.180+0000 7f94ce044640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94c80ffad0 0x7f94c819e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.180+0000 7f94ce044640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c8100830 0x7f94c819e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.180+0000 7f94ce044640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94c81011f0 0x7f94c81a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.180+0000 7f94ce044640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94c8113b50 con 0x7f94c8100830 2024-03-19T01:19:09.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.180+0000 7f94ce044640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f94c81139d0 con 0x7f94c80ffad0 2024-03-19T01:19:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.180+0000 7f94ce044640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94c8113cd0 con 0x7f94c81011f0 2024-03-19T01:19:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.181+0000 7f94c7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94c81011f0 0x7f94c81a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.181+0000 7f94c7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94c81011f0 0x7f94c81a5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51824/0 (socket says 172.21.15.116:51824) 2024-03-19T01:19:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.181+0000 7f94c7fff640 1 -- 172.21.15.116:0/2926837774 learned_addr learned my addr 172.21.15.116:0/2926837774 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.181+0000 7f94c77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94c80ffad0 0x7f94c819e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.181+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/2926837774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c8100830 0x7f94c819e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.181+0000 7f94c7fff640 1 -- 172.21.15.116:0/2926837774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94c80ffad0 msgr2=0x7f94c819e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.181+0000 7f94c7fff640 1 --2- 172.21.15.116:0/2926837774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94c80ffad0 0x7f94c819e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.181+0000 7f94c7fff640 1 -- 172.21.15.116:0/2926837774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c8100830 msgr2=0x7f94c819e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.181+0000 7f94c7fff640 1 --2- 172.21.15.116:0/2926837774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c8100830 0x7f94c819e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.181+0000 7f94c7fff640 1 -- 172.21.15.116:0/2926837774 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c81a5ca0 con 0x7f94c81011f0 2024-03-19T01:19:09.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.181+0000 7f94c7fff640 1 --2- 172.21.15.116:0/2926837774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94c81011f0 0x7f94c81a5630 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f94c0037040 tx=0x7f94c0032f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:09.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.181+0000 7f94c4ff9640 1 -- 172.21.15.116:0/2926837774 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c0039040 con 0x7f94c81011f0 2024-03-19T01:19:09.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.182+0000 7f94ce044640 1 -- 172.21.15.116:0/2926837774 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c81a5f30 con 0x7f94c81011f0 2024-03-19T01:19:09.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.182+0000 7f94c4ff9640 1 -- 172.21.15.116:0/2926837774 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94c0044070 con 0x7f94c81011f0 2024-03-19T01:19:09.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.182+0000 7f94ce044640 1 -- 172.21.15.116:0/2926837774 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c81a63f0 con 0x7f94c81011f0 2024-03-19T01:19:09.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.182+0000 7f94c4ff9640 1 -- 172.21.15.116:0/2926837774 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c0004260 con 0x7f94c81011f0 2024-03-19T01:19:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94d0c2d740 1 Processor -- start 2024-03-19T01:19:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94c4ff9640 1 -- 172.21.15.116:0/2926837774 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f94c0004560 con 0x7f94c81011f0 2024-03-19T01:19:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94d0c2d740 1 -- start start 2024-03-19T01:19:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94d0c2d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9100 0x56541c6a9520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94d0c2d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9af0 0x56541c6ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94d0c2d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6aec60 0x56541c6b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94d0c2d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56541c60dbe0 con 0x56541c6aec60 2024-03-19T01:19:09.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94d0c2d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56541c5b6960 con 0x56541c6a9100 2024-03-19T01:19:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94d0c2d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56541c609c80 con 0x56541c6a9af0 2024-03-19T01:19:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94c77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9af0 0x56541c6ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94c77fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9af0 0x56541c6ae5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51840/0 (socket says 172.21.15.116:51840) 2024-03-19T01:19:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94c77fe640 1 -- 172.21.15.116:0/1580235623 learned_addr learned my addr 172.21.15.116:0/1580235623 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/1580235623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9100 0x56541c6a9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.185+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/1580235623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6aec60 0x56541c6b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.186+0000 7f94c4ff9640 1 --2- 172.21.15.116:0/2926837774 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f94b00a29a0 0x7f94b00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.186+0000 7f94c77fe640 1 -- 172.21.15.116:0/1580235623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9100 msgr2=0x56541c6a9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.186+0000 7f94c77fe640 1 --2- 172.21.15.116:0/1580235623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9100 0x56541c6a9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.186+0000 7f94c77fe640 1 -- 172.21.15.116:0/1580235623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6aec60 msgr2=0x56541c6b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:09.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.186+0000 7f94c77fe640 1 --2- 172.21.15.116:0/1580235623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6aec60 0x56541c6b10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.186+0000 7f94c77fe640 1 -- 172.21.15.116:0/1580235623 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56541c5add20 con 0x56541c6a9af0 2024-03-19T01:19:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.186+0000 7f94c4ff9640 1 -- 172.21.15.116:0/2926837774 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f94c00f10c0 con 0x7f94c81011f0 2024-03-19T01:19:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.186+0000 7f94c77fe640 1 --2- 172.21.15.116:0/1580235623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9af0 0x56541c6ae5f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009a70 tx=0x7f94b8009f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4003bd9ceb1d2cff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.186+0000 7f94c7fff640 1 --2- 172.21.15.116:0/2926837774 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f94b00a29a0 0x7f94b00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.186+0000 7f94adffb640 1 -- 172.21.15.116:0/1580235623 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8017070 con 0x56541c6a9af0 2024-03-19T01:19:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.186+0000 7f94adffb640 1 -- 172.21.15.116:0/1580235623 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94b8013470 con 0x56541c6a9af0 2024-03-19T01:19:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.187+0000 7f94adffb640 1 -- 172.21.15.116:0/1580235623 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8012630 con 0x56541c6a9af0 2024-03-19T01:19:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.187+0000 7f94d0c2d740 1 -- 172.21.15.116:0/1580235623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9af0 msgr2=0x56541c6ae5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.187+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/1580235623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9af0 0x56541c6ae5f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f94b8009a70 tx=0x7f94b8009f30 comp rx=0 tx=0).stop 2024-03-19T01:19:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.187+0000 7f94c7fff640 1 --2- 172.21.15.116:0/2926837774 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f94b00a29a0 0x7f94b00a4e60 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f94c0009620 tx=0x7f94c003e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.187+0000 7f94d0c2d740 1 -- 172.21.15.116:0/1580235623 shutdown_connections 2024-03-19T01:19:09.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.187+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/1580235623 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6aec60 0x56541c6b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.187+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/1580235623 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9af0 0x56541c6ae5f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.187+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/1580235623 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9100 0x56541c6a9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.187+0000 7f94d0c2d740 1 -- 172.21.15.116:0/1580235623 >> 172.21.15.116:0/1580235623 conn(0x56541c6a4670 msgr2=0x56541c6a4a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.187+0000 7f94d0c2d740 1 -- 172.21.15.116:0/1580235623 shutdown_connections 2024-03-19T01:19:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.187+0000 7f94d0c2d740 1 -- 172.21.15.116:0/1580235623 wait complete. 2024-03-19T01:19:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.188+0000 7f94d0c2d740 1 -- start start 2024-03-19T01:19:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.188+0000 7f94d0c2d740 1 Processor -- start 2024-03-19T01:19:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.188+0000 7f94d0c2d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6a9100 0x56541c6b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.188+0000 7f94d0c2d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9af0 0x56541c6b3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.188+0000 7f94c6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6a9100 0x56541c6b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.188+0000 7f94c6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6a9100 0x56541c6b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45522/0 (socket says 172.21.15.116:45522) 2024-03-19T01:19:09.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.188+0000 7f94c6ffd640 1 -- 172.21.15.116:0/3183665014 learned_addr learned my addr 172.21.15.116:0/3183665014 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.188+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6aec60 0x56541c6ba6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.188+0000 7f94d0c2d740 1 -- 172.21.15.116:0/3183665014 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56541c6b1cb0 con 0x56541c6a9100 2024-03-19T01:19:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94d0c2d740 1 -- 172.21.15.116:0/3183665014 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56541c5c9db0 con 0x56541c6aec60 2024-03-19T01:19:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94d0c2d740 1 -- 172.21.15.116:0/3183665014 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56541c6b1e30 con 0x56541c6a9af0 2024-03-19T01:19:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94c77fe640 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9af0 0x56541c6b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94d0c2d740 10 client.0 _ll_register_callbacks cb 0x56541c5cee70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:19:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94c77fe640 1 -- 172.21.15.116:0/3183665014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6aec60 msgr2=0x56541c6ba6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6aec60 0x56541c6ba6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94c77fe640 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6aec60 0x56541c6ba6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94c77fe640 1 -- 172.21.15.116:0/3183665014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6a9100 msgr2=0x56541c6b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94c77fe640 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6a9100 0x56541c6b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94c77fe640 1 -- 172.21.15.116:0/3183665014 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56541c5c8c60 con 0x56541c6a9af0 2024-03-19T01:19:09.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6aec60 0x56541c6ba6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:19:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94c77fe640 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9af0 0x56541c6b3670 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x56541c6a9ef0 tx=0x7f94b8006e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e9e44d8f6fba4d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:19:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94a6ffd640 1 -- 172.21.15.116:0/3183665014 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8017050 con 0x56541c6a9af0 2024-03-19T01:19:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94a6ffd640 1 -- 172.21.15.116:0/3183665014 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f94b801c450 con 0x56541c6a9af0 2024-03-19T01:19:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.189+0000 7f94d0c2d740 1 -- 172.21.15.116:0/3183665014 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94b8013470 con 0x56541c6a9af0 2024-03-19T01:19:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.190+0000 7f94a6ffd640 1 -- 172.21.15.116:0/3183665014 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f94b801c670 con 0x56541c6a9af0 2024-03-19T01:19:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.190+0000 7f94d0c2d740 10 client.25636 fetch_fsmap learned FSMap version 45 2024-03-19T01:19:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.190+0000 7f94d0c2d740 1 -- 172.21.15.116:0/3183665014 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56541c322210 con 0x56541c6a9af0 2024-03-19T01:19:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.190+0000 7f94a6ffd640 1 -- 172.21.15.116:0/3183665014 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94b8013470 con 0x56541c6a9af0 2024-03-19T01:19:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.190+0000 7f94a6ffd640 1 -- 172.21.15.116:0/3183665014 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f94b8031950 con 0x56541c6a9af0 2024-03-19T01:19:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.190+0000 7f94d0c2d740 10 client.25636 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:19:09.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.190+0000 7f94d0c2d740 10 client.25636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:19:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.190+0000 7f94d0c2d740 20 client.25636 populate_metadata read hostname 'smithi116' 2024-03-19T01:19:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.190+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56541c6bec30 0x56541c6c1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.190+0000 7f94d0c2d740 4 client.25636 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:19:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.190+0000 7f94d0c2d740 1 -- 172.21.15.116:0/3183665014 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56541c61b120 con 0x56541c6bec30 2024-03-19T01:19:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.190+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56541c6bec30 0x56541c6c1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.191+0000 7f94a6ffd640 10 client.25636 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.191+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56541c6bec30 0x56541c6c1020 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7932b81cfce1d3f2 server_cookie=6c5c14be713d31f6 in_seq=0 out_seq=0 2024-03-19T01:19:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94a6ffd640 1 -- 172.21.15.116:0/3183665014 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56541c61b120 con 0x56541c6bec30 2024-03-19T01:19:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94a6ffd640 10 client.25636 handle_command_reply: tid=0 2024-03-19T01:19:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94d0c2d740 1 client.25636 shutdown 2024-03-19T01:19:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94d0c2d740 1 -- 172.21.15.116:0/3183665014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9af0 msgr2=0x56541c6b3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9af0 0x56541c6b3670 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x56541c6a9ef0 tx=0x7f94b8006e40 comp rx=0 tx=0).stop 2024-03-19T01:19:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94d0c2d740 1 -- 172.21.15.116:0/3183665014 shutdown_connections 2024-03-19T01:19:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6aec60 0x56541c6ba6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56541c6bec30 0x56541c6c1020 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9af0 0x56541c6b3670 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/3183665014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6a9100 0x56541c6b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94d0c2d740 1 -- 172.21.15.116:0/3183665014 >> 172.21.15.116:0/3183665014 conn(0x56541c6a4670 msgr2=0x56541c6a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94a6ffd640 0 client.25636 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94d0c2d740 1 -- 172.21.15.116:0/3183665014 shutdown_connections 2024-03-19T01:19:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94d0c2d740 1 -- 172.21.15.116:0/3183665014 wait complete. 2024-03-19T01:19:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.192+0000 7f94d0c2d740 20 client.25636 trim_cache size 0 max 16384 2024-03-19T01:19:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.211+0000 7f94d0c2d740 1 Processor -- start 2024-03-19T01:19:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.211+0000 7f94d0c2d740 1 -- start start 2024-03-19T01:19:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.211+0000 7f94d0c2d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9100 0x56541c6a3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.211+0000 7f94d0c2d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6a9af0 0x56541c6ba670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.211+0000 7f94d0c2d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6aec60 0x56541c6aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.211+0000 7f94d0c2d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94b400e530 con 0x56541c6a9af0 2024-03-19T01:19:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94d0c2d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f94b4010070 con 0x56541c6a9100 2024-03-19T01:19:09.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6a9af0 0x56541c6ba670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6a9af0 0x56541c6ba670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45538/0 (socket says 172.21.15.116:45538) 2024-03-19T01:19:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94c77fe640 1 -- 172.21.15.116:0/329240017 learned_addr learned my addr 172.21.15.116:0/329240017 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/329240017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9100 0x56541c6a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94d0c2d740 1 -- 172.21.15.116:0/329240017 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94b400ec00 con 0x56541c6aec60 2024-03-19T01:19:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/329240017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6aec60 0x56541c6aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94c6ffd640 1 -- 172.21.15.116:0/329240017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6aec60 msgr2=0x56541c6aa4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/329240017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6aec60 0x56541c6aa4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94c6ffd640 1 -- 172.21.15.116:0/329240017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6a9af0 msgr2=0x56541c6ba670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/329240017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6a9af0 0x56541c6ba670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94c6ffd640 1 -- 172.21.15.116:0/329240017 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56541c3ee620 con 0x56541c6a9100 2024-03-19T01:19:09.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94c77fe640 1 --2- 172.21.15.116:0/329240017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6a9af0 0x56541c6ba670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:19:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/329240017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9100 0x56541c6a3930 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x56541c6b1f60 tx=0x7f94bc00a2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b54ecf783cf56980 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.212+0000 7f94ad7fa640 1 -- 172.21.15.116:0/329240017 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc012110 con 0x56541c6a9100 2024-03-19T01:19:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.213+0000 7f94ad7fa640 1 -- 172.21.15.116:0/329240017 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94bc0122b0 con 0x56541c6a9100 2024-03-19T01:19:09.215 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.213+0000 7f94ad7fa640 1 -- 172.21.15.116:0/329240017 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc0051a0 con 0x56541c6a9100 2024-03-19T01:19:09.217 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.213+0000 7f94d0c2d740 1 -- 172.21.15.116:0/329240017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9100 msgr2=0x56541c6a3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:09.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.213+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/329240017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9100 0x56541c6a3930 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x56541c6b1f60 tx=0x7f94bc00a2d0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.213+0000 7f94d0c2d740 1 -- 172.21.15.116:0/329240017 shutdown_connections 2024-03-19T01:19:09.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.213+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/329240017 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6aec60 0x56541c6aa4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.213+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/329240017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6a9af0 0x56541c6ba670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.213+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/329240017 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9100 0x56541c6a3930 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.213+0000 7f94d0c2d740 1 -- 172.21.15.116:0/329240017 >> 172.21.15.116:0/329240017 conn(0x56541c6a4670 msgr2=0x56541c6a3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:09.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.213+0000 7f94d0c2d740 1 -- 172.21.15.116:0/329240017 shutdown_connections 2024-03-19T01:19:09.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.213+0000 7f94d0c2d740 1 -- 172.21.15.116:0/329240017 wait complete. 2024-03-19T01:19:09.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94d0c2d740 1 -- start start 2024-03-19T01:19:09.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94d0c2d740 1 Processor -- start 2024-03-19T01:19:09.218 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94d0c2d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9100 0x56541c6b80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94d0c2d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9af0 0x56541c6b8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94d0c2d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6aec60 0x56541c6b9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94d0c2d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94b400f9a0 con 0x56541c6aec60 2024-03-19T01:19:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94d0c2d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f94b400f2d0 con 0x56541c6a9af0 2024-03-19T01:19:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94d0c2d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f94b400c510 con 0x56541c6a9100 2024-03-19T01:19:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94d0c2d740 10 client.0 _ll_register_callbacks cb 0x7f94bc00a950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:19:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94c6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9100 0x56541c6b80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94c77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9af0 0x56541c6b8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94c77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9af0 0x56541c6b8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:52810/0 (socket says 172.21.15.116:52810) 2024-03-19T01:19:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94c77fe640 1 -- 172.21.15.116:0/1221338378 learned_addr learned my addr 172.21.15.116:0/1221338378 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.214+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/1221338378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6aec60 0x56541c6b9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.219 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.215+0000 7f94c77fe640 1 -- 172.21.15.116:0/1221338378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9100 msgr2=0x56541c6b80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.215+0000 7f94c77fe640 1 --2- 172.21.15.116:0/1221338378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9100 0x56541c6b80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.215+0000 7f94c77fe640 1 -- 172.21.15.116:0/1221338378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6aec60 msgr2=0x56541c6b9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.215+0000 7f94c77fe640 1 --2- 172.21.15.116:0/1221338378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6aec60 0x56541c6b9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.215+0000 7f94c77fe640 1 -- 172.21.15.116:0/1221338378 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56541c6b9c20 con 0x56541c6a9af0 2024-03-19T01:19:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.215+0000 7f94c77fe640 1 --2- 172.21.15.116:0/1221338378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9af0 0x56541c6b8620 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f94b8031d40 tx=0x7f94b8009180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d77228dfb21141c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.215+0000 7f94a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:19:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.215+0000 7f94a6ffd640 1 -- 172.21.15.116:0/1221338378 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8035030 con 0x56541c6a9af0 2024-03-19T01:19:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.215+0000 7f94a6ffd640 1 -- 172.21.15.116:0/1221338378 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f94b80138f0 con 0x56541c6a9af0 2024-03-19T01:19:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.215+0000 7f94d0c2d740 1 -- 172.21.15.116:0/1221338378 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94bc0122b0 con 0x56541c6a9af0 2024-03-19T01:19:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.216+0000 7f94a6ffd640 1 -- 172.21.15.116:0/1221338378 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f94b8017040 con 0x56541c6a9af0 2024-03-19T01:19:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.216+0000 7f94d0c2d740 10 client.25664 fetch_fsmap learned FSMap version 45 2024-03-19T01:19:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.216+0000 7f94d0c2d740 1 -- 172.21.15.116:0/1221338378 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56541c725250 con 0x56541c6a9af0 2024-03-19T01:19:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.216+0000 7f94a6ffd640 1 -- 172.21.15.116:0/1221338378 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94b800ee50 con 0x56541c6a9af0 2024-03-19T01:19:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.216+0000 7f94a6ffd640 1 -- 172.21.15.116:0/1221338378 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f94b801c8a0 con 0x56541c6a9af0 2024-03-19T01:19:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.216+0000 7f94d0c2d740 10 client.25664 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:19:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.216+0000 7f94d0c2d740 10 client.25664 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:19:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.216+0000 7f94d0c2d740 20 client.25664 populate_metadata read hostname 'smithi116' 2024-03-19T01:19:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.216+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/1221338378 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56541c725bd0 0x56541c727ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.216+0000 7f94d0c2d740 4 client.25664 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:19:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.216+0000 7f94d0c2d740 1 -- 172.21.15.116:0/1221338378 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f94bc0122b0 con 0x56541c725bd0 2024-03-19T01:19:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.216+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/1221338378 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56541c725bd0 0x56541c727ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.217+0000 7f94a6ffd640 10 client.25664 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.217+0000 7f94c6ffd640 1 --2- 172.21.15.116:0/1221338378 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56541c725bd0 0x56541c727ff0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d0f130abd1b98a5 server_cookie=7fd6d11fe11d9ea0 in_seq=0 out_seq=0 2024-03-19T01:19:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94a6ffd640 1 -- 172.21.15.116:0/1221338378 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f94bc0122b0 con 0x56541c725bd0 2024-03-19T01:19:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94a6ffd640 10 client.25664 handle_command_reply: tid=0 2024-03-19T01:19:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94d0c2d740 1 client.25664 shutdown 2024-03-19T01:19:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94d0c2d740 1 -- 172.21.15.116:0/1221338378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9af0 msgr2=0x56541c6b8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/1221338378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9af0 0x56541c6b8620 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f94b8031d40 tx=0x7f94b8009180 comp rx=0 tx=0).stop 2024-03-19T01:19:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94d0c2d740 1 -- 172.21.15.116:0/1221338378 shutdown_connections 2024-03-19T01:19:09.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/1221338378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56541c6aec60 0x56541c6b9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/1221338378 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56541c6a9af0 0x56541c6b8620 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/1221338378 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56541c725bd0 0x56541c727ff0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94d0c2d740 1 --2- 172.21.15.116:0/1221338378 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56541c6a9100 0x56541c6b80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94d0c2d740 1 -- 172.21.15.116:0/1221338378 >> 172.21.15.116:0/1221338378 conn(0x56541c6a4670 msgr2=0x56541c6aa4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94a6ffd640 0 client.25664 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94d0c2d740 1 -- 172.21.15.116:0/1221338378 shutdown_connections 2024-03-19T01:19:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94d0c2d740 1 -- 172.21.15.116:0/1221338378 wait complete. 2024-03-19T01:19:09.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.225+0000 7f94d0c2d740 20 client.25664 trim_cache size 0 max 16384 2024-03-19T01:19:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:19:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:19:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:19:09.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:19:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.226+0000 7f94ce044640 1 -- 172.21.15.116:0/2926837774 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f94b00a29a0 msgr2=0x7f94b00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.226+0000 7f94ce044640 1 --2- 172.21.15.116:0/2926837774 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f94b00a29a0 0x7f94b00a4e60 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f94c0009620 tx=0x7f94c003e000 comp rx=0 tx=0).stop 2024-03-19T01:19:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.226+0000 7f94ce044640 1 -- 172.21.15.116:0/2926837774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94c81011f0 msgr2=0x7f94c81a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.226+0000 7f94ce044640 1 --2- 172.21.15.116:0/2926837774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94c81011f0 0x7f94c81a5630 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f94c0037040 tx=0x7f94c0032f90 comp rx=0 tx=0).stop 2024-03-19T01:19:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.226+0000 7f94ce044640 1 -- 172.21.15.116:0/2926837774 shutdown_connections 2024-03-19T01:19:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.226+0000 7f94ce044640 1 --2- 172.21.15.116:0/2926837774 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f94c81011f0 0x7f94c81a5630 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.226+0000 7f94ce044640 1 --2- 172.21.15.116:0/2926837774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94c8100830 0x7f94c819e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.226+0000 7f94ce044640 1 --2- 172.21.15.116:0/2926837774 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f94b00a29a0 0x7f94b00a4e60 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.226+0000 7f94ce044640 1 --2- 172.21.15.116:0/2926837774 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f94c80ffad0 0x7f94c819e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.226+0000 7f94ce044640 1 -- 172.21.15.116:0/2926837774 >> 172.21.15.116:0/2926837774 conn(0x7f94c80fb440 msgr2=0x7f94c8101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.226+0000 7f94ce044640 1 -- 172.21.15.116:0/2926837774 shutdown_connections 2024-03-19T01:19:09.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:09.226+0000 7f94ce044640 1 -- 172.21.15.116:0/2926837774 wait complete. 2024-03-19T01:19:09.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:09 smithi162 ceph-mon[67178]: pgmap v548: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 39 MiB/s wr, 1.59k op/s 2024-03-19T01:19:09.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:09 smithi116 ceph-mon[62701]: pgmap v548: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 39 MiB/s wr, 1.59k op/s 2024-03-19T01:19:09.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:09 smithi156 ceph-mon[66799]: pgmap v548: 97 pgs: 97 active+clean; 1.6 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 39 MiB/s wr, 1.59k op/s 2024-03-19T01:19:11.544 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 385.0 s, 2214.6 tps, lat 44.729 ms stddev 53.079 2024-03-19T01:19:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:11 smithi162 ceph-mon[67178]: pgmap v549: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 37 MiB/s wr, 1.54k op/s 2024-03-19T01:19:11.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:11 smithi116 ceph-mon[62701]: pgmap v549: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 37 MiB/s wr, 1.54k op/s 2024-03-19T01:19:11.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:11 smithi156 ceph-mon[66799]: pgmap v549: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 37 MiB/s wr, 1.54k op/s 2024-03-19T01:19:13.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:13 smithi162 ceph-mon[67178]: pgmap v550: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 35 MiB/s wr, 1.49k op/s 2024-03-19T01:19:13.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:13 smithi116 ceph-mon[62701]: pgmap v550: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 35 MiB/s wr, 1.49k op/s 2024-03-19T01:19:13.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:13 smithi156 ceph-mon[66799]: pgmap v550: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 35 MiB/s wr, 1.49k op/s 2024-03-19T01:19:15.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:15 smithi162 ceph-mon[67178]: pgmap v551: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 34 MiB/s wr, 1.51k op/s 2024-03-19T01:19:15.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:15 smithi116 ceph-mon[62701]: pgmap v551: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 34 MiB/s wr, 1.51k op/s 2024-03-19T01:19:15.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:15 smithi156 ceph-mon[66799]: pgmap v551: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 34 MiB/s wr, 1.51k op/s 2024-03-19T01:19:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 390.0 s, 2230.6 tps, lat 44.725 ms stddev 51.015 2024-03-19T01:19:17.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:17 smithi162 ceph-mon[67178]: pgmap v552: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 909 op/s 2024-03-19T01:19:17.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:17 smithi116 ceph-mon[62701]: pgmap v552: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 909 op/s 2024-03-19T01:19:17.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:17 smithi156 ceph-mon[66799]: pgmap v552: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 909 op/s 2024-03-19T01:19:19.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:19 smithi162 ceph-mon[67178]: pgmap v553: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 708 op/s 2024-03-19T01:19:19.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:19 smithi116 ceph-mon[62701]: pgmap v553: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 708 op/s 2024-03-19T01:19:19.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:19 smithi156 ceph-mon[66799]: pgmap v553: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 10 MiB/s wr, 708 op/s 2024-03-19T01:19:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 395.0 s, 2326.4 tps, lat 43.479 ms stddev 51.314 2024-03-19T01:19:21.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:21 smithi162 ceph-mon[67178]: pgmap v554: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 674 op/s 2024-03-19T01:19:21.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:21 smithi116 ceph-mon[62701]: pgmap v554: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 674 op/s 2024-03-19T01:19:21.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:21 smithi156 ceph-mon[66799]: pgmap v554: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 MiB/s wr, 674 op/s 2024-03-19T01:19:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.360+0000 7f7050080640 1 -- 172.21.15.116:0/4157884831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7048101820 msgr2=0x7f7048101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.360+0000 7f7050080640 1 --2- 172.21.15.116:0/4157884831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7048101820 0x7f7048101c40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f7044009900 tx=0x7f704402bc30 comp rx=0 tx=0).stop 2024-03-19T01:19:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.360+0000 7f7050080640 1 -- 172.21.15.116:0/4157884831 shutdown_connections 2024-03-19T01:19:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.360+0000 7f7050080640 1 --2- 172.21.15.116:0/4157884831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70481033d0 0x7f7048109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.360+0000 7f7050080640 1 --2- 172.21.15.116:0/4157884831 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7048102a10 0x7f7048102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.360+0000 7f7050080640 1 --2- 172.21.15.116:0/4157884831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7048101820 0x7f7048101c40 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.360+0000 7f7050080640 1 -- 172.21.15.116:0/4157884831 >> 172.21.15.116:0/4157884831 conn(0x7f70480fcd90 msgr2=0x7f70480ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.361+0000 7f7050080640 1 -- 172.21.15.116:0/4157884831 shutdown_connections 2024-03-19T01:19:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.361+0000 7f7050080640 1 -- 172.21.15.116:0/4157884831 wait complete. 2024-03-19T01:19:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.361+0000 7f7050080640 1 Processor -- start 2024-03-19T01:19:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.361+0000 7f7050080640 1 -- start start 2024-03-19T01:19:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.361+0000 7f7050080640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7048101820 0x7f7048199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f7050080640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7048102a10 0x7f7048199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f7050080640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70481033d0 0x7f70481a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f7050080640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f70480744a0 con 0x7f7048101820 2024-03-19T01:19:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f7050080640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7048074320 con 0x7f7048102a10 2024-03-19T01:19:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f7050080640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7048074620 con 0x7f70481033d0 2024-03-19T01:19:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f704de1c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7048101820 0x7f7048199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:22.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f704de1c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7048101820 0x7f7048199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33096/0 (socket says 172.21.15.116:33096) 2024-03-19T01:19:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f704de1c640 1 -- 172.21.15.116:0/714583145 learned_addr learned my addr 172.21.15.116:0/714583145 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f704d61b640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7048102a10 0x7f7048199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:22.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f704e61d640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70481033d0 0x7f70481a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f704e61d640 1 -- 172.21.15.116:0/714583145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7048102a10 msgr2=0x7f7048199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f704e61d640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7048102a10 0x7f7048199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f704e61d640 1 -- 172.21.15.116:0/714583145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7048101820 msgr2=0x7f7048199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f704e61d640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7048101820 0x7f7048199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f704e61d640 1 -- 172.21.15.116:0/714583145 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70481a16b0 con 0x7f70481033d0 2024-03-19T01:19:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f704d61b640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7048102a10 0x7f7048199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:19:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f704de1c640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7048101820 0x7f7048199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:19:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f704e61d640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70481033d0 0x7f70481a1040 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7040009fd0 tx=0x7f704000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f703effd640 1 -- 172.21.15.116:0/714583145 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704000ce20 con 0x7f70481033d0 2024-03-19T01:19:22.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.362+0000 7f7050080640 1 -- 172.21.15.116:0/714583145 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70481a19a0 con 0x7f70481033d0 2024-03-19T01:19:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.363+0000 7f703effd640 1 -- 172.21.15.116:0/714583145 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7040005020 con 0x7f70481033d0 2024-03-19T01:19:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.363+0000 7f7050080640 1 -- 172.21.15.116:0/714583145 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70481a1ef0 con 0x7f70481033d0 2024-03-19T01:19:22.366 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.363+0000 7f703effd640 1 -- 172.21.15.116:0/714583145 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040010720 con 0x7f70481033d0 2024-03-19T01:19:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.366+0000 7f703effd640 1 -- 172.21.15.116:0/714583145 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7040005540 con 0x7f70481033d0 2024-03-19T01:19:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.366+0000 7f7052c5f740 1 Processor -- start 2024-03-19T01:19:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.366+0000 7f7052c5f740 1 -- start start 2024-03-19T01:19:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.366+0000 7f7052c5f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949100 0x561cff949520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.366+0000 7f7052c5f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff949af0 0x561cff94e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.366+0000 7f7052c5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff94ec60 0x561cff9510f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.366+0000 7f7052c5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561cff8adbe0 con 0x561cff94ec60 2024-03-19T01:19:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.366+0000 7f7052c5f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561cff856960 con 0x561cff949af0 2024-03-19T01:19:22.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.366+0000 7f7052c5f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561cff8a9c80 con 0x561cff949100 2024-03-19T01:19:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.367+0000 7f704d61b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949100 0x561cff949520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.367+0000 7f704de1c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff949af0 0x561cff94e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.367+0000 7f704d61b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949100 0x561cff949520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:39224/0 (socket says 172.21.15.116:39224) 2024-03-19T01:19:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.367+0000 7f704de1c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff949af0 0x561cff94e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:35432/0 (socket says 172.21.15.116:35432) 2024-03-19T01:19:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.367+0000 7f704d61b640 1 -- 172.21.15.116:0/3734054588 learned_addr learned my addr 172.21.15.116:0/3734054588 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.367+0000 7f703effd640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70340a28f0 0x7f70340a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.367+0000 7f704d61b640 1 --2- 172.21.15.116:0/3734054588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff94ec60 0x561cff9510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.367+0000 7f703effd640 1 -- 172.21.15.116:0/714583145 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f70400cf130 con 0x7f70481033d0 2024-03-19T01:19:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.367+0000 7f704d61b640 1 -- 172.21.15.116:0/3734054588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff949af0 msgr2=0x561cff94e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.367+0000 7f704d61b640 1 --2- 172.21.15.116:0/3734054588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff949af0 0x561cff94e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.367+0000 7f704d61b640 1 -- 172.21.15.116:0/3734054588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff94ec60 msgr2=0x561cff9510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.367+0000 7f704d61b640 1 --2- 172.21.15.116:0/3734054588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff94ec60 0x561cff9510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.367+0000 7f704d61b640 1 -- 172.21.15.116:0/3734054588 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cff84dd20 con 0x561cff949100 2024-03-19T01:19:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.368+0000 7f704d61b640 1 --2- 172.21.15.116:0/3734054588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949100 0x561cff949520 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7038005c50 tx=0x7f703800bb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f07aee7d6b96f196 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.368+0000 7f7033fff640 1 -- 172.21.15.116:0/3734054588 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7038027070 con 0x561cff949100 2024-03-19T01:19:22.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.368+0000 7f7033fff640 1 -- 172.21.15.116:0/3734054588 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7038022be0 con 0x561cff949100 2024-03-19T01:19:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.369+0000 7f7052c5f740 1 -- 172.21.15.116:0/3734054588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949100 msgr2=0x561cff949520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.369+0000 7f7052c5f740 1 --2- 172.21.15.116:0/3734054588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949100 0x561cff949520 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7038005c50 tx=0x7f703800bb90 comp rx=0 tx=0).stop 2024-03-19T01:19:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.369+0000 7f7052c5f740 1 -- 172.21.15.116:0/3734054588 shutdown_connections 2024-03-19T01:19:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.369+0000 7f7052c5f740 1 --2- 172.21.15.116:0/3734054588 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff94ec60 0x561cff9510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.369+0000 7f7052c5f740 1 --2- 172.21.15.116:0/3734054588 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff949af0 0x561cff94e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.369+0000 7f7052c5f740 1 --2- 172.21.15.116:0/3734054588 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949100 0x561cff949520 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.369+0000 7f7052c5f740 1 -- 172.21.15.116:0/3734054588 >> 172.21.15.116:0/3734054588 conn(0x561cff944670 msgr2=0x561cff944a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.369+0000 7f7052c5f740 1 -- 172.21.15.116:0/3734054588 shutdown_connections 2024-03-19T01:19:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.369+0000 7f7052c5f740 1 -- 172.21.15.116:0/3734054588 wait complete. 2024-03-19T01:19:22.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.369+0000 7f704e61d640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70340a28f0 0x7f70340a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:22.373 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.370+0000 7f7052c5f740 1 -- start start 2024-03-19T01:19:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.370+0000 7f704e61d640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70340a28f0 0x7f70340a4db0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7040005ba0 tx=0x7f7040086040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.370+0000 7f7052c5f740 1 Processor -- start 2024-03-19T01:19:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f7052c5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff949100 0x561cff964260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f7052c5f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff949af0 0x561cff9647a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f7052c5f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff94ec60 0x561cff96b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f7052c5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561cff951cb0 con 0x561cff949100 2024-03-19T01:19:22.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f7052c5f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561cff869db0 con 0x561cff949af0 2024-03-19T01:19:22.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f7052c5f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561cff951e30 con 0x561cff94ec60 2024-03-19T01:19:22.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f704d61b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff94ec60 0x561cff96b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:22.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f704d61b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff94ec60 0x561cff96b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:39238/0 (socket says 172.21.15.116:39238) 2024-03-19T01:19:22.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f704d61b640 1 -- 172.21.15.116:0/3801786518 learned_addr learned my addr 172.21.15.116:0/3801786518 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f7052c5f740 10 client.0 _ll_register_callbacks cb 0x561cff86ee70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:19:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f704d61b640 1 -- 172.21.15.116:0/3801786518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff949af0 msgr2=0x561cff9647a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f704d61b640 1 --2- 172.21.15.116:0/3801786518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff949af0 0x561cff9647a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f704d61b640 1 -- 172.21.15.116:0/3801786518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff949100 msgr2=0x561cff964260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f704d61b640 1 --2- 172.21.15.116:0/3801786518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff949100 0x561cff964260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.371+0000 7f704d61b640 1 -- 172.21.15.116:0/3801786518 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cff868c60 con 0x561cff94ec60 2024-03-19T01:19:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.372+0000 7f7052c5f740 1 -- 172.21.15.116:0/3801786518 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7038022be0 con 0x561cff94ec60 2024-03-19T01:19:22.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.372+0000 7f704d61b640 1 --2- 172.21.15.116:0/3801786518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff94ec60 0x561cff96b770 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f703800bd50 tx=0x7f7038010960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eedfa94a004b948d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.372+0000 7f70317fa640 10 client.25654 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.372+0000 7f70317fa640 1 -- 172.21.15.116:0/3801786518 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7038027020 con 0x561cff94ec60 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.372+0000 7f70317fa640 1 -- 172.21.15.116:0/3801786518 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7038005310 con 0x561cff94ec60 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.372+0000 7f70317fa640 1 -- 172.21.15.116:0/3801786518 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f7038005510 con 0x561cff94ec60 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.372+0000 7f7052c5f740 10 client.25654 fetch_fsmap learned FSMap version 45 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.372+0000 7f7052c5f740 1 -- 172.21.15.116:0/3801786518 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({fsmap=45}) -- 0x561cff5c2210 con 0x561cff94ec60 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.373+0000 7f70317fa640 1 -- 172.21.15.116:0/3801786518 <== mon.1 v2:172.21.15.162:3300/0 4 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f7038020020 con 0x561cff94ec60 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.373+0000 7f7052c5f740 10 client.25654 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.373+0000 7f7052c5f740 10 client.25654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.373+0000 7f7052c5f740 20 client.25654 populate_metadata read hostname 'smithi116' 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.373+0000 7f7052c5f740 1 --2- 172.21.15.116:0/3801786518 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561cff9700c0 0x561cff972550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.373+0000 7f7052c5f740 4 client.25654 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.373+0000 7f7052c5f740 1 -- 172.21.15.116:0/3801786518 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cff8bb120 con 0x561cff9700c0 2024-03-19T01:19:22.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.373+0000 7f704de1c640 1 --2- 172.21.15.116:0/3801786518 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561cff9700c0 0x561cff972550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.374+0000 7f704de1c640 1 --2- 172.21.15.116:0/3801786518 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561cff9700c0 0x561cff972550 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62f9d9e8f6bce61f server_cookie=41dc82d760b3214d in_seq=0 out_seq=0 2024-03-19T01:19:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.374+0000 7f70317fa640 10 client.25654 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.375+0000 7f70317fa640 1 -- 172.21.15.116:0/3801786518 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561cff8bb120 con 0x561cff9700c0 2024-03-19T01:19:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.375+0000 7f70317fa640 10 client.25654 handle_command_reply: tid=0 2024-03-19T01:19:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.375+0000 7f7052c5f740 1 client.25654 shutdown 2024-03-19T01:19:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.375+0000 7f7052c5f740 1 -- 172.21.15.116:0/3801786518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff94ec60 msgr2=0x561cff96b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.375+0000 7f7052c5f740 1 --2- 172.21.15.116:0/3801786518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff94ec60 0x561cff96b770 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f703800bd50 tx=0x7f7038010960 comp rx=0 tx=0).stop 2024-03-19T01:19:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.375+0000 7f7052c5f740 1 -- 172.21.15.116:0/3801786518 shutdown_connections 2024-03-19T01:19:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.375+0000 7f7052c5f740 1 --2- 172.21.15.116:0/3801786518 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561cff9700c0 0x561cff972550 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.375+0000 7f7052c5f740 1 --2- 172.21.15.116:0/3801786518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff94ec60 0x561cff96b770 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.375+0000 7f7052c5f740 1 --2- 172.21.15.116:0/3801786518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff949af0 0x561cff9647a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.375+0000 7f70317fa640 0 client.25654 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.375+0000 7f7052c5f740 1 --2- 172.21.15.116:0/3801786518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff949100 0x561cff964260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.375+0000 7f7052c5f740 1 -- 172.21.15.116:0/3801786518 >> 172.21.15.116:0/3801786518 conn(0x561cff944670 msgr2=0x561cff9458c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.376+0000 7f7052c5f740 1 -- 172.21.15.116:0/3801786518 shutdown_connections 2024-03-19T01:19:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.376+0000 7f7052c5f740 1 -- 172.21.15.116:0/3801786518 wait complete. 2024-03-19T01:19:22.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.376+0000 7f7052c5f740 20 client.25654 trim_cache size 0 max 16384 2024-03-19T01:19:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.406+0000 7f7052c5f740 1 Processor -- start 2024-03-19T01:19:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.406+0000 7f7052c5f740 1 -- start start 2024-03-19T01:19:22.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.406+0000 7f7052c5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff949100 0x561cff951050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.406+0000 7f7052c5f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 0x561cff96b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.406+0000 7f7052c5f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff94ec60 0x561cff943220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.406+0000 7f7052c5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f70481a6810 con 0x561cff949100 2024-03-19T01:19:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.406+0000 7f7052c5f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f70481a8350 con 0x561cff94ec60 2024-03-19T01:19:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.407+0000 7f7052c5f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f70481a6ee0 con 0x561cff949af0 2024-03-19T01:19:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.407+0000 7f704de1c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 0x561cff96b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.407+0000 7f704de1c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 0x561cff96b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:39252/0 (socket says 172.21.15.116:39252) 2024-03-19T01:19:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.407+0000 7f704de1c640 1 -- 172.21.15.116:0/4083256234 learned_addr learned my addr 172.21.15.116:0/4083256234 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.407+0000 7f704de1c640 1 -- 172.21.15.116:0/4083256234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff94ec60 msgr2=0x561cff943220 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:19:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.407+0000 7f704de1c640 1 --2- 172.21.15.116:0/4083256234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff94ec60 0x561cff943220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.407+0000 7f704de1c640 1 -- 172.21.15.116:0/4083256234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff949100 msgr2=0x561cff951050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.407+0000 7f704de1c640 1 --2- 172.21.15.116:0/4083256234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff949100 0x561cff951050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.407+0000 7f704de1c640 1 -- 172.21.15.116:0/4083256234 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cff68e620 con 0x561cff949af0 2024-03-19T01:19:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.407+0000 7f704de1c640 1 --2- 172.21.15.116:0/4083256234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 0x561cff96b730 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7044004a90 tx=0x7f7044000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c69f848d0b17bad server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.408+0000 7f70337fe640 1 -- 172.21.15.116:0/4083256234 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704404aab0 con 0x561cff949af0 2024-03-19T01:19:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.408+0000 7f70337fe640 1 -- 172.21.15.116:0/4083256234 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f704404ac50 con 0x561cff949af0 2024-03-19T01:19:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.408+0000 7f70337fe640 1 -- 172.21.15.116:0/4083256234 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704404a530 con 0x561cff949af0 2024-03-19T01:19:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.409+0000 7f7052c5f740 1 -- 172.21.15.116:0/4083256234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 msgr2=0x561cff96b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.409+0000 7f7052c5f740 1 --2- 172.21.15.116:0/4083256234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 0x561cff96b730 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7044004a90 tx=0x7f7044000bc0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.409+0000 7f7052c5f740 1 -- 172.21.15.116:0/4083256234 shutdown_connections 2024-03-19T01:19:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.409+0000 7f7052c5f740 1 --2- 172.21.15.116:0/4083256234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff94ec60 0x561cff943220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.409+0000 7f7052c5f740 1 --2- 172.21.15.116:0/4083256234 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 0x561cff96b730 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.409+0000 7f7052c5f740 1 --2- 172.21.15.116:0/4083256234 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff949100 0x561cff951050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.409+0000 7f7052c5f740 1 -- 172.21.15.116:0/4083256234 >> 172.21.15.116:0/4083256234 conn(0x561cff944670 msgr2=0x561cff944a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:22.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.410+0000 7f7052c5f740 1 -- 172.21.15.116:0/4083256234 shutdown_connections 2024-03-19T01:19:22.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.410+0000 7f7052c5f740 1 -- 172.21.15.116:0/4083256234 wait complete. 2024-03-19T01:19:22.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.410+0000 7f7052c5f740 1 -- start start 2024-03-19T01:19:22.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.411+0000 7f7052c5f740 1 Processor -- start 2024-03-19T01:19:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.411+0000 7f7052c5f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff949100 0x561cff968a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.411+0000 7f7052c5f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 0x561cff968f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.411+0000 7f7052c5f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff94ec60 0x561cff969fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.411+0000 7f7052c5f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f70481a7c80 con 0x561cff949100 2024-03-19T01:19:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.411+0000 7f7052c5f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f70481a75b0 con 0x561cff94ec60 2024-03-19T01:19:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.411+0000 7f7052c5f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f70481a4850 con 0x561cff949af0 2024-03-19T01:19:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.411+0000 7f7052c5f740 10 client.0 _ll_register_callbacks cb 0x7f7038012f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:19:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.411+0000 7f704de1c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 0x561cff968f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:22.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.411+0000 7f704de1c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 0x561cff968f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:39264/0 (socket says 172.21.15.116:39264) 2024-03-19T01:19:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.411+0000 7f704de1c640 1 -- 172.21.15.116:0/2735951518 learned_addr learned my addr 172.21.15.116:0/2735951518 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.412+0000 7f704de1c640 1 -- 172.21.15.116:0/2735951518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff94ec60 msgr2=0x561cff969fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.412+0000 7f704de1c640 1 --2- 172.21.15.116:0/2735951518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff94ec60 0x561cff969fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.412+0000 7f704de1c640 1 -- 172.21.15.116:0/2735951518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff949100 msgr2=0x561cff968a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.412+0000 7f704de1c640 1 --2- 172.21.15.116:0/2735951518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff949100 0x561cff968a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.412+0000 7f704de1c640 1 -- 172.21.15.116:0/2735951518 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cff96a590 con 0x561cff949af0 2024-03-19T01:19:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.412+0000 7f704de1c640 1 --2- 172.21.15.116:0/2735951518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 0x561cff968f90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f70440096c0 tx=0x7f7044005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fec4d404dd67afe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.413+0000 7f70317fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:19:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.413+0000 7f70317fa640 1 -- 172.21.15.116:0/2735951518 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704402c8f0 con 0x561cff949af0 2024-03-19T01:19:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.413+0000 7f70317fa640 1 -- 172.21.15.116:0/2735951518 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f704402ca90 con 0x561cff949af0 2024-03-19T01:19:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.413+0000 7f7052c5f740 1 -- 172.21.15.116:0/2735951518 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f704404ac50 con 0x561cff949af0 2024-03-19T01:19:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.413+0000 7f70317fa640 1 -- 172.21.15.116:0/2735951518 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f704404a2d0 con 0x561cff949af0 2024-03-19T01:19:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.413+0000 7f7052c5f740 10 client.25660 fetch_fsmap learned FSMap version 45 2024-03-19T01:19:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.413+0000 7f7052c5f740 1 -- 172.21.15.116:0/2735951518 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x561cff96b8f0 con 0x561cff949af0 2024-03-19T01:19:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.414+0000 7f70317fa640 1 -- 172.21.15.116:0/2735951518 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f704404ac50 con 0x561cff949af0 2024-03-19T01:19:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.414+0000 7f70317fa640 1 -- 172.21.15.116:0/2735951518 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f70440035f0 con 0x561cff949af0 2024-03-19T01:19:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.414+0000 7f7052c5f740 10 client.25660 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:19:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.414+0000 7f7052c5f740 10 client.25660 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:19:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.414+0000 7f7052c5f740 20 client.25660 populate_metadata read hostname 'smithi116' 2024-03-19T01:19:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.414+0000 7f7052c5f740 1 --2- 172.21.15.116:0/2735951518 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561cff9d69e0 0x561cff9d8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.414+0000 7f7052c5f740 4 client.25660 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:19:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.414+0000 7f7052c5f740 1 -- 172.21.15.116:0/2735951518 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561cff8b5470 con 0x561cff9d69e0 2024-03-19T01:19:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.414+0000 7f704d61b640 1 --2- 172.21.15.116:0/2735951518 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561cff9d69e0 0x561cff9d8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.415+0000 7f704d61b640 1 --2- 172.21.15.116:0/2735951518 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561cff9d69e0 0x561cff9d8e00 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdaaf3af6e20a2af server_cookie=d6b79a40850504e1 in_seq=0 out_seq=0 2024-03-19T01:19:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.415+0000 7f70317fa640 10 client.25660 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:22.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f70317fa640 1 -- 172.21.15.116:0/2735951518 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561cff8b5470 con 0x561cff9d69e0 2024-03-19T01:19:22.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f70317fa640 10 client.25660 handle_command_reply: tid=0 2024-03-19T01:19:22.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f7052c5f740 1 client.25660 shutdown 2024-03-19T01:19:22.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f7052c5f740 1 -- 172.21.15.116:0/2735951518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 msgr2=0x561cff968f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:22.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f7052c5f740 1 --2- 172.21.15.116:0/2735951518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 0x561cff968f90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f70440096c0 tx=0x7f7044005bd0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f7052c5f740 1 -- 172.21.15.116:0/2735951518 shutdown_connections 2024-03-19T01:19:22.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f7052c5f740 1 --2- 172.21.15.116:0/2735951518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561cff94ec60 0x561cff969fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f7052c5f740 1 --2- 172.21.15.116:0/2735951518 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561cff9d69e0 0x561cff9d8e00 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f7052c5f740 1 --2- 172.21.15.116:0/2735951518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561cff949af0 0x561cff968f90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f7052c5f740 1 --2- 172.21.15.116:0/2735951518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561cff949100 0x561cff968a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f7052c5f740 1 -- 172.21.15.116:0/2735951518 >> 172.21.15.116:0/2735951518 conn(0x561cff944670 msgr2=0x561cff943220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f70317fa640 0 client.25660 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f7052c5f740 1 -- 172.21.15.116:0/2735951518 shutdown_connections 2024-03-19T01:19:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.423+0000 7f7052c5f740 1 -- 172.21.15.116:0/2735951518 wait complete. 2024-03-19T01:19:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.424+0000 7f7052c5f740 20 client.25660 trim_cache size 0 max 16384 2024-03-19T01:19:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:19:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:19:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:19:22.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:19:22.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.424+0000 7f7050080640 1 -- 172.21.15.116:0/714583145 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70340a28f0 msgr2=0x7f70340a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:22.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.424+0000 7f7050080640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70340a28f0 0x7f70340a4db0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7040005ba0 tx=0x7f7040086040 comp rx=0 tx=0).stop 2024-03-19T01:19:22.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.425+0000 7f7050080640 1 -- 172.21.15.116:0/714583145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70481033d0 msgr2=0x7f70481a1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:22.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.425+0000 7f7050080640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70481033d0 0x7f70481a1040 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7040009fd0 tx=0x7f704000ef10 comp rx=0 tx=0).stop 2024-03-19T01:19:22.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.425+0000 7f7050080640 1 -- 172.21.15.116:0/714583145 shutdown_connections 2024-03-19T01:19:22.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.425+0000 7f7050080640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f70481033d0 0x7f70481a1040 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.425+0000 7f7050080640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f70340a28f0 0x7f70340a4db0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.425+0000 7f7050080640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7048102a10 0x7f7048199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.425+0000 7f7050080640 1 --2- 172.21.15.116:0/714583145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7048101820 0x7f7048199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:22.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.425+0000 7f7050080640 1 -- 172.21.15.116:0/714583145 >> 172.21.15.116:0/714583145 conn(0x7f70480fcd90 msgr2=0x7f70480fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:22.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.426+0000 7f7050080640 1 -- 172.21.15.116:0/714583145 shutdown_connections 2024-03-19T01:19:22.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:22.426+0000 7f7050080640 1 -- 172.21.15.116:0/714583145 wait complete. 2024-03-19T01:19:23.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:23 smithi162 ceph-mon[67178]: pgmap v555: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 8.4 MiB/s wr, 609 op/s 2024-03-19T01:19:23.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:23 smithi116 ceph-mon[62701]: pgmap v555: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 8.4 MiB/s wr, 609 op/s 2024-03-19T01:19:23.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:23 smithi156 ceph-mon[66799]: pgmap v555: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 8.4 MiB/s wr, 609 op/s 2024-03-19T01:19:25.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:25 smithi162 ceph-mon[67178]: pgmap v556: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 774 op/s 2024-03-19T01:19:25.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:25 smithi116 ceph-mon[62701]: pgmap v556: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 774 op/s 2024-03-19T01:19:25.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:25 smithi156 ceph-mon[66799]: pgmap v556: 97 pgs: 97 active+clean; 1.6 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 774 op/s 2024-03-19T01:19:26.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 400.0 s, 2388.6 tps, lat 41.403 ms stddev 50.758 2024-03-19T01:19:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:27 smithi162 ceph-mon[67178]: pgmap v557: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.8 MiB/s wr, 752 op/s 2024-03-19T01:19:27.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:27 smithi116 ceph-mon[62701]: pgmap v557: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.8 MiB/s wr, 752 op/s 2024-03-19T01:19:27.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:27 smithi156 ceph-mon[66799]: pgmap v557: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.8 MiB/s wr, 752 op/s 2024-03-19T01:19:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:29 smithi162 ceph-mon[67178]: pgmap v558: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.9 MiB/s wr, 803 op/s 2024-03-19T01:19:29.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:29 smithi116 ceph-mon[62701]: pgmap v558: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.9 MiB/s wr, 803 op/s 2024-03-19T01:19:29.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:29 smithi156 ceph-mon[66799]: pgmap v558: 97 pgs: 97 active+clean; 1.6 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 9.9 MiB/s wr, 803 op/s 2024-03-19T01:19:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 405.0 s, 2336.4 tps, lat 43.271 ms stddev 49.947 2024-03-19T01:19:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:31 smithi162 ceph-mon[67178]: pgmap v559: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 9.4 MiB/s wr, 780 op/s 2024-03-19T01:19:31.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:31 smithi116 ceph-mon[62701]: pgmap v559: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 9.4 MiB/s wr, 780 op/s 2024-03-19T01:19:31.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:31 smithi156 ceph-mon[66799]: pgmap v559: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 9.4 MiB/s wr, 780 op/s 2024-03-19T01:19:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:33 smithi162 ceph-mon[67178]: pgmap v560: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.8 MiB/s wr, 722 op/s 2024-03-19T01:19:33.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:33 smithi116 ceph-mon[62701]: pgmap v560: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.8 MiB/s wr, 722 op/s 2024-03-19T01:19:33.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:33 smithi156 ceph-mon[66799]: pgmap v560: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 8.8 MiB/s wr, 722 op/s 2024-03-19T01:19:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.563+0000 7f49b771b640 1 -- 172.21.15.116:0/252047445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49b01074c0 msgr2=0x7f49b0109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.563+0000 7f49b771b640 1 --2- 172.21.15.116:0/252047445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49b01074c0 0x7f49b0109980 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009ca0 tx=0x7f49a802c0f0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.564+0000 7f49b771b640 1 -- 172.21.15.116:0/252047445 shutdown_connections 2024-03-19T01:19:35.565 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.564+0000 7f49b771b640 1 --2- 172.21.15.116:0/252047445 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49b01074c0 0x7f49b0109980 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.564+0000 7f49b771b640 1 --2- 172.21.15.116:0/252047445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b00fff30 0x7f49b0106f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.564+0000 7f49b771b640 1 --2- 172.21.15.116:0/252047445 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49b00ff5d0 0x7f49b00ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.564+0000 7f49b771b640 1 -- 172.21.15.116:0/252047445 >> 172.21.15.116:0/252047445 conn(0x7f49b00fb000 msgr2=0x7f49b00fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.564+0000 7f49b771b640 1 -- 172.21.15.116:0/252047445 shutdown_connections 2024-03-19T01:19:35.566 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.565+0000 7f49b771b640 1 -- 172.21.15.116:0/252047445 wait complete. 2024-03-19T01:19:35.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.565+0000 7f49b771b640 1 Processor -- start 2024-03-19T01:19:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.565+0000 7f49b771b640 1 -- start start 2024-03-19T01:19:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.565+0000 7f49b771b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b00ff5d0 0x7f49b0199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.565+0000 7f49b771b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49b00fff30 0x7f49b019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.565+0000 7f49b771b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49b01074c0 0x7f49b01a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.565+0000 7f49b771b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f49b010d4d0 con 0x7f49b01074c0 2024-03-19T01:19:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.565+0000 7f49b771b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49b010d350 con 0x7f49b00ff5d0 2024-03-19T01:19:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.565+0000 7f49b771b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f49b010d650 con 0x7f49b00fff30 2024-03-19T01:19:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.566+0000 7f49b4cb6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49b00fff30 0x7f49b019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.566+0000 7f49b54b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b00ff5d0 0x7f49b0199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.566+0000 7f49b4cb6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49b00fff30 0x7f49b019a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41286/0 (socket says 172.21.15.116:41286) 2024-03-19T01:19:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.566+0000 7f49b4cb6640 1 -- 172.21.15.116:0/2286620682 learned_addr learned my addr 172.21.15.116:0/2286620682 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:35.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.566+0000 7f49b5cb8640 1 --2- 172.21.15.116:0/2286620682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49b01074c0 0x7f49b01a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.566+0000 7f49b4cb6640 1 -- 172.21.15.116:0/2286620682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b00ff5d0 msgr2=0x7f49b0199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:35.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.566+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/2286620682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b00ff5d0 0x7f49b0199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.566+0000 7f49b4cb6640 1 -- 172.21.15.116:0/2286620682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49b01074c0 msgr2=0x7f49b01a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:35.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.566+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/2286620682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49b01074c0 0x7f49b01a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.566+0000 7f49b4cb6640 1 -- 172.21.15.116:0/2286620682 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b01a18b0 con 0x7f49b00fff30 2024-03-19T01:19:35.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.567+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/2286620682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49b00fff30 0x7f49b019a1b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f49a000e9c0 tx=0x7f49a000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:35.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.567+0000 7f49a67fc640 1 -- 172.21.15.116:0/2286620682 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a000cd80 con 0x7f49b00fff30 2024-03-19T01:19:35.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.567+0000 7f49a67fc640 1 -- 172.21.15.116:0/2286620682 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49a00035f0 con 0x7f49b00fff30 2024-03-19T01:19:35.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.567+0000 7f49a67fc640 1 -- 172.21.15.116:0/2286620682 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0010690 con 0x7f49b00fff30 2024-03-19T01:19:35.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.567+0000 7f49b771b640 1 -- 172.21.15.116:0/2286620682 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b01a1ba0 con 0x7f49b00fff30 2024-03-19T01:19:35.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.567+0000 7f49b771b640 1 -- 172.21.15.116:0/2286620682 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b01a20a0 con 0x7f49b00fff30 2024-03-19T01:19:35.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.570+0000 7f49a67fc640 1 -- 172.21.15.116:0/2286620682 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f49a00037a0 con 0x7f49b00fff30 2024-03-19T01:19:35.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.570+0000 7f49ba3e3740 1 Processor -- start 2024-03-19T01:19:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.570+0000 7f49ba3e3740 1 -- start start 2024-03-19T01:19:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.570+0000 7f49ba3e3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2100 0x55c056bd2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.570+0000 7f49ba3e3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd2af0 0x55c056bd75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.570+0000 7f49ba3e3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd7c60 0x55c056bda0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.570+0000 7f49ba3e3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c056b36be0 con 0x55c056bd7c60 2024-03-19T01:19:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.570+0000 7f49ba3e3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c056adf960 con 0x55c056bd2af0 2024-03-19T01:19:35.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.570+0000 7f49ba3e3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c056b32c80 con 0x55c056bd2100 2024-03-19T01:19:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.571+0000 7f49a67fc640 1 --2- 172.21.15.116:0/2286620682 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f499c0a2a60 0x7f499c0a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.571+0000 7f49b4cb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd7c60 0x55c056bda0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.571+0000 7f49b4cb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd7c60 0x55c056bda0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58740/0 (socket says 172.21.15.116:58740) 2024-03-19T01:19:35.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.571+0000 7f49b4cb6640 1 -- 172.21.15.116:0/3021603186 learned_addr learned my addr 172.21.15.116:0/3021603186 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.571+0000 7f49a67fc640 1 -- 172.21.15.116:0/2286620682 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f49a00ceea0 con 0x7f49b00fff30 2024-03-19T01:19:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.571+0000 7f49b5cb8640 1 --2- 172.21.15.116:0/3021603186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2100 0x55c056bd2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.572+0000 7f49b4cb6640 1 -- 172.21.15.116:0/3021603186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2100 msgr2=0x55c056bd2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.572+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/3021603186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2100 0x55c056bd2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.572+0000 7f49b4cb6640 1 -- 172.21.15.116:0/3021603186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd2af0 msgr2=0x55c056bd75f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:19:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.572+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/3021603186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd2af0 0x55c056bd75f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.572+0000 7f49b4cb6640 1 -- 172.21.15.116:0/3021603186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c056ad6d20 con 0x55c056bd7c60 2024-03-19T01:19:35.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.572+0000 7f49b5cb8640 1 --2- 172.21.15.116:0/2286620682 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f499c0a2a60 0x7f499c0a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.573+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/3021603186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd7c60 0x55c056bda0f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f49a009b7b0 tx=0x7f49a006e5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be3c3ae1cfa40cc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.574+0000 7f499b7fe640 1 -- 172.21.15.116:0/3021603186 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0066ac0 con 0x55c056bd7c60 2024-03-19T01:19:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.574+0000 7f499b7fe640 1 -- 172.21.15.116:0/3021603186 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49a009bf20 con 0x55c056bd7c60 2024-03-19T01:19:35.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.574+0000 7f499b7fe640 1 -- 172.21.15.116:0/3021603186 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0003b10 con 0x55c056bd7c60 2024-03-19T01:19:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.574+0000 7f49ba3e3740 1 -- 172.21.15.116:0/3021603186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd7c60 msgr2=0x55c056bda0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:35.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.574+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/3021603186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd7c60 0x55c056bda0f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f49a009b7b0 tx=0x7f49a006e5d0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.574+0000 7f49b5cb8640 1 --2- 172.21.15.116:0/2286620682 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f499c0a2a60 0x7f499c0a4f20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f49a8006010 tx=0x7f49a802e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.575+0000 7f49ba3e3740 1 -- 172.21.15.116:0/3021603186 shutdown_connections 2024-03-19T01:19:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.575+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/3021603186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd7c60 0x55c056bda0f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.575+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/3021603186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd2af0 0x55c056bd75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.575+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/3021603186 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2100 0x55c056bd2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.575+0000 7f49ba3e3740 1 -- 172.21.15.116:0/3021603186 >> 172.21.15.116:0/3021603186 conn(0x55c056bcd670 msgr2=0x55c056bcda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.575+0000 7f49ba3e3740 1 -- 172.21.15.116:0/3021603186 shutdown_connections 2024-03-19T01:19:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.575+0000 7f49ba3e3740 1 -- 172.21.15.116:0/3021603186 wait complete. 2024-03-19T01:19:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.575+0000 7f49ba3e3740 1 -- start start 2024-03-19T01:19:35.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.575+0000 7f49ba3e3740 1 Processor -- start 2024-03-19T01:19:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49ba3e3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056be4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49ba3e3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056be4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49ba3e3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056bebed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49ba3e3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55c056bdabb0 con 0x55c056bd2100 2024-03-19T01:19:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49ba3e3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55c056af2db0 con 0x55c056bd7c60 2024-03-19T01:19:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49ba3e3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55c056bdad30 con 0x55c056bd2af0 2024-03-19T01:19:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49ba3e3740 10 client.0 _ll_register_callbacks cb 0x55c056af7e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:19:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49b5cb8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056bebed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49b5cb8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056bebed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:57288/0 (socket says 172.21.15.116:57288) 2024-03-19T01:19:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49b5cb8640 1 -- 172.21.15.116:0/2844544832 learned_addr learned my addr 172.21.15.116:0/2844544832 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056be4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49b54b7640 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056be4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49b5cb8640 1 -- 172.21.15.116:0/2844544832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 msgr2=0x55c056be4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49b5cb8640 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056be4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49b5cb8640 1 -- 172.21.15.116:0/2844544832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 msgr2=0x55c056be4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:35.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49b5cb8640 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056be4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.576+0000 7f49b5cb8640 1 -- 172.21.15.116:0/2844544832 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c056af1c60 con 0x55c056bd7c60 2024-03-19T01:19:35.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.577+0000 7f49b54b7640 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056be4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:19:35.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.577+0000 7f49b5cb8640 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056bebed0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f49a8036000 tx=0x7f49a80095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3bf35942db3f487 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:35.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.577+0000 7f4998ff9640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:19:35.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.577+0000 7f4998ff9640 1 -- 172.21.15.116:0/2844544832 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a8051060 con 0x55c056bd7c60 2024-03-19T01:19:35.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.578+0000 7f4998ff9640 1 -- 172.21.15.116:0/2844544832 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f49a8003b70 con 0x55c056bd7c60 2024-03-19T01:19:35.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.578+0000 7f49ba3e3740 1 -- 172.21.15.116:0/2844544832 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49a009bf20 con 0x55c056bd7c60 2024-03-19T01:19:35.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.579+0000 7f4998ff9640 1 -- 172.21.15.116:0/2844544832 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f49a8039040 con 0x55c056bd7c60 2024-03-19T01:19:35.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.579+0000 7f49ba3e3740 10 client.25682 fetch_fsmap learned FSMap version 45 2024-03-19T01:19:35.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.579+0000 7f49ba3e3740 1 -- 172.21.15.116:0/2844544832 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55c05684b210 con 0x55c056bd7c60 2024-03-19T01:19:35.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.579+0000 7f4998ff9640 1 -- 172.21.15.116:0/2844544832 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49a009bf20 con 0x55c056bd7c60 2024-03-19T01:19:35.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.579+0000 7f4998ff9640 1 -- 172.21.15.116:0/2844544832 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f49a803f040 con 0x55c056bd7c60 2024-03-19T01:19:35.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.579+0000 7f49ba3e3740 10 client.25682 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:19:35.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.579+0000 7f49ba3e3740 10 client.25682 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:19:35.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.579+0000 7f49ba3e3740 20 client.25682 populate_metadata read hostname 'smithi116' 2024-03-19T01:19:35.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.579+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c056bf07c0 0x55c056bf2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.579+0000 7f49ba3e3740 4 client.25682 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:19:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.579+0000 7f49ba3e3740 1 -- 172.21.15.116:0/2844544832 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c056b44120 con 0x55c056bf07c0 2024-03-19T01:19:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.579+0000 7f49b54b7640 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c056bf07c0 0x55c056bf2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.580+0000 7f49b54b7640 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c056bf07c0 0x55c056bf2bb0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af339a890d018781 server_cookie=9734c70b371ed278 in_seq=0 out_seq=0 2024-03-19T01:19:35.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.580+0000 7f4998ff9640 10 client.25682 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.581+0000 7f4998ff9640 1 -- 172.21.15.116:0/2844544832 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c056b44120 con 0x55c056bf07c0 2024-03-19T01:19:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.581+0000 7f4998ff9640 10 client.25682 handle_command_reply: tid=0 2024-03-19T01:19:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.581+0000 7f49ba3e3740 1 client.25682 shutdown 2024-03-19T01:19:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.582+0000 7f49ba3e3740 1 -- 172.21.15.116:0/2844544832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 msgr2=0x55c056bebed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.582+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056bebed0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f49a8036000 tx=0x7f49a80095c0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.582+0000 7f49ba3e3740 1 -- 172.21.15.116:0/2844544832 shutdown_connections 2024-03-19T01:19:35.583 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.582+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056bebed0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.582+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c056bf07c0 0x55c056bf2bb0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.582+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056be4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.582+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/2844544832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056be4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.582+0000 7f4998ff9640 0 client.25682 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:35.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.582+0000 7f49ba3e3740 1 -- 172.21.15.116:0/2844544832 >> 172.21.15.116:0/2844544832 conn(0x55c056bcd670 msgr2=0x55c056bce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:35.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.583+0000 7f49ba3e3740 1 -- 172.21.15.116:0/2844544832 shutdown_connections 2024-03-19T01:19:35.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.583+0000 7f49ba3e3740 1 -- 172.21.15.116:0/2844544832 wait complete. 2024-03-19T01:19:35.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.583+0000 7f49ba3e3740 20 client.25682 trim_cache size 0 max 16384 2024-03-19T01:19:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.608+0000 7f49ba3e3740 1 Processor -- start 2024-03-19T01:19:35.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.608+0000 7f49ba3e3740 1 -- start start 2024-03-19T01:19:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.609+0000 7f49ba3e3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056bcc220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.609+0000 7f49ba3e3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056bebe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.609+0000 7f49b54b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056bcc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.609+0000 7f49b54b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056bcc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58756/0 (socket says 172.21.15.116:58756) 2024-03-19T01:19:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.609+0000 7f49b54b7640 1 -- 172.21.15.116:0/4182217605 learned_addr learned my addr 172.21.15.116:0/4182217605 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.609+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/4182217605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056bcc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.610+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/4182217605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056bebe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.610+0000 7f49ba3e3740 1 -- 172.21.15.116:0/4182217605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f49b01a6c70 con 0x55c056bd2100 2024-03-19T01:19:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.610+0000 7f49ba3e3740 1 -- 172.21.15.116:0/4182217605 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49b01a87b0 con 0x55c056bd7c60 2024-03-19T01:19:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.610+0000 7f49b5cb8640 1 --2- 172.21.15.116:0/4182217605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056bcc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.610+0000 7f49ba3e3740 1 -- 172.21.15.116:0/4182217605 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f49b01a7340 con 0x55c056bd2af0 2024-03-19T01:19:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.610+0000 7f49b4cb6640 1 -- 172.21.15.116:0/4182217605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 msgr2=0x55c056bcc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.610+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/4182217605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056bcc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.610+0000 7f49b4cb6640 1 -- 172.21.15.116:0/4182217605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 msgr2=0x55c056bcc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:35.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.610+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/4182217605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056bcc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.611+0000 7f49b4cb6640 1 -- 172.21.15.116:0/4182217605 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c056bf2f20 con 0x55c056bd2af0 2024-03-19T01:19:35.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.619+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/4182217605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056bebe90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55c056bda7e0 tx=0x7f49a0009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36d6ed6dd5a0a759 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.624+0000 7f499affd640 1 -- 172.21.15.116:0/4182217605 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a009f970 con 0x55c056bd2af0 2024-03-19T01:19:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.624+0000 7f499affd640 1 -- 172.21.15.116:0/4182217605 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49a009b820 con 0x55c056bd2af0 2024-03-19T01:19:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.624+0000 7f499affd640 1 -- 172.21.15.116:0/4182217605 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0003b10 con 0x55c056bd2af0 2024-03-19T01:19:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.627+0000 7f49ba3e3740 1 -- 172.21.15.116:0/4182217605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 msgr2=0x55c056bebe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.627+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/4182217605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056bebe90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x55c056bda7e0 tx=0x7f49a0009310 comp rx=0 tx=0).stop 2024-03-19T01:19:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.629+0000 7f49ba3e3740 1 -- 172.21.15.116:0/4182217605 shutdown_connections 2024-03-19T01:19:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.629+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/4182217605 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056bcc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.629 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.629+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/4182217605 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056bebe90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.629+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/4182217605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056bcc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.630 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.629+0000 7f49ba3e3740 1 -- 172.21.15.116:0/4182217605 >> 172.21.15.116:0/4182217605 conn(0x55c056bcd670 msgr2=0x55c056bcda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:35.633 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.633+0000 7f49ba3e3740 1 -- 172.21.15.116:0/4182217605 shutdown_connections 2024-03-19T01:19:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.634+0000 7f49ba3e3740 1 -- 172.21.15.116:0/4182217605 wait complete. 2024-03-19T01:19:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.634+0000 7f49ba3e3740 1 -- start start 2024-03-19T01:19:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.634+0000 7f49ba3e3740 1 Processor -- start 2024-03-19T01:19:35.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.635+0000 7f49ba3e3740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056be9970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.635+0000 7f49ba3e3740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056be9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.635+0000 7f49ba3e3740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056beae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.635+0000 7f49ba3e3740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f49b01a80e0 con 0x55c056bd2100 2024-03-19T01:19:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49ba3e3740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49b01a7a10 con 0x55c056bd7c60 2024-03-19T01:19:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49b4cb6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056be9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49ba3e3740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f49b01a4c70 con 0x55c056bd2af0 2024-03-19T01:19:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49ba3e3740 10 client.0 _ll_register_callbacks cb 0x7f49a0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:19:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49b4cb6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056be9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41318/0 (socket says 172.21.15.116:41318) 2024-03-19T01:19:35.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49b5cb8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056beae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49b4cb6640 1 -- 172.21.15.116:0/3444792827 learned_addr learned my addr 172.21.15.116:0/3444792827 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49b4cb6640 1 -- 172.21.15.116:0/3444792827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 msgr2=0x55c056beae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/3444792827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056beae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49b54b7640 1 --2- 172.21.15.116:0/3444792827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056be9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49b4cb6640 1 -- 172.21.15.116:0/3444792827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 msgr2=0x55c056be9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/3444792827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056be9970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49b4cb6640 1 -- 172.21.15.116:0/3444792827 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c056917620 con 0x55c056bd2af0 2024-03-19T01:19:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.636+0000 7f49b54b7640 1 --2- 172.21.15.116:0/3444792827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056be9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:19:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.637+0000 7f49b4cb6640 1 --2- 172.21.15.116:0/3444792827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056be9eb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009380 tx=0x7f49a00cf230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1eb5f8767ebed6e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.637+0000 7f4998ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:19:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.637+0000 7f4998ff9640 1 -- 172.21.15.116:0/3444792827 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a00a3fc0 con 0x55c056bd2af0 2024-03-19T01:19:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.637+0000 7f4998ff9640 1 -- 172.21.15.116:0/3444792827 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f49a0066a50 con 0x55c056bd2af0 2024-03-19T01:19:35.638 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.637+0000 7f49ba3e3740 1 -- 172.21.15.116:0/3444792827 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49a009b820 con 0x55c056bd2af0 2024-03-19T01:19:35.639 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.638+0000 7f4998ff9640 1 -- 172.21.15.116:0/3444792827 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f49a001fdc0 con 0x55c056bd2af0 2024-03-19T01:19:35.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.638+0000 7f49ba3e3740 10 client.25678 fetch_fsmap learned FSMap version 45 2024-03-19T01:19:35.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.638+0000 7f49ba3e3740 1 -- 172.21.15.116:0/3444792827 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55c056c56c70 con 0x55c056bd2af0 2024-03-19T01:19:35.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.638+0000 7f4998ff9640 1 -- 172.21.15.116:0/3444792827 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49a0003b10 con 0x55c056bd2af0 2024-03-19T01:19:35.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.639+0000 7f4998ff9640 1 -- 172.21.15.116:0/3444792827 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f49a009ac70 con 0x55c056bd2af0 2024-03-19T01:19:35.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.639+0000 7f49ba3e3740 10 client.25678 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:19:35.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.639+0000 7f49ba3e3740 10 client.25678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:19:35.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.639+0000 7f49ba3e3740 20 client.25678 populate_metadata read hostname 'smithi116' 2024-03-19T01:19:35.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.639+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/3444792827 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c056c57060 0x55c056c59480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.639+0000 7f49ba3e3740 4 client.25678 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:19:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.639+0000 7f49ba3e3740 1 -- 172.21.15.116:0/3444792827 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f49a009b820 con 0x55c056c57060 2024-03-19T01:19:35.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.641+0000 7f49b54b7640 1 --2- 172.21.15.116:0/3444792827 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c056c57060 0x55c056c59480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:35.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.643+0000 7f49b54b7640 1 --2- 172.21.15.116:0/3444792827 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c056c57060 0x55c056c59480 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ab378b9cea9d9df server_cookie=e98693525e4d8f06 in_seq=0 out_seq=0 2024-03-19T01:19:35.644 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.643+0000 7f4998ff9640 10 client.25678 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:35 smithi162 ceph-mon[67178]: pgmap v561: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 857 op/s 2024-03-19T01:19:35.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:35 smithi116 ceph-mon[62701]: pgmap v561: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 857 op/s 2024-03-19T01:19:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.660+0000 7f4998ff9640 1 -- 172.21.15.116:0/3444792827 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f49a009b820 con 0x55c056c57060 2024-03-19T01:19:35.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.660+0000 7f4998ff9640 10 client.25678 handle_command_reply: tid=0 2024-03-19T01:19:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.661+0000 7f49ba3e3740 1 client.25678 shutdown 2024-03-19T01:19:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.661+0000 7f49ba3e3740 1 -- 172.21.15.116:0/3444792827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 msgr2=0x55c056be9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.661+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/3444792827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056be9eb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009380 tx=0x7f49a00cf230 comp rx=0 tx=0).stop 2024-03-19T01:19:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.661+0000 7f49ba3e3740 1 -- 172.21.15.116:0/3444792827 shutdown_connections 2024-03-19T01:19:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.661+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/3444792827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55c056bd7c60 0x55c056beae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.661+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/3444792827 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55c056c57060 0x55c056c59480 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.661+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/3444792827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55c056bd2af0 0x55c056be9eb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.661+0000 7f49ba3e3740 1 --2- 172.21.15.116:0/3444792827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55c056bd2100 0x55c056be9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.661+0000 7f49ba3e3740 1 -- 172.21.15.116:0/3444792827 >> 172.21.15.116:0/3444792827 conn(0x55c056bcd670 msgr2=0x55c056bcc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.661+0000 7f4998ff9640 0 client.25678 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:35.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.661+0000 7f49ba3e3740 1 -- 172.21.15.116:0/3444792827 shutdown_connections 2024-03-19T01:19:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.661+0000 7f49ba3e3740 1 -- 172.21.15.116:0/3444792827 wait complete. 2024-03-19T01:19:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.661+0000 7f49ba3e3740 20 client.25678 trim_cache size 0 max 16384 2024-03-19T01:19:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:19:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:19:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:19:35.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:19:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.662+0000 7f49b771b640 1 -- 172.21.15.116:0/2286620682 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f499c0a2a60 msgr2=0x7f499c0a4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.662+0000 7f49b771b640 1 --2- 172.21.15.116:0/2286620682 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f499c0a2a60 0x7f499c0a4f20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f49a8006010 tx=0x7f49a802e040 comp rx=0 tx=0).stop 2024-03-19T01:19:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.662+0000 7f49b771b640 1 -- 172.21.15.116:0/2286620682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49b00fff30 msgr2=0x7f49b019a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.662+0000 7f49b771b640 1 --2- 172.21.15.116:0/2286620682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49b00fff30 0x7f49b019a1b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f49a000e9c0 tx=0x7f49a000ee80 comp rx=0 tx=0).stop 2024-03-19T01:19:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.663+0000 7f49b771b640 1 -- 172.21.15.116:0/2286620682 shutdown_connections 2024-03-19T01:19:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.663+0000 7f49b771b640 1 --2- 172.21.15.116:0/2286620682 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49b01074c0 0x7f49b01a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.663+0000 7f49b771b640 1 --2- 172.21.15.116:0/2286620682 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f49b00fff30 0x7f49b019a1b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.663+0000 7f49b771b640 1 --2- 172.21.15.116:0/2286620682 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f499c0a2a60 0x7f499c0a4f20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.663+0000 7f49b771b640 1 --2- 172.21.15.116:0/2286620682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b00ff5d0 0x7f49b0199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:35.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.663+0000 7f49b771b640 1 -- 172.21.15.116:0/2286620682 >> 172.21.15.116:0/2286620682 conn(0x7f49b00fb000 msgr2=0x7f49b00fb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.663+0000 7f49b771b640 1 -- 172.21.15.116:0/2286620682 shutdown_connections 2024-03-19T01:19:35.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:35.663+0000 7f49b771b640 1 -- 172.21.15.116:0/2286620682 wait complete. 2024-03-19T01:19:35.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:35 smithi156 ceph-mon[66799]: pgmap v561: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 MiB/s wr, 857 op/s 2024-03-19T01:19:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 410.0 s, 2658.6 tps, lat 37.480 ms stddev 44.012 2024-03-19T01:19:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:37 smithi162 ceph-mon[67178]: pgmap v562: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.6 MiB/s wr, 708 op/s 2024-03-19T01:19:37.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:37 smithi116 ceph-mon[62701]: pgmap v562: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.6 MiB/s wr, 708 op/s 2024-03-19T01:19:37.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:37 smithi156 ceph-mon[66799]: pgmap v562: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.6 MiB/s wr, 708 op/s 2024-03-19T01:19:39.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:39 smithi162 ceph-mon[67178]: pgmap v563: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 711 op/s 2024-03-19T01:19:39.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:39 smithi116 ceph-mon[62701]: pgmap v563: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 711 op/s 2024-03-19T01:19:39.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:39 smithi156 ceph-mon[66799]: pgmap v563: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.3 MiB/s wr, 711 op/s 2024-03-19T01:19:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 415.0 s, 2213.8 tps, lat 45.305 ms stddev 51.582 2024-03-19T01:19:41.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:41 smithi162 ceph-mon[67178]: pgmap v564: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.6 MiB/s wr, 648 op/s 2024-03-19T01:19:41.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:41 smithi116 ceph-mon[62701]: pgmap v564: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.6 MiB/s wr, 648 op/s 2024-03-19T01:19:41.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:41 smithi156 ceph-mon[66799]: pgmap v564: 97 pgs: 97 active+clean; 1.6 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.6 MiB/s wr, 648 op/s 2024-03-19T01:19:43.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:43 smithi162 ceph-mon[67178]: pgmap v565: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.5 MiB/s wr, 572 op/s 2024-03-19T01:19:43.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:43 smithi116 ceph-mon[62701]: pgmap v565: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.5 MiB/s wr, 572 op/s 2024-03-19T01:19:43.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:43 smithi156 ceph-mon[66799]: pgmap v565: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.5 MiB/s wr, 572 op/s 2024-03-19T01:19:45.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:45 smithi162 ceph-mon[67178]: pgmap v566: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.5 MiB/s wr, 732 op/s 2024-03-19T01:19:45.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:45 smithi116 ceph-mon[62701]: pgmap v566: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.5 MiB/s wr, 732 op/s 2024-03-19T01:19:45.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:45 smithi156 ceph-mon[66799]: pgmap v566: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.5 MiB/s wr, 732 op/s 2024-03-19T01:19:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 420.0 s, 2735.8 tps, lat 36.118 ms stddev 44.868 2024-03-19T01:19:47.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:47 smithi162 ceph-mon[67178]: pgmap v567: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 640 op/s 2024-03-19T01:19:47.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:47 smithi116 ceph-mon[62701]: pgmap v567: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 640 op/s 2024-03-19T01:19:47.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:47 smithi156 ceph-mon[66799]: pgmap v567: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.2 MiB/s wr, 640 op/s 2024-03-19T01:19:48.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.773+0000 7f71c86b2640 1 -- 172.21.15.116:0/2783732886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0109930 msgr2=0x7f71c010bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.773+0000 7f71c86b2640 1 --2- 172.21.15.116:0/2783732886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0109930 0x7f71c010bd20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f71b0009900 tx=0x7f71b002bc30 comp rx=0 tx=0).stop 2024-03-19T01:19:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.773+0000 7f71c86b2640 1 -- 172.21.15.116:0/2783732886 shutdown_connections 2024-03-19T01:19:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.773+0000 7f71c86b2640 1 --2- 172.21.15.116:0/2783732886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71c010ebe0 0x7f71c0111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.773+0000 7f71c86b2640 1 --2- 172.21.15.116:0/2783732886 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71c010c260 0x7f71c010e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.773+0000 7f71c86b2640 1 --2- 172.21.15.116:0/2783732886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0109930 0x7f71c010bd20 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.773+0000 7f71c86b2640 1 -- 172.21.15.116:0/2783732886 >> 172.21.15.116:0/2783732886 conn(0x7f71c006ea20 msgr2=0x7f71c0070e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.773+0000 7f71c86b2640 1 -- 172.21.15.116:0/2783732886 shutdown_connections 2024-03-19T01:19:48.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.773+0000 7f71c86b2640 1 -- 172.21.15.116:0/2783732886 wait complete. 2024-03-19T01:19:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c86b2640 1 Processor -- start 2024-03-19T01:19:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c86b2640 1 -- start start 2024-03-19T01:19:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c86b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0109930 0x7f71c01a9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c86b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71c010c260 0x7f71c01ab670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c86b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71c010ebe0 0x7f71c01adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c86b2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0114c30 con 0x7f71c0109930 2024-03-19T01:19:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c86b2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0114ab0 con 0x7f71c010c260 2024-03-19T01:19:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c86b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71c0114db0 con 0x7f71c010ebe0 2024-03-19T01:19:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c644e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0109930 0x7f71c01a9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c644e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0109930 0x7f71c01a9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46524/0 (socket says 172.21.15.116:46524) 2024-03-19T01:19:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c644e640 1 -- 172.21.15.116:0/4217923615 learned_addr learned my addr 172.21.15.116:0/4217923615 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c6c4f640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71c010ebe0 0x7f71c01adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c5c4d640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71c010c260 0x7f71c01ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c644e640 1 -- 172.21.15.116:0/4217923615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71c010ebe0 msgr2=0x7f71c01adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c644e640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71c010ebe0 0x7f71c01adc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c644e640 1 -- 172.21.15.116:0/4217923615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71c010c260 msgr2=0x7f71c01ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c644e640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71c010c260 0x7f71c01ab670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.774+0000 7f71c644e640 1 -- 172.21.15.116:0/4217923615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c01ae4e0 con 0x7f71c0109930 2024-03-19T01:19:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.775+0000 7f71c5c4d640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71c010c260 0x7f71c01ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:19:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.775+0000 7f71c644e640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0109930 0x7f71c01a9120 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f71b002c240 tx=0x7f71b002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.775+0000 7f71b77fe640 1 -- 172.21.15.116:0/4217923615 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0043020 con 0x7f71c0109930 2024-03-19T01:19:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.775+0000 7f71b77fe640 1 -- 172.21.15.116:0/4217923615 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b00054e0 con 0x7f71c0109930 2024-03-19T01:19:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.775+0000 7f71c86b2640 1 -- 172.21.15.116:0/4217923615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71c01ae770 con 0x7f71c0109930 2024-03-19T01:19:48.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.775+0000 7f71c86b2640 1 -- 172.21.15.116:0/4217923615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71c01aece0 con 0x7f71c0109930 2024-03-19T01:19:48.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.775+0000 7f71b77fe640 1 -- 172.21.15.116:0/4217923615 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0003c40 con 0x7f71c0109930 2024-03-19T01:19:48.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.778+0000 7f71b77fe640 1 -- 172.21.15.116:0/4217923615 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f71b0046020 con 0x7f71c0109930 2024-03-19T01:19:48.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.778+0000 7f71cb379740 1 Processor -- start 2024-03-19T01:19:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71cb379740 1 -- start start 2024-03-19T01:19:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71b77fe640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f71ac0a2920 0x7f71ac0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71cb379740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486100 0x562a68486520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71cb379740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486af0 0x562a6848b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71cb379740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 0x562a6848e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71cb379740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562a683eabe0 con 0x562a68486af0 2024-03-19T01:19:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71cb379740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562a68393960 con 0x562a6848bc60 2024-03-19T01:19:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71cb379740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562a683e6c80 con 0x562a68486100 2024-03-19T01:19:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71c6c4f640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f71ac0a2920 0x7f71ac0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71c644e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486100 0x562a68486520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71c5c4d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486af0 0x562a6848b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71c5c4d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486af0 0x562a6848b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46526/0 (socket says 172.21.15.116:46526) 2024-03-19T01:19:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71c5c4d640 1 -- 172.21.15.116:0/4239966680 learned_addr learned my addr 172.21.15.116:0/4239966680 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71c6c4f640 1 --2- 172.21.15.116:0/4239966680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 0x562a6848e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71b77fe640 1 -- 172.21.15.116:0/4217923615 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f71b00f0ad0 con 0x7f71c0109930 2024-03-19T01:19:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71c5c4d640 1 -- 172.21.15.116:0/4239966680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486100 msgr2=0x562a68486520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71c5c4d640 1 --2- 172.21.15.116:0/4239966680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486100 0x562a68486520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71c5c4d640 1 -- 172.21.15.116:0/4239966680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 msgr2=0x562a6848e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71c5c4d640 1 --2- 172.21.15.116:0/4239966680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 0x562a6848e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71c5c4d640 1 -- 172.21.15.116:0/4239966680 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a6838ad20 con 0x562a68486af0 2024-03-19T01:19:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.779+0000 7f71c644e640 1 --2- 172.21.15.116:0/4239966680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486100 0x562a68486520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:19:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.780+0000 7f71c6c4f640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f71ac0a2920 0x7f71ac0a4de0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f71b8005fd0 tx=0x7f71b8009f10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.780+0000 7f71c5c4d640 1 --2- 172.21.15.116:0/4239966680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486af0 0x562a6848b5f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009c20 tx=0x7f71bc010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85cd76915112cdb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.780+0000 7f71b4ff9640 1 -- 172.21.15.116:0/4239966680 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc009e50 con 0x562a68486af0 2024-03-19T01:19:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.780+0000 7f71b4ff9640 1 -- 172.21.15.116:0/4239966680 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71bc0035f0 con 0x562a68486af0 2024-03-19T01:19:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.780+0000 7f71b4ff9640 1 -- 172.21.15.116:0/4239966680 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc0126b0 con 0x562a68486af0 2024-03-19T01:19:48.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.780+0000 7f71cb379740 1 -- 172.21.15.116:0/4239966680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486af0 msgr2=0x562a6848b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.780+0000 7f71cb379740 1 --2- 172.21.15.116:0/4239966680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486af0 0x562a6848b5f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009c20 tx=0x7f71bc010740 comp rx=0 tx=0).stop 2024-03-19T01:19:48.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 -- 172.21.15.116:0/4239966680 shutdown_connections 2024-03-19T01:19:48.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 --2- 172.21.15.116:0/4239966680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 0x562a6848e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 --2- 172.21.15.116:0/4239966680 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486af0 0x562a6848b5f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 --2- 172.21.15.116:0/4239966680 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486100 0x562a68486520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 -- 172.21.15.116:0/4239966680 >> 172.21.15.116:0/4239966680 conn(0x562a68481670 msgr2=0x562a68481a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:48.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 -- 172.21.15.116:0/4239966680 shutdown_connections 2024-03-19T01:19:48.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 -- 172.21.15.116:0/4239966680 wait complete. 2024-03-19T01:19:48.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 -- start start 2024-03-19T01:19:48.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 Processor -- start 2024-03-19T01:19:48.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a68486100 0x562a68490070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a684905b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a6848bc60 0x562a684975f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562a6848ec10 con 0x562a6848bc60 2024-03-19T01:19:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562a683a6db0 con 0x562a68486100 2024-03-19T01:19:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562a6848ed90 con 0x562a68486af0 2024-03-19T01:19:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71cb379740 10 client.0 _ll_register_callbacks cb 0x562a683abe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:19:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71c6c4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a6848bc60 0x562a684975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71c6c4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a6848bc60 0x562a684975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46530/0 (socket says 172.21.15.116:46530) 2024-03-19T01:19:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71c6c4f640 1 -- 172.21.15.116:0/532404554 learned_addr learned my addr 172.21.15.116:0/532404554 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.781+0000 7f71c644e640 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a68486100 0x562a68490070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71c5c4d640 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a684905b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71c644e640 1 -- 172.21.15.116:0/532404554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 msgr2=0x562a684905b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71c644e640 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a684905b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71c644e640 1 -- 172.21.15.116:0/532404554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a6848bc60 msgr2=0x562a684975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:48.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71c644e640 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a6848bc60 0x562a684975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71c644e640 1 -- 172.21.15.116:0/532404554 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a683a5c60 con 0x562a68486100 2024-03-19T01:19:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71c5c4d640 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a684905b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:19:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71c6c4f640 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a6848bc60 0x562a684975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:19:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71c644e640 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a68486100 0x562a68490070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f71b00bd0f0 tx=0x7f71b0048f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=995a14adeca16d24 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:19:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71a9ffb640 1 -- 172.21.15.116:0/532404554 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b0005690 con 0x562a68486100 2024-03-19T01:19:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71cb379740 1 -- 172.21.15.116:0/532404554 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f71bc0035f0 con 0x562a68486100 2024-03-19T01:19:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71a9ffb640 1 -- 172.21.15.116:0/532404554 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f71b00043c0 con 0x562a68486100 2024-03-19T01:19:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.782+0000 7f71a9ffb640 1 -- 172.21.15.116:0/532404554 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f71b0004c60 con 0x562a68486100 2024-03-19T01:19:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.783+0000 7f71cb379740 10 client.25697 fetch_fsmap learned FSMap version 45 2024-03-19T01:19:48.792 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.783+0000 7f71cb379740 1 -- 172.21.15.116:0/532404554 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x562a680ff210 con 0x562a68486100 2024-03-19T01:19:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.783+0000 7f71a9ffb640 1 -- 172.21.15.116:0/532404554 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71bc0035f0 con 0x562a68486100 2024-03-19T01:19:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.783+0000 7f71a9ffb640 1 -- 172.21.15.116:0/532404554 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f71b00c1140 con 0x562a68486100 2024-03-19T01:19:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.783+0000 7f71cb379740 10 client.25697 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:19:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.783+0000 7f71cb379740 10 client.25697 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:19:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.783+0000 7f71cb379740 20 client.25697 populate_metadata read hostname 'smithi116' 2024-03-19T01:19:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.783+0000 7f71cb379740 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562a6849c000 0x562a6849e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.783+0000 7f71cb379740 4 client.25697 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:19:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.783+0000 7f71cb379740 1 -- 172.21.15.116:0/532404554 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a683f8120 con 0x562a6849c000 2024-03-19T01:19:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.784+0000 7f71c5c4d640 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562a6849c000 0x562a6849e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.786+0000 7f71c5c4d640 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562a6849c000 0x562a6849e490 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbb2d784e5febce1 server_cookie=a26e0dd5c4e6f404 in_seq=0 out_seq=0 2024-03-19T01:19:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.786+0000 7f71a9ffb640 10 client.25697 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:48.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.786+0000 7f71a9ffb640 1 -- 172.21.15.116:0/532404554 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a683f8120 con 0x562a6849c000 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.786+0000 7f71a9ffb640 10 client.25697 handle_command_reply: tid=0 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.786+0000 7f71cb379740 1 client.25697 shutdown 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.787+0000 7f71cb379740 1 -- 172.21.15.116:0/532404554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a68486100 msgr2=0x562a68490070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.787+0000 7f71cb379740 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a68486100 0x562a68490070 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f71b00bd0f0 tx=0x7f71b0048f90 comp rx=0 tx=0).stop 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.787+0000 7f71cb379740 1 -- 172.21.15.116:0/532404554 shutdown_connections 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.787+0000 7f71cb379740 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a6848bc60 0x562a684975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.787+0000 7f71cb379740 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562a6849c000 0x562a6849e490 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.787+0000 7f71cb379740 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a684905b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.787+0000 7f71cb379740 1 --2- 172.21.15.116:0/532404554 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a68486100 0x562a68490070 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.787+0000 7f71a9ffb640 0 client.25697 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.787+0000 7f71cb379740 1 -- 172.21.15.116:0/532404554 >> 172.21.15.116:0/532404554 conn(0x562a68481670 msgr2=0x562a684828c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.787+0000 7f71cb379740 1 -- 172.21.15.116:0/532404554 shutdown_connections 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.787+0000 7f71cb379740 1 -- 172.21.15.116:0/532404554 wait complete. 2024-03-19T01:19:48.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.787+0000 7f71cb379740 20 client.25697 trim_cache size 0 max 16384 2024-03-19T01:19:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71cb379740 1 Processor -- start 2024-03-19T01:19:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71cb379740 1 -- start start 2024-03-19T01:19:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71cb379740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486100 0x562a6848b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71cb379740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a684975b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71cb379740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 0x562a6848b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71cb379740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01b7a30 con 0x562a68486100 2024-03-19T01:19:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71cb379740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01b9570 con 0x562a6848bc60 2024-03-19T01:19:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71cb379740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01b8100 con 0x562a68486af0 2024-03-19T01:19:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71c644e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486100 0x562a6848b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71c6c4f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 0x562a6848b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71c644e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486100 0x562a6848b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46534/0 (socket says 172.21.15.116:46534) 2024-03-19T01:19:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71c6c4f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 0x562a6848b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:46792/0 (socket says 172.21.15.116:46792) 2024-03-19T01:19:48.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71c5c4d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a684975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71c644e640 1 -- 172.21.15.116:0/3881936716 learned_addr learned my addr 172.21.15.116:0/3881936716 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71c644e640 1 -- 172.21.15.116:0/3881936716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 msgr2=0x562a684975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71c644e640 1 --2- 172.21.15.116:0/3881936716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a684975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71c644e640 1 -- 172.21.15.116:0/3881936716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 msgr2=0x562a6848b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71c644e640 1 --2- 172.21.15.116:0/3881936716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 0x562a6848b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71c644e640 1 -- 172.21.15.116:0/3881936716 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a681cb620 con 0x562a68486100 2024-03-19T01:19:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.806+0000 7f71c5c4d640 1 --2- 172.21.15.116:0/3881936716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a684975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:19:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.807+0000 7f71c644e640 1 --2- 172.21.15.116:0/3881936716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486100 0x562a6848b070 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f71b0006010 tx=0x7f71b00ee7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddfd329fbc1bf672 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.807+0000 7f71abfff640 1 -- 172.21.15.116:0/3881936716 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b00bcf30 con 0x562a68486100 2024-03-19T01:19:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.807+0000 7f71abfff640 1 -- 172.21.15.116:0/3881936716 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f71b0005690 con 0x562a68486100 2024-03-19T01:19:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.807+0000 7f71abfff640 1 -- 172.21.15.116:0/3881936716 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71b00c5bd0 con 0x562a68486100 2024-03-19T01:19:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.807+0000 7f71cb379740 1 -- 172.21.15.116:0/3881936716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486100 msgr2=0x562a6848b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:48.810 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.807+0000 7f71cb379740 1 --2- 172.21.15.116:0/3881936716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486100 0x562a6848b070 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f71b0006010 tx=0x7f71b00ee7e0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.808+0000 7f71cb379740 1 -- 172.21.15.116:0/3881936716 shutdown_connections 2024-03-19T01:19:48.813 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.808+0000 7f71cb379740 1 --2- 172.21.15.116:0/3881936716 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 0x562a6848b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.808+0000 7f71cb379740 1 --2- 172.21.15.116:0/3881936716 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a684975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.808+0000 7f71cb379740 1 --2- 172.21.15.116:0/3881936716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486100 0x562a6848b070 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.808+0000 7f71cb379740 1 -- 172.21.15.116:0/3881936716 >> 172.21.15.116:0/3881936716 conn(0x562a68481670 msgr2=0x562a68481a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.808+0000 7f71cb379740 1 -- 172.21.15.116:0/3881936716 shutdown_connections 2024-03-19T01:19:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.808+0000 7f71cb379740 1 -- 172.21.15.116:0/3881936716 wait complete. 2024-03-19T01:19:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.809+0000 7f71cb379740 1 -- start start 2024-03-19T01:19:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.809+0000 7f71cb379740 1 Processor -- start 2024-03-19T01:19:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.809+0000 7f71cb379740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486100 0x562a68497460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71cb379740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a68493f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71cb379740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 0x562a68494440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71cb379740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01b8ea0 con 0x562a68486100 2024-03-19T01:19:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71cb379740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01b87d0 con 0x562a6848bc60 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71cb379740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f71c01b5a10 con 0x562a68486af0 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71c5c4d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a68493f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71c5c4d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a68493f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47324/0 (socket says 172.21.15.116:47324) 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71c5c4d640 1 -- 172.21.15.116:0/1551021481 learned_addr learned my addr 172.21.15.116:0/1551021481 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71cb379740 10 client.0 _ll_register_callbacks cb 0x7f71b00035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71c644e640 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486100 0x562a68497460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71c6c4f640 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 0x562a68494440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71c5c4d640 1 -- 172.21.15.116:0/1551021481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 msgr2=0x562a68494440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71c5c4d640 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 0x562a68494440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71c5c4d640 1 -- 172.21.15.116:0/1551021481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486100 msgr2=0x562a68497460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71c5c4d640 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486100 0x562a68497460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71c5c4d640 1 -- 172.21.15.116:0/1551021481 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a685003c0 con 0x562a68486af0 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.810+0000 7f71c644e640 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486100 0x562a68497460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.811+0000 7f71c5c4d640 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a68493f00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f71bc000c00 tx=0x7f71bc005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3bde8a602f4e722 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:19:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.811+0000 7f71a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.811+0000 7f71a9ffb640 1 -- 172.21.15.116:0/1551021481 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc009e50 con 0x562a68486af0 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.811+0000 7f71a9ffb640 1 -- 172.21.15.116:0/1551021481 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f71bc0050b0 con 0x562a68486af0 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.811+0000 7f71cb379740 1 -- 172.21.15.116:0/1551021481 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562a683f2470 con 0x562a68486af0 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.811+0000 7f71a9ffb640 1 -- 172.21.15.116:0/1551021481 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f71bc0052d0 con 0x562a68486af0 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.811+0000 7f71cb379740 10 client.25696 fetch_fsmap learned FSMap version 45 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.811+0000 7f71cb379740 1 -- 172.21.15.116:0/1551021481 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x562a685015e0 con 0x562a68486af0 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.811+0000 7f71a9ffb640 1 -- 172.21.15.116:0/1551021481 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562a683f2470 con 0x562a68486af0 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.811+0000 7f71a9ffb640 1 -- 172.21.15.116:0/1551021481 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f71bc00f040 con 0x562a68486af0 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.811+0000 7f71cb379740 10 client.25696 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.812+0000 7f71cb379740 10 client.25696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.812+0000 7f71cb379740 20 client.25696 populate_metadata read hostname 'smithi116' 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.812+0000 7f71cb379740 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562a68501d60 0x562a68504180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.812+0000 7f71cb379740 4 client.25696 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.812+0000 7f71cb379740 1 -- 172.21.15.116:0/1551021481 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a68370770 con 0x562a68501d60 2024-03-19T01:19:48.816 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.812+0000 7f71c644e640 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562a68501d60 0x562a68504180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:19:48.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.812+0000 7f71a9ffb640 10 client.25696 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:48.817 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.812+0000 7f71c644e640 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562a68501d60 0x562a68504180 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba4c630970c54988 server_cookie=91cdfe6c0ebf3052 in_seq=0 out_seq=0 2024-03-19T01:19:48.868 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.867+0000 7f71a9ffb640 1 -- 172.21.15.116:0/1551021481 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562a68370770 con 0x562a68501d60 2024-03-19T01:19:48.868 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.867+0000 7f71a9ffb640 10 client.25696 handle_command_reply: tid=0 2024-03-19T01:19:48.868 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.867+0000 7f71cb379740 1 client.25696 shutdown 2024-03-19T01:19:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.870+0000 7f71cb379740 1 -- 172.21.15.116:0/1551021481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 msgr2=0x562a68493f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:48.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.870+0000 7f71cb379740 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a68493f00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f71bc000c00 tx=0x7f71bc005d30 comp rx=0 tx=0).stop 2024-03-19T01:19:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.871+0000 7f71cb379740 1 -- 172.21.15.116:0/1551021481 shutdown_connections 2024-03-19T01:19:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.871+0000 7f71cb379740 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562a6848bc60 0x562a68494440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.871+0000 7f71cb379740 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562a68501d60 0x562a68504180 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.871+0000 7f71cb379740 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562a68486af0 0x562a68493f00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.871+0000 7f71cb379740 1 --2- 172.21.15.116:0/1551021481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562a68486100 0x562a68497460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.871+0000 7f71cb379740 1 -- 172.21.15.116:0/1551021481 >> 172.21.15.116:0/1551021481 conn(0x562a68481670 msgr2=0x562a68480220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:48.876 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.871+0000 7f71a9ffb640 0 client.25696 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:19:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.874+0000 7f71cb379740 1 -- 172.21.15.116:0/1551021481 shutdown_connections 2024-03-19T01:19:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.877+0000 7f71cb379740 1 -- 172.21.15.116:0/1551021481 wait complete. 2024-03-19T01:19:48.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.878+0000 7f71cb379740 20 client.25696 trim_cache size 0 max 16384 2024-03-19T01:19:48.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:19:48.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:19:48.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:19:48.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:19:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.882+0000 7f71c86b2640 1 -- 172.21.15.116:0/4217923615 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f71ac0a2920 msgr2=0x7f71ac0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.882+0000 7f71c86b2640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f71ac0a2920 0x7f71ac0a4de0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f71b8005fd0 tx=0x7f71b8009f10 comp rx=0 tx=0).stop 2024-03-19T01:19:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.882+0000 7f71c86b2640 1 -- 172.21.15.116:0/4217923615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0109930 msgr2=0x7f71c01a9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:19:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.882+0000 7f71c86b2640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0109930 0x7f71c01a9120 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f71b002c240 tx=0x7f71b002c650 comp rx=0 tx=0).stop 2024-03-19T01:19:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.882+0000 7f71c86b2640 1 -- 172.21.15.116:0/4217923615 shutdown_connections 2024-03-19T01:19:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.882+0000 7f71c86b2640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f71c010ebe0 0x7f71c01adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.882+0000 7f71c86b2640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f71ac0a2920 0x7f71ac0a4de0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.882+0000 7f71c86b2640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71c010c260 0x7f71c01ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.882+0000 7f71c86b2640 1 --2- 172.21.15.116:0/4217923615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f71c0109930 0x7f71c01a9120 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:19:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.882+0000 7f71c86b2640 1 -- 172.21.15.116:0/4217923615 >> 172.21.15.116:0/4217923615 conn(0x7f71c006ea20 msgr2=0x7f71c0108c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:19:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.882+0000 7f71c86b2640 1 -- 172.21.15.116:0/4217923615 shutdown_connections 2024-03-19T01:19:48.885 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:19:48.883+0000 7f71c86b2640 1 -- 172.21.15.116:0/4217923615 wait complete. 2024-03-19T01:19:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:49 smithi162 ceph-mon[67178]: pgmap v568: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.1 MiB/s wr, 697 op/s 2024-03-19T01:19:49.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:49 smithi116 ceph-mon[62701]: pgmap v568: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.1 MiB/s wr, 697 op/s 2024-03-19T01:19:49.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:49 smithi156 ceph-mon[66799]: pgmap v568: 97 pgs: 97 active+clean; 1.6 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 8.1 MiB/s wr, 697 op/s 2024-03-19T01:19:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 425.0 s, 2114.8 tps, lat 47.666 ms stddev 54.151 2024-03-19T01:19:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:51 smithi162 ceph-mon[67178]: pgmap v569: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 674 op/s 2024-03-19T01:19:51.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:51 smithi116 ceph-mon[62701]: pgmap v569: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 674 op/s 2024-03-19T01:19:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:51 smithi156 ceph-mon[66799]: pgmap v569: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 674 op/s 2024-03-19T01:19:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:53 smithi162 ceph-mon[67178]: pgmap v570: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 626 op/s 2024-03-19T01:19:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:53 smithi116 ceph-mon[62701]: pgmap v570: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 626 op/s 2024-03-19T01:19:53.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:53 smithi156 ceph-mon[66799]: pgmap v570: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.8 MiB/s wr, 626 op/s 2024-03-19T01:19:55.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:55 smithi162 ceph-mon[67178]: pgmap v571: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.5 MiB/s wr, 774 op/s 2024-03-19T01:19:55.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:55 smithi116 ceph-mon[62701]: pgmap v571: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.5 MiB/s wr, 774 op/s 2024-03-19T01:19:55.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:55 smithi156 ceph-mon[66799]: pgmap v571: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.5 MiB/s wr, 774 op/s 2024-03-19T01:19:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 430.0 s, 1913.2 tps, lat 52.025 ms stddev 59.457 2024-03-19T01:19:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:57 smithi162 ceph-mon[67178]: pgmap v572: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.7 MiB/s wr, 650 op/s 2024-03-19T01:19:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:57 smithi116 ceph-mon[62701]: pgmap v572: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.7 MiB/s wr, 650 op/s 2024-03-19T01:19:57.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:57 smithi156 ceph-mon[66799]: pgmap v572: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.7 MiB/s wr, 650 op/s 2024-03-19T01:19:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:19:59 smithi162 ceph-mon[67178]: pgmap v573: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.8 MiB/s wr, 680 op/s 2024-03-19T01:19:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:19:59 smithi116 ceph-mon[62701]: pgmap v573: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.8 MiB/s wr, 680 op/s 2024-03-19T01:19:59.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:19:59 smithi156 ceph-mon[66799]: pgmap v573: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.8 MiB/s wr, 680 op/s 2024-03-19T01:20:00.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:00 smithi162 ceph-mon[67178]: overall HEALTH_OK 2024-03-19T01:20:00.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:00 smithi116 ceph-mon[62701]: overall HEALTH_OK 2024-03-19T01:20:00.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:00 smithi156 ceph-mon[66799]: overall HEALTH_OK 2024-03-19T01:20:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 435.0 s, 1838.8 tps, lat 54.203 ms stddev 66.843 2024-03-19T01:20:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:01 smithi162 ceph-mon[67178]: pgmap v574: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 709 op/s 2024-03-19T01:20:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:01 smithi116 ceph-mon[62701]: pgmap v574: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 709 op/s 2024-03-19T01:20:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:01 smithi156 ceph-mon[66799]: pgmap v574: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.1 MiB/s wr, 709 op/s 2024-03-19T01:20:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.029+0000 7f5a59590640 1 -- 172.21.15.116:0/1813343647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a54101970 msgr2=0x7f5a54101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.029+0000 7f5a59590640 1 --2- 172.21.15.116:0/1813343647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a54101970 0x7f5a54101df0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5a48009900 tx=0x7f5a4802bc30 comp rx=0 tx=0).stop 2024-03-19T01:20:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.029+0000 7f5a59590640 1 -- 172.21.15.116:0/1813343647 shutdown_connections 2024-03-19T01:20:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.029+0000 7f5a59590640 1 --2- 172.21.15.116:0/1813343647 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a54102330 0x7f5a54108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.029+0000 7f5a59590640 1 --2- 172.21.15.116:0/1813343647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a54101970 0x7f5a54101df0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.029+0000 7f5a59590640 1 --2- 172.21.15.116:0/1813343647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a54100780 0x7f5a54100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.029+0000 7f5a59590640 1 -- 172.21.15.116:0/1813343647 >> 172.21.15.116:0/1813343647 conn(0x7f5a540fbcf0 msgr2=0x7f5a540fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.029+0000 7f5a59590640 1 -- 172.21.15.116:0/1813343647 shutdown_connections 2024-03-19T01:20:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.029+0000 7f5a59590640 1 -- 172.21.15.116:0/1813343647 wait complete. 2024-03-19T01:20:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.030+0000 7f5a59590640 1 Processor -- start 2024-03-19T01:20:02.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.030+0000 7f5a59590640 1 -- start start 2024-03-19T01:20:02.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.030+0000 7f5a59590640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a54100780 0x7f5a5406a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.030+0000 7f5a59590640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a54101970 0x7f5a5419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.030+0000 7f5a59590640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a54102330 0x7f5a5419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.030+0000 7f5a59590640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5410c450 con 0x7f5a54100780 2024-03-19T01:20:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.031+0000 7f5a59590640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5410c2d0 con 0x7f5a54102330 2024-03-19T01:20:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.031+0000 7f5a59590640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5a5410c5d0 con 0x7f5a54101970 2024-03-19T01:20:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.031+0000 7f5a537fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a54102330 0x7f5a5419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.031+0000 7f5a537fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a54102330 0x7f5a5419ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:57298/0 (socket says 172.21.15.116:57298) 2024-03-19T01:20:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.031+0000 7f5a537fe640 1 -- 172.21.15.116:0/805168149 learned_addr learned my addr 172.21.15.116:0/805168149 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.031+0000 7f5a52ffd640 1 --2- 172.21.15.116:0/805168149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a54100780 0x7f5a5406a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.031+0000 7f5a537fe640 1 -- 172.21.15.116:0/805168149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a54101970 msgr2=0x7f5a5419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.031+0000 7f5a537fe640 1 --2- 172.21.15.116:0/805168149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a54101970 0x7f5a5419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.031+0000 7f5a537fe640 1 -- 172.21.15.116:0/805168149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a54100780 msgr2=0x7f5a5406a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:02.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.031+0000 7f5a537fe640 1 --2- 172.21.15.116:0/805168149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a54100780 0x7f5a5406a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.031+0000 7f5a537fe640 1 -- 172.21.15.116:0/805168149 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a541966f0 con 0x7f5a54102330 2024-03-19T01:20:02.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.032+0000 7f5a537fe640 1 --2- 172.21.15.116:0/805168149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a54102330 0x7f5a5419ee40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c00eb10 tx=0x7f5a4c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:02.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.032+0000 7f5a3bfff640 1 -- 172.21.15.116:0/805168149 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c004b20 con 0x7f5a54102330 2024-03-19T01:20:02.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.032+0000 7f5a59590640 1 -- 172.21.15.116:0/805168149 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a54196980 con 0x7f5a54102330 2024-03-19T01:20:02.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.032+0000 7f5a3bfff640 1 -- 172.21.15.116:0/805168149 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a4c004cc0 con 0x7f5a54102330 2024-03-19T01:20:02.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.032+0000 7f5a3bfff640 1 -- 172.21.15.116:0/805168149 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c010760 con 0x7f5a54102330 2024-03-19T01:20:02.034 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.032+0000 7f5a59590640 1 -- 172.21.15.116:0/805168149 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a54196f20 con 0x7f5a54102330 2024-03-19T01:20:02.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.036+0000 7f5a3bfff640 1 -- 172.21.15.116:0/805168149 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f5a4c010900 con 0x7f5a54102330 2024-03-19T01:20:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.036+0000 7f5a5c17f740 1 Processor -- start 2024-03-19T01:20:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.036+0000 7f5a5c17f740 1 -- start start 2024-03-19T01:20:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.036+0000 7f5a3bfff640 1 --2- 172.21.15.116:0/805168149 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5a3c0a2920 0x7f5a3c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.036+0000 7f5a5c17f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298a100 0x558c1298a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.037+0000 7f5a5c17f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298aaf0 0x558c1298f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.037+0000 7f5a5c17f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298fc60 0x558c129920f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.037 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.037+0000 7f5a527fc640 1 --2- 172.21.15.116:0/805168149 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5a3c0a2920 0x7f5a3c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.038 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.037+0000 7f5a3bfff640 1 -- 172.21.15.116:0/805168149 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f5a4c0cf280 con 0x7f5a54102330 2024-03-19T01:20:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.037+0000 7f5a52ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298a100 0x558c1298a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.037+0000 7f5a537fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298fc60 0x558c129920f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.037+0000 7f5a537fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298fc60 0x558c129920f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:57302/0 (socket says 172.21.15.116:57302) 2024-03-19T01:20:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.037+0000 7f5a537fe640 1 -- 172.21.15.116:0/2558600197 learned_addr learned my addr 172.21.15.116:0/2558600197 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.037+0000 7f5a5c17f740 1 -- 172.21.15.116:0/2558600197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558c128eebe0 con 0x558c1298aaf0 2024-03-19T01:20:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.037+0000 7f5a5c17f740 1 -- 172.21.15.116:0/2558600197 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558c12897960 con 0x558c1298fc60 2024-03-19T01:20:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.037+0000 7f5a5c17f740 1 -- 172.21.15.116:0/2558600197 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558c128eac80 con 0x558c1298a100 2024-03-19T01:20:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.037+0000 7f5a527fc640 1 --2- 172.21.15.116:0/2558600197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298aaf0 0x558c1298f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.039 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.038+0000 7f5a537fe640 1 -- 172.21.15.116:0/2558600197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298a100 msgr2=0x558c1298a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.038+0000 7f5a537fe640 1 --2- 172.21.15.116:0/2558600197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298a100 0x558c1298a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.038+0000 7f5a537fe640 1 -- 172.21.15.116:0/2558600197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298aaf0 msgr2=0x558c1298f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.038+0000 7f5a537fe640 1 --2- 172.21.15.116:0/2558600197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298aaf0 0x558c1298f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.038+0000 7f5a537fe640 1 -- 172.21.15.116:0/2558600197 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c1288ed20 con 0x558c1298fc60 2024-03-19T01:20:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.038+0000 7f5a537fe640 1 --2- 172.21.15.116:0/2558600197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298fc60 0x558c129920f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c000f80 tx=0x7f5a4c097f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a2a01456ecc4143 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.038+0000 7f5a527fc640 1 --2- 172.21.15.116:0/805168149 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5a3c0a2920 0x7f5a3c0a4de0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5a480096c0 tx=0x7f5a4802cb20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.038+0000 7f5a397fa640 1 -- 172.21.15.116:0/2558600197 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c0b3020 con 0x558c1298fc60 2024-03-19T01:20:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.038+0000 7f5a397fa640 1 -- 172.21.15.116:0/2558600197 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a4c09c2a0 con 0x558c1298fc60 2024-03-19T01:20:02.040 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.038+0000 7f5a397fa640 1 -- 172.21.15.116:0/2558600197 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c0a3eb0 con 0x558c1298fc60 2024-03-19T01:20:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.039+0000 7f5a5c17f740 1 -- 172.21.15.116:0/2558600197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298fc60 msgr2=0x558c129920f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:02.041 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.039+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/2558600197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298fc60 0x558c129920f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c000f80 tx=0x7f5a4c097f10 comp rx=0 tx=0).stop 2024-03-19T01:20:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.039+0000 7f5a5c17f740 1 -- 172.21.15.116:0/2558600197 shutdown_connections 2024-03-19T01:20:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.039+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/2558600197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298fc60 0x558c129920f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.039+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/2558600197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298aaf0 0x558c1298f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.039+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/2558600197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298a100 0x558c1298a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.039+0000 7f5a5c17f740 1 -- 172.21.15.116:0/2558600197 >> 172.21.15.116:0/2558600197 conn(0x558c12985670 msgr2=0x558c12985a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.039+0000 7f5a5c17f740 1 -- 172.21.15.116:0/2558600197 shutdown_connections 2024-03-19T01:20:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.039+0000 7f5a5c17f740 1 -- 172.21.15.116:0/2558600197 wait complete. 2024-03-19T01:20:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.040+0000 7f5a5c17f740 1 -- start start 2024-03-19T01:20:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.040+0000 7f5a5c17f740 1 Processor -- start 2024-03-19T01:20:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.040+0000 7f5a5c17f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298a100 0x558c12993ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.040+0000 7f5a5c17f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298aaf0 0x558c12994530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.040+0000 7f5a5c17f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298fc60 0x558c1299b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.042 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.040+0000 7f5a5c17f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x558c12992bb0 con 0x558c1298a100 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.040+0000 7f5a5c17f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558c128aadb0 con 0x558c1298fc60 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.040+0000 7f5a5c17f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x558c12992d30 con 0x558c1298aaf0 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.040+0000 7f5a5c17f740 10 client.0 _ll_register_callbacks cb 0x558c128afe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.040+0000 7f5a527fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298aaf0 0x558c12994530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.040+0000 7f5a527fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298aaf0 0x558c12994530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47784/0 (socket says 172.21.15.116:47784) 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.040+0000 7f5a527fc640 1 -- 172.21.15.116:0/1907782942 learned_addr learned my addr 172.21.15.116:0/1907782942 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.040+0000 7f5a537fe640 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298fc60 0x558c1299b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.041+0000 7f5a52ffd640 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298a100 0x558c12993ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.041+0000 7f5a527fc640 1 -- 172.21.15.116:0/1907782942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298fc60 msgr2=0x558c1299b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.041+0000 7f5a527fc640 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298fc60 0x558c1299b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.041+0000 7f5a527fc640 1 -- 172.21.15.116:0/1907782942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298a100 msgr2=0x558c12993ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.041+0000 7f5a527fc640 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298a100 0x558c12993ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.041+0000 7f5a527fc640 1 -- 172.21.15.116:0/1907782942 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c128a9c60 con 0x558c1298aaf0 2024-03-19T01:20:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.041+0000 7f5a52ffd640 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298a100 0x558c12993ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:20:02.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.041+0000 7f5a527fc640 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298aaf0 0x558c12994530 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5a48009230 tx=0x7f5a48000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cbfa2f27fc7e2cc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:02.044 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.041+0000 7f5a36ff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:20:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.041+0000 7f5a36ff5640 1 -- 172.21.15.116:0/1907782942 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a48048070 con 0x558c1298aaf0 2024-03-19T01:20:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.042+0000 7f5a36ff5640 1 -- 172.21.15.116:0/1907782942 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f5a48004ad0 con 0x558c1298aaf0 2024-03-19T01:20:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.042+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1907782942 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a4c09c2a0 con 0x558c1298aaf0 2024-03-19T01:20:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.042+0000 7f5a36ff5640 1 -- 172.21.15.116:0/1907782942 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f5a48004cd0 con 0x558c1298aaf0 2024-03-19T01:20:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.042+0000 7f5a5c17f740 10 client.25705 fetch_fsmap learned FSMap version 45 2024-03-19T01:20:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.042+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1907782942 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x558c12603210 con 0x558c1298aaf0 2024-03-19T01:20:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.042+0000 7f5a36ff5640 1 -- 172.21.15.116:0/1907782942 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a48053070 con 0x558c1298aaf0 2024-03-19T01:20:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.042+0000 7f5a36ff5640 1 -- 172.21.15.116:0/1907782942 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f5a480424d0 con 0x558c1298aaf0 2024-03-19T01:20:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.042+0000 7f5a5c17f740 10 client.25705 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:20:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.042+0000 7f5a5c17f740 10 client.25705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:20:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.042+0000 7f5a5c17f740 20 client.25705 populate_metadata read hostname 'smithi116' 2024-03-19T01:20:02.046 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.042+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558c1299d040 0x558c1299f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.043+0000 7f5a5c17f740 4 client.25705 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.043+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1907782942 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5a4c09c2a0 con 0x558c1299d040 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.043+0000 7f5a52ffd640 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558c1299d040 0x558c1299f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.044+0000 7f5a36ff5640 10 client.25705 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.044+0000 7f5a52ffd640 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558c1299d040 0x558c1299f4e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1239c07cd25a325 server_cookie=de6fceb7031cda8d in_seq=0 out_seq=0 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a36ff5640 1 -- 172.21.15.116:0/1907782942 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f5a4c09c2a0 con 0x558c1299d040 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a36ff5640 10 client.25705 handle_command_reply: tid=0 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a5c17f740 1 client.25705 shutdown 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1907782942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298aaf0 msgr2=0x558c12994530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298aaf0 0x558c12994530 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5a48009230 tx=0x7f5a48000bc0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1907782942 shutdown_connections 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298fc60 0x558c1299b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558c1299d040 0x558c1299f4e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.047 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298aaf0 0x558c12994530 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.048 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a36ff5640 0 client.25705 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:02.048 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/1907782942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298a100 0x558c12993ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.048 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1907782942 >> 172.21.15.116:0/1907782942 conn(0x558c12985670 msgr2=0x558c129868c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:02.048 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1907782942 shutdown_connections 2024-03-19T01:20:02.048 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1907782942 wait complete. 2024-03-19T01:20:02.048 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.045+0000 7f5a5c17f740 20 client.25705 trim_cache size 0 max 16384 2024-03-19T01:20:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.065+0000 7f5a5c17f740 1 Processor -- start 2024-03-19T01:20:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.065+0000 7f5a5c17f740 1 -- start start 2024-03-19T01:20:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a5c17f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298a100 0x558c12988fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a5c17f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298aaf0 0x558c1299b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a5c17f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298fc60 0x558c129a3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a5c17f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5a54072830 con 0x558c1298a100 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a5c17f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5a541a4390 con 0x558c1298aaf0 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a5c17f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5a54072f00 con 0x558c1298fc60 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a527fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298aaf0 0x558c1299b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a52ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298a100 0x558c12988fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a537fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298fc60 0x558c129a3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a52ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298a100 0x558c12988fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38344/0 (socket says 172.21.15.116:38344) 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a537fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298fc60 0x558c129a3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47792/0 (socket says 172.21.15.116:47792) 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a527fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298aaf0 0x558c1299b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:57314/0 (socket says 172.21.15.116:57314) 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a537fe640 1 -- 172.21.15.116:0/4272684747 learned_addr learned my addr 172.21.15.116:0/4272684747 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a537fe640 1 -- 172.21.15.116:0/4272684747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298aaf0 msgr2=0x558c1299b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a537fe640 1 --2- 172.21.15.116:0/4272684747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298aaf0 0x558c1299b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a537fe640 1 -- 172.21.15.116:0/4272684747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298a100 msgr2=0x558c12988fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a537fe640 1 --2- 172.21.15.116:0/4272684747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298a100 0x558c12988fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.066+0000 7f5a537fe640 1 -- 172.21.15.116:0/4272684747 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c126cf620 con 0x558c1298fc60 2024-03-19T01:20:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.067+0000 7f5a537fe640 1 --2- 172.21.15.116:0/4272684747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298fc60 0x558c129a3da0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c09bb30 tx=0x7f5a4c00cee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd7a162cdea6d344 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.067+0000 7f5a38ff9640 1 -- 172.21.15.116:0/4272684747 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c0b3040 con 0x558c1298fc60 2024-03-19T01:20:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.067+0000 7f5a38ff9640 1 -- 172.21.15.116:0/4272684747 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a4c09fcf0 con 0x558c1298fc60 2024-03-19T01:20:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.067+0000 7f5a38ff9640 1 -- 172.21.15.116:0/4272684747 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c098210 con 0x558c1298fc60 2024-03-19T01:20:02.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 -- 172.21.15.116:0/4272684747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298fc60 msgr2=0x558c129a3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/4272684747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298fc60 0x558c129a3da0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c09bb30 tx=0x7f5a4c00cee0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 -- 172.21.15.116:0/4272684747 shutdown_connections 2024-03-19T01:20:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/4272684747 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298fc60 0x558c129a3da0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/4272684747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298aaf0 0x558c1299b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/4272684747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298a100 0x558c12988fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 -- 172.21.15.116:0/4272684747 >> 172.21.15.116:0/4272684747 conn(0x558c12985670 msgr2=0x558c129888d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 -- 172.21.15.116:0/4272684747 shutdown_connections 2024-03-19T01:20:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 -- 172.21.15.116:0/4272684747 wait complete. 2024-03-19T01:20:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 -- start start 2024-03-19T01:20:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 Processor -- start 2024-03-19T01:20:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298a100 0x558c12997520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298aaf0 0x558c12997a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.068+0000 7f5a5c17f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298fc60 0x558c1299aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a5c17f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5a541a3cc0 con 0x558c1298fc60 2024-03-19T01:20:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a5c17f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5a540735d0 con 0x558c1298aaf0 2024-03-19T01:20:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a5c17f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5a54070810 con 0x558c1298a100 2024-03-19T01:20:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a5c17f740 10 client.0 _ll_register_callbacks cb 0x7f5a48005490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:20:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298fc60 0x558c1299aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a52ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298a100 0x558c12997520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298fc60 0x558c1299aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38354/0 (socket says 172.21.15.116:38354) 2024-03-19T01:20:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a537fe640 1 -- 172.21.15.116:0/1300305163 learned_addr learned my addr 172.21.15.116:0/1300305163 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a527fc640 1 --2- 172.21.15.116:0/1300305163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298aaf0 0x558c12997a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a537fe640 1 -- 172.21.15.116:0/1300305163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298a100 msgr2=0x558c12997520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a537fe640 1 --2- 172.21.15.116:0/1300305163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298a100 0x558c12997520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a537fe640 1 -- 172.21.15.116:0/1300305163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298aaf0 msgr2=0x558c12997a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a537fe640 1 --2- 172.21.15.116:0/1300305163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298aaf0 0x558c12997a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a537fe640 1 -- 172.21.15.116:0/1300305163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c1299b070 con 0x558c1298fc60 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a527fc640 1 --2- 172.21.15.116:0/1300305163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298aaf0 0x558c12997a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a537fe640 1 --2- 172.21.15.116:0/1300305163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298fc60 0x558c1299aa80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c000f80 tx=0x7f5a4c005b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6871a0d65627b33c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a36ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a36ff5640 1 -- 172.21.15.116:0/1300305163 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4c0a4340 con 0x558c1298fc60 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a36ff5640 1 -- 172.21.15.116:0/1300305163 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f5a4c0a40c0 con 0x558c1298fc60 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1300305163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a4c09fcf0 con 0x558c1298fc60 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a36ff5640 1 -- 172.21.15.116:0/1300305163 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f5a4c0ccd80 con 0x558c1298fc60 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a5c17f740 10 client.15888 fetch_fsmap learned FSMap version 45 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.069+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1300305163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x558c12a063b0 con 0x558c1298fc60 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.070+0000 7f5a36ff5640 1 -- 172.21.15.116:0/1300305163 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a4c09fcf0 con 0x558c1298fc60 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.070+0000 7f5a36ff5640 1 -- 172.21.15.116:0/1300305163 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f5a4c0a0c10 con 0x558c1298fc60 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.070+0000 7f5a5c17f740 10 client.15888 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:20:02.075 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.070+0000 7f5a5c17f740 10 client.15888 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:20:02.076 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.070+0000 7f5a5c17f740 20 client.15888 populate_metadata read hostname 'smithi116' 2024-03-19T01:20:02.076 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.070+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/1300305163 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558c12a067e0 0x558c12a08bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:02.076 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.070+0000 7f5a5c17f740 4 client.15888 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:20:02.076 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.070+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1300305163 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558c128fc120 con 0x558c12a067e0 2024-03-19T01:20:02.076 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.070+0000 7f5a52ffd640 1 --2- 172.21.15.116:0/1300305163 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558c12a067e0 0x558c12a08bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:02.076 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.071+0000 7f5a52ffd640 1 --2- 172.21.15.116:0/1300305163 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558c12a067e0 0x558c12a08bd0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f588573dbe15156 server_cookie=dc23989c5965a509 in_seq=0 out_seq=0 2024-03-19T01:20:02.076 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.072+0000 7f5a36ff5640 10 client.15888 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.079+0000 7f5a36ff5640 1 -- 172.21.15.116:0/1300305163 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558c128fc120 con 0x558c12a067e0 2024-03-19T01:20:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.079+0000 7f5a36ff5640 10 client.15888 handle_command_reply: tid=0 2024-03-19T01:20:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.079+0000 7f5a5c17f740 1 client.15888 shutdown 2024-03-19T01:20:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.079+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1300305163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298fc60 msgr2=0x558c1299aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.079+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/1300305163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298fc60 0x558c1299aa80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c000f80 tx=0x7f5a4c005b40 comp rx=0 tx=0).stop 2024-03-19T01:20:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.079+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1300305163 shutdown_connections 2024-03-19T01:20:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.079+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/1300305163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x558c1298fc60 0x558c1299aa80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.079+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/1300305163 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558c1298aaf0 0x558c12997a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.080+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/1300305163 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x558c12a067e0 0x558c12a08bd0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.080+0000 7f5a5c17f740 1 --2- 172.21.15.116:0/1300305163 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x558c1298a100 0x558c12997520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.080+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1300305163 >> 172.21.15.116:0/1300305163 conn(0x558c12985670 msgr2=0x558c12989160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.080+0000 7f5a36ff5640 0 client.15888 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.080+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1300305163 shutdown_connections 2024-03-19T01:20:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.080+0000 7f5a5c17f740 1 -- 172.21.15.116:0/1300305163 wait complete. 2024-03-19T01:20:02.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.080+0000 7f5a5c17f740 20 client.15888 trim_cache size 0 max 16384 2024-03-19T01:20:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:20:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:20:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:20:02.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:20:02.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.080+0000 7f5a59590640 1 -- 172.21.15.116:0/805168149 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5a3c0a2920 msgr2=0x7f5a3c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:02.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.080+0000 7f5a59590640 1 --2- 172.21.15.116:0/805168149 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5a3c0a2920 0x7f5a3c0a4de0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5a480096c0 tx=0x7f5a4802cb20 comp rx=0 tx=0).stop 2024-03-19T01:20:02.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.080+0000 7f5a59590640 1 -- 172.21.15.116:0/805168149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a54102330 msgr2=0x7f5a5419ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:02.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.080+0000 7f5a59590640 1 --2- 172.21.15.116:0/805168149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a54102330 0x7f5a5419ee40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c00eb10 tx=0x7f5a4c00c2d0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.081+0000 7f5a59590640 1 -- 172.21.15.116:0/805168149 shutdown_connections 2024-03-19T01:20:02.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.081+0000 7f5a59590640 1 --2- 172.21.15.116:0/805168149 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5a3c0a2920 0x7f5a3c0a4de0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.081+0000 7f5a59590640 1 --2- 172.21.15.116:0/805168149 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5a54102330 0x7f5a5419ee40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.081+0000 7f5a59590640 1 --2- 172.21.15.116:0/805168149 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5a54101970 0x7f5a5419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.081+0000 7f5a59590640 1 --2- 172.21.15.116:0/805168149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a54100780 0x7f5a5406a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:02.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.081+0000 7f5a59590640 1 -- 172.21.15.116:0/805168149 >> 172.21.15.116:0/805168149 conn(0x7f5a540fbcf0 msgr2=0x7f5a54104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:02.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.081+0000 7f5a59590640 1 -- 172.21.15.116:0/805168149 shutdown_connections 2024-03-19T01:20:02.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:02.081+0000 7f5a59590640 1 -- 172.21.15.116:0/805168149 wait complete. 2024-03-19T01:20:03.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:03 smithi162 ceph-mon[67178]: pgmap v575: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.8 MiB/s wr, 615 op/s 2024-03-19T01:20:03.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:03 smithi116 ceph-mon[62701]: pgmap v575: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.8 MiB/s wr, 615 op/s 2024-03-19T01:20:03.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:03 smithi156 ceph-mon[66799]: pgmap v575: 97 pgs: 97 active+clean; 1.6 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.8 MiB/s wr, 615 op/s 2024-03-19T01:20:04.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:04 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:20:04.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:20:04.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:04 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:20:05.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi162 to 174.1M 2024-03-19T01:20:05.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: Unable to set osd_memory_target on smithi162 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-19T01:20:05.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: Adjusting osd_memory_target on smithi156 to 174.1M 2024-03-19T01:20:05.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: Unable to set osd_memory_target on smithi156 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-19T01:20:05.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:20:05.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:20:05.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:20:05.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:05 smithi162 ceph-mon[67178]: pgmap v576: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.0 MiB/s wr, 711 op/s 2024-03-19T01:20:05.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi162 to 174.1M 2024-03-19T01:20:05.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: Unable to set osd_memory_target on smithi162 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-19T01:20:05.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: Adjusting osd_memory_target on smithi156 to 174.1M 2024-03-19T01:20:05.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: Unable to set osd_memory_target on smithi156 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-19T01:20:05.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:20:05.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:20:05.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:20:05.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:05 smithi116 ceph-mon[62701]: pgmap v576: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.0 MiB/s wr, 711 op/s 2024-03-19T01:20:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi162 to 174.1M 2024-03-19T01:20:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: Unable to set osd_memory_target on smithi162 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-19T01:20:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-19T01:20:05.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: Adjusting osd_memory_target on smithi156 to 174.1M 2024-03-19T01:20:05.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: Unable to set osd_memory_target on smithi156 to 182636339: error parsing value: Value '182636339' is below minimum 939524096 2024-03-19T01:20:05.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:20:05.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:20:05.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:20:05.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:05 smithi156 ceph-mon[66799]: pgmap v576: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.0 MiB/s wr, 711 op/s 2024-03-19T01:20:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 440.0 s, 2566.0 tps, lat 39.330 ms stddev 45.469 2024-03-19T01:20:07.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:07 smithi162 ceph-mon[67178]: pgmap v577: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 659 op/s 2024-03-19T01:20:07.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:07 smithi116 ceph-mon[62701]: pgmap v577: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 659 op/s 2024-03-19T01:20:07.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:07 smithi156 ceph-mon[66799]: pgmap v577: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 659 op/s 2024-03-19T01:20:09.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:09 smithi162 ceph-mon[67178]: pgmap v578: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 702 op/s 2024-03-19T01:20:09.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:09 smithi116 ceph-mon[62701]: pgmap v578: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 702 op/s 2024-03-19T01:20:09.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:09 smithi156 ceph-mon[66799]: pgmap v578: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 702 op/s 2024-03-19T01:20:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 445.0 s, 2113.4 tps, lat 47.108 ms stddev 50.749 2024-03-19T01:20:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:11 smithi162 ceph-mon[67178]: pgmap v579: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.3 MiB/s wr, 726 op/s 2024-03-19T01:20:11.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:11 smithi116 ceph-mon[62701]: pgmap v579: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.3 MiB/s wr, 726 op/s 2024-03-19T01:20:11.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:11 smithi156 ceph-mon[66799]: pgmap v579: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.3 MiB/s wr, 726 op/s 2024-03-19T01:20:13.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:13 smithi162 ceph-mon[67178]: pgmap v580: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.4 MiB/s wr, 572 op/s 2024-03-19T01:20:13.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:13 smithi116 ceph-mon[62701]: pgmap v580: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.4 MiB/s wr, 572 op/s 2024-03-19T01:20:13.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:13 smithi156 ceph-mon[66799]: pgmap v580: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.4 MiB/s wr, 572 op/s 2024-03-19T01:20:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.220+0000 7f37ed95b640 1 -- 172.21.15.116:0/2850231386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8100780 msgr2=0x7f37e8100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.220+0000 7f37ed95b640 1 --2- 172.21.15.116:0/2850231386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8100780 0x7f37e8100ba0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f37dc009900 tx=0x7f37dc02bc30 comp rx=0 tx=0).stop 2024-03-19T01:20:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.220+0000 7f37ed95b640 1 -- 172.21.15.116:0/2850231386 shutdown_connections 2024-03-19T01:20:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.220+0000 7f37ed95b640 1 --2- 172.21.15.116:0/2850231386 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e8102330 0x7f37e8108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.220+0000 7f37ed95b640 1 --2- 172.21.15.116:0/2850231386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37e8101970 0x7f37e8101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.221+0000 7f37ed95b640 1 --2- 172.21.15.116:0/2850231386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8100780 0x7f37e8100ba0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.221+0000 7f37ed95b640 1 -- 172.21.15.116:0/2850231386 >> 172.21.15.116:0/2850231386 conn(0x7f37e80fbcf0 msgr2=0x7f37e80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.221+0000 7f37ed95b640 1 -- 172.21.15.116:0/2850231386 shutdown_connections 2024-03-19T01:20:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.221+0000 7f37ed95b640 1 -- 172.21.15.116:0/2850231386 wait complete. 2024-03-19T01:20:15.224 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.221+0000 7f37ed95b640 1 Processor -- start 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37ed95b640 1 -- start start 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37ed95b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37e8100780 0x7f37e81958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37ed95b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e8101970 0x7f37e8195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37ed95b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8102330 0x7f37e819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37e8100780 0x7f37e81958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37e8100780 0x7f37e81958b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:32926/0 (socket says 172.21.15.116:32926) 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37e67fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e8101970 0x7f37e8195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37e67fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e8101970 0x7f37e8195df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46272/0 (socket says 172.21.15.116:46272) 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37e77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8102330 0x7f37e819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37e6ffd640 1 -- 172.21.15.116:0/2561675047 learned_addr learned my addr 172.21.15.116:0/2561675047 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37ed95b640 1 -- 172.21.15.116:0/2561675047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f37e810b350 con 0x7f37e8102330 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37ed95b640 1 -- 172.21.15.116:0/2561675047 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f37e810b1d0 con 0x7f37e8100780 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37ed95b640 1 -- 172.21.15.116:0/2561675047 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f37e810b4d0 con 0x7f37e8101970 2024-03-19T01:20:15.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37e6ffd640 1 -- 172.21.15.116:0/2561675047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e8101970 msgr2=0x7f37e8195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/2561675047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e8101970 0x7f37e8195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37e6ffd640 1 -- 172.21.15.116:0/2561675047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8102330 msgr2=0x7f37e819ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.222+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/2561675047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8102330 0x7f37e819ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.223+0000 7f37e6ffd640 1 -- 172.21.15.116:0/2561675047 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e819d4d0 con 0x7f37e8100780 2024-03-19T01:20:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.223+0000 7f37e67fc640 1 --2- 172.21.15.116:0/2561675047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e8101970 0x7f37e8195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:20:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.223+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/2561675047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37e8100780 0x7f37e81958b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f37dc000c00 tx=0x7f37dc02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.223+0000 7f37ec959640 1 -- 172.21.15.116:0/2561675047 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc043020 con 0x7f37e8100780 2024-03-19T01:20:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.223+0000 7f37ed95b640 1 -- 172.21.15.116:0/2561675047 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e819d760 con 0x7f37e8100780 2024-03-19T01:20:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.223+0000 7f37ec959640 1 -- 172.21.15.116:0/2561675047 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37dc0054e0 con 0x7f37e8100780 2024-03-19T01:20:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.223+0000 7f37ec959640 1 -- 172.21.15.116:0/2561675047 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc003ce0 con 0x7f37e8100780 2024-03-19T01:20:15.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.223+0000 7f37ed95b640 1 -- 172.21.15.116:0/2561675047 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e819dcd0 con 0x7f37e8100780 2024-03-19T01:20:15.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.226+0000 7f37ec959640 1 -- 172.21.15.116:0/2561675047 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f37dc046020 con 0x7f37e8100780 2024-03-19T01:20:15.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.227+0000 7f37ec959640 1 --2- 172.21.15.116:0/2561675047 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37d00a4ab0 0x7f37d00a6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.227+0000 7f37e67fc640 1 --2- 172.21.15.116:0/2561675047 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37d00a4ab0 0x7f37d00a6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.227+0000 7f37ec959640 1 -- 172.21.15.116:0/2561675047 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f37dc0f0b80 con 0x7f37e8100780 2024-03-19T01:20:15.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.227+0000 7f37e67fc640 1 --2- 172.21.15.116:0/2561675047 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37d00a4ab0 0x7f37d00a6f70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f37e8196dc0 tx=0x7f37d8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:15.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.227+0000 7f37f0625740 1 Processor -- start 2024-03-19T01:20:15.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37f0625740 1 -- start start 2024-03-19T01:20:15.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37f0625740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b59520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.229 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37f0625740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b59af0 0x556a15b5e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37f0625740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b5ec60 0x556a15b610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37f0625740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556a15abdbe0 con 0x556a15b59af0 2024-03-19T01:20:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37f0625740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556a15a66960 con 0x556a15b5ec60 2024-03-19T01:20:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37f0625740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556a15ab9c80 con 0x556a15b59100 2024-03-19T01:20:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37e67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b59af0 0x556a15b5e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b59520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b59520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46284/0 (socket says 172.21.15.116:46284) 2024-03-19T01:20:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37e67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b59af0 0x556a15b5e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36838/0 (socket says 172.21.15.116:36838) 2024-03-19T01:20:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3924982862 learned_addr learned my addr 172.21.15.116:0/3924982862 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37e77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b5ec60 0x556a15b610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3924982862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b5ec60 msgr2=0x556a15b610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/3924982862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b5ec60 0x556a15b610f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.230 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3924982862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b59af0 msgr2=0x556a15b5e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:15.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/3924982862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b59af0 0x556a15b5e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3924982862 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a15a5dd20 con 0x556a15b59100 2024-03-19T01:20:15.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.228+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/3924982862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b59520 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0bd200 tx=0x7f37dc009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33b81744271f3e01 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:15.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.229+0000 7f37cdffb640 1 -- 172.21.15.116:0/3924982862 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0d6070 con 0x556a15b59100 2024-03-19T01:20:15.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.229+0000 7f37cdffb640 1 -- 172.21.15.116:0/3924982862 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37dc0044a0 con 0x556a15b59100 2024-03-19T01:20:15.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.229+0000 7f37cdffb640 1 -- 172.21.15.116:0/3924982862 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0bb5a0 con 0x556a15b59100 2024-03-19T01:20:15.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.230+0000 7f37f0625740 1 -- 172.21.15.116:0/3924982862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 msgr2=0x556a15b59520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:15.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.230+0000 7f37f0625740 1 --2- 172.21.15.116:0/3924982862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b59520 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0bd200 tx=0x7f37dc009260 comp rx=0 tx=0).stop 2024-03-19T01:20:15.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.230+0000 7f37f0625740 1 -- 172.21.15.116:0/3924982862 shutdown_connections 2024-03-19T01:20:15.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.230+0000 7f37f0625740 1 --2- 172.21.15.116:0/3924982862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b5ec60 0x556a15b610f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.230+0000 7f37f0625740 1 --2- 172.21.15.116:0/3924982862 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b59af0 0x556a15b5e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.230+0000 7f37f0625740 1 --2- 172.21.15.116:0/3924982862 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b59520 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.230+0000 7f37f0625740 1 -- 172.21.15.116:0/3924982862 >> 172.21.15.116:0/3924982862 conn(0x556a15b54670 msgr2=0x556a15b54a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:15.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.230+0000 7f37f0625740 1 -- 172.21.15.116:0/3924982862 shutdown_connections 2024-03-19T01:20:15.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.230+0000 7f37f0625740 1 -- 172.21.15.116:0/3924982862 wait complete. 2024-03-19T01:20:15.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.230+0000 7f37f0625740 1 -- start start 2024-03-19T01:20:15.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.231+0000 7f37f0625740 1 Processor -- start 2024-03-19T01:20:15.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.231+0000 7f37f0625740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b67210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.231+0000 7f37f0625740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b59af0 0x556a15b67750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.231+0000 7f37e67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b59af0 0x556a15b67750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.231+0000 7f37e67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b59af0 0x556a15b67750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36852/0 (socket says 172.21.15.116:36852) 2024-03-19T01:20:15.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.231+0000 7f37f0625740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b5ec60 0x556a15b6e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.231+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b67210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.231+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b67210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46290/0 (socket says 172.21.15.116:46290) 2024-03-19T01:20:15.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.232+0000 7f37e67fc640 1 -- 172.21.15.116:0/1540634425 learned_addr learned my addr 172.21.15.116:0/1540634425 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:15.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.232+0000 7f37f0625740 1 -- 172.21.15.116:0/1540634425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556a15b65de0 con 0x556a15b59af0 2024-03-19T01:20:15.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.232+0000 7f37f0625740 1 -- 172.21.15.116:0/1540634425 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556a15a79db0 con 0x556a15b5ec60 2024-03-19T01:20:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.232+0000 7f37f0625740 1 -- 172.21.15.116:0/1540634425 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556a15b65f60 con 0x556a15b59100 2024-03-19T01:20:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.232+0000 7f37f0625740 10 client.0 _ll_register_callbacks cb 0x556a15a7ee70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:20:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.232+0000 7f37e77fe640 1 --2- 172.21.15.116:0/1540634425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b5ec60 0x556a15b6e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.232+0000 7f37e6ffd640 1 -- 172.21.15.116:0/1540634425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b5ec60 msgr2=0x556a15b6e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.232+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/1540634425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b5ec60 0x556a15b6e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.232+0000 7f37e6ffd640 1 -- 172.21.15.116:0/1540634425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b59af0 msgr2=0x556a15b67750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.232+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/1540634425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b59af0 0x556a15b67750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.232+0000 7f37e6ffd640 1 -- 172.21.15.116:0/1540634425 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a15a78c60 con 0x556a15b59100 2024-03-19T01:20:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.233+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/1540634425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b67210 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0bd200 tx=0x7f37dc0bd620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b4dbe78ccf26ea4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.233+0000 7f37c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:20:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.233+0000 7f37f0625740 1 -- 172.21.15.116:0/1540634425 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556a15acb120 con 0x556a15b59100 2024-03-19T01:20:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.233+0000 7f37c6ffd640 1 -- 172.21.15.116:0/1540634425 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0da070 con 0x556a15b59100 2024-03-19T01:20:15.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.233+0000 7f37c6ffd640 1 -- 172.21.15.116:0/1540634425 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f37dc0035f0 con 0x556a15b59100 2024-03-19T01:20:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.233+0000 7f37c6ffd640 1 -- 172.21.15.116:0/1540634425 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f37dc0d6040 con 0x556a15b59100 2024-03-19T01:20:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.233+0000 7f37f0625740 10 client.25723 fetch_fsmap learned FSMap version 45 2024-03-19T01:20:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.233+0000 7f37f0625740 1 -- 172.21.15.116:0/1540634425 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({fsmap=45}) -- 0x556a157d2210 con 0x556a15b59100 2024-03-19T01:20:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.234+0000 7f37c6ffd640 1 -- 172.21.15.116:0/1540634425 <== mon.1 v2:172.21.15.162:3300/0 4 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f37dc0d61e0 con 0x556a15b59100 2024-03-19T01:20:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.234+0000 7f37f0625740 10 client.25723 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:20:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.234+0000 7f37f0625740 10 client.25723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:20:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.234+0000 7f37f0625740 20 client.25723 populate_metadata read hostname 'smithi116' 2024-03-19T01:20:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.234+0000 7f37f0625740 1 --2- 172.21.15.116:0/1540634425 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556a15b72b70 0x556a15b74fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.235+0000 7f37f0625740 4 client.25723 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:20:15.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.235+0000 7f37f0625740 1 -- 172.21.15.116:0/1540634425 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a15acb120 con 0x556a15b72b70 2024-03-19T01:20:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.235+0000 7f37e67fc640 1 --2- 172.21.15.116:0/1540634425 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556a15b72b70 0x556a15b74fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.236+0000 7f37e67fc640 1 --2- 172.21.15.116:0/1540634425 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556a15b72b70 0x556a15b74fa0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2e1d3f624c0db60 server_cookie=9681f984844353ba in_seq=0 out_seq=0 2024-03-19T01:20:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.236+0000 7f37c6ffd640 10 client.25723 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37c6ffd640 1 -- 172.21.15.116:0/1540634425 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556a15acb120 con 0x556a15b72b70 2024-03-19T01:20:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37c6ffd640 10 client.25723 handle_command_reply: tid=0 2024-03-19T01:20:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37f0625740 1 client.25723 shutdown 2024-03-19T01:20:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37f0625740 1 -- 172.21.15.116:0/1540634425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 msgr2=0x556a15b67210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37f0625740 1 --2- 172.21.15.116:0/1540634425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b67210 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0bd200 tx=0x7f37dc0bd620 comp rx=0 tx=0).stop 2024-03-19T01:20:15.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37f0625740 1 -- 172.21.15.116:0/1540634425 shutdown_connections 2024-03-19T01:20:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37f0625740 1 --2- 172.21.15.116:0/1540634425 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b5ec60 0x556a15b6e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37f0625740 1 --2- 172.21.15.116:0/1540634425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b59af0 0x556a15b67750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37f0625740 1 --2- 172.21.15.116:0/1540634425 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556a15b72b70 0x556a15b74fa0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37f0625740 1 --2- 172.21.15.116:0/1540634425 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b67210 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37f0625740 1 -- 172.21.15.116:0/1540634425 >> 172.21.15.116:0/1540634425 conn(0x556a15b54670 msgr2=0x556a15b558c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37c6ffd640 0 client.25723 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37f0625740 1 -- 172.21.15.116:0/1540634425 shutdown_connections 2024-03-19T01:20:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.237+0000 7f37f0625740 1 -- 172.21.15.116:0/1540634425 wait complete. 2024-03-19T01:20:15.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.238+0000 7f37f0625740 20 client.25723 trim_cache size 0 max 16384 2024-03-19T01:20:15.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.257+0000 7f37f0625740 1 Processor -- start 2024-03-19T01:20:15.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.257+0000 7f37f0625740 1 -- start start 2024-03-19T01:20:15.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.257+0000 7f37f0625740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b53890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.257+0000 7f37f0625740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b59af0 0x556a15b676b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.258 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.257+0000 7f37f0625740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b5ec60 0x556a15b57fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.257+0000 7f37f0625740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81a6a40 con 0x556a15b5ec60 2024-03-19T01:20:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.257+0000 7f37f0625740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81a8580 con 0x556a15b59af0 2024-03-19T01:20:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.257+0000 7f37f0625740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81a7110 con 0x556a15b59100 2024-03-19T01:20:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.258+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b53890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.258+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b53890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46292/0 (socket says 172.21.15.116:46292) 2024-03-19T01:20:15.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.258+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3287435692 learned_addr learned my addr 172.21.15.116:0/3287435692 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.258+0000 7f37e67fc640 1 --2- 172.21.15.116:0/3287435692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b59af0 0x556a15b676b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.258+0000 7f37e77fe640 1 --2- 172.21.15.116:0/3287435692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b5ec60 0x556a15b57fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.258+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3287435692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b59af0 msgr2=0x556a15b676b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.259+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/3287435692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b59af0 0x556a15b676b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.259+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3287435692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b5ec60 msgr2=0x556a15b57fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.259+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/3287435692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b5ec60 0x556a15b57fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.259+0000 7f37e6ffd640 1 -- 172.21.15.116:0/3287435692 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a15b4fee0 con 0x556a15b59100 2024-03-19T01:20:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.259+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/3287435692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b53890 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0b9ed0 tx=0x7f37dc0d7000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb524ec9641f43cb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.259+0000 7f37cd7fa640 1 -- 172.21.15.116:0/3287435692 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0cc070 con 0x556a15b59100 2024-03-19T01:20:15.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.259+0000 7f37cd7fa640 1 -- 172.21.15.116:0/3287435692 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37dc0bdc00 con 0x556a15b59100 2024-03-19T01:20:15.261 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.259+0000 7f37cd7fa640 1 -- 172.21.15.116:0/3287435692 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0da070 con 0x556a15b59100 2024-03-19T01:20:15.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.260+0000 7f37f0625740 1 -- 172.21.15.116:0/3287435692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 msgr2=0x556a15b53890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:15.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.260+0000 7f37f0625740 1 --2- 172.21.15.116:0/3287435692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b53890 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0b9ed0 tx=0x7f37dc0d7000 comp rx=0 tx=0).stop 2024-03-19T01:20:15.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.260+0000 7f37f0625740 1 -- 172.21.15.116:0/3287435692 shutdown_connections 2024-03-19T01:20:15.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.260+0000 7f37f0625740 1 --2- 172.21.15.116:0/3287435692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b5ec60 0x556a15b57fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.260+0000 7f37f0625740 1 --2- 172.21.15.116:0/3287435692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b59af0 0x556a15b676b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.260+0000 7f37f0625740 1 --2- 172.21.15.116:0/3287435692 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59100 0x556a15b53890 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.260+0000 7f37f0625740 1 -- 172.21.15.116:0/3287435692 >> 172.21.15.116:0/3287435692 conn(0x556a15b54670 msgr2=0x556a15b578d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:15.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.260+0000 7f37f0625740 1 -- 172.21.15.116:0/3287435692 shutdown_connections 2024-03-19T01:20:15.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.260+0000 7f37f0625740 1 -- 172.21.15.116:0/3287435692 wait complete. 2024-03-19T01:20:15.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.260+0000 7f37f0625740 1 -- start start 2024-03-19T01:20:15.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.261+0000 7f37f0625740 1 Processor -- start 2024-03-19T01:20:15.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.261+0000 7f37f0625740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b59100 0x556a15b6a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.261+0000 7f37f0625740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59af0 0x556a15b6cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.261+0000 7f37f0625740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b5ec60 0x556a15b6d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.261+0000 7f37f0625740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81a7eb0 con 0x556a15b5ec60 2024-03-19T01:20:15.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.261+0000 7f37f0625740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81a77e0 con 0x556a15b59100 2024-03-19T01:20:15.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.261+0000 7f37f0625740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f37e81a4a20 con 0x556a15b59af0 2024-03-19T01:20:15.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.261+0000 7f37f0625740 10 client.0 _ll_register_callbacks cb 0x7f37dc0d4070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:20:15.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.261+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b59100 0x556a15b6a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.261+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b59100 0x556a15b6a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:32948/0 (socket says 172.21.15.116:32948) 2024-03-19T01:20:15.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.261+0000 7f37e6ffd640 1 -- 172.21.15.116:0/2013166011 learned_addr learned my addr 172.21.15.116:0/2013166011 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:15.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.261+0000 7f37e67fc640 1 --2- 172.21.15.116:0/2013166011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59af0 0x556a15b6cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.261+0000 7f37e77fe640 1 --2- 172.21.15.116:0/2013166011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b5ec60 0x556a15b6d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37e6ffd640 1 -- 172.21.15.116:0/2013166011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59af0 msgr2=0x556a15b6cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/2013166011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59af0 0x556a15b6cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37e6ffd640 1 -- 172.21.15.116:0/2013166011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b5ec60 msgr2=0x556a15b6d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/2013166011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b5ec60 0x556a15b6d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37e6ffd640 1 -- 172.21.15.116:0/2013166011 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a1589e620 con 0x556a15b59100 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37e6ffd640 1 --2- 172.21.15.116:0/2013166011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b59100 0x556a15b6a730 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0a6820 tx=0x7f37dc0d61e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cac54eec946f3658 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37c6ffd640 1 -- 172.21.15.116:0/2013166011 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0da070 con 0x556a15b59100 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37c6ffd640 1 -- 172.21.15.116:0/2013166011 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f37dc0c5a20 con 0x556a15b59100 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37f0625740 1 -- 172.21.15.116:0/2013166011 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37dc0bdc00 con 0x556a15b59100 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37c6ffd640 1 -- 172.21.15.116:0/2013166011 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f37dc0d63c0 con 0x556a15b59100 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37f0625740 10 client.25739 fetch_fsmap learned FSMap version 45 2024-03-19T01:20:15.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37f0625740 1 -- 172.21.15.116:0/2013166011 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x556a1589e620 con 0x556a15b59100 2024-03-19T01:20:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37c6ffd640 1 -- 172.21.15.116:0/2013166011 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37dc0bdc00 con 0x556a15b59100 2024-03-19T01:20:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.262+0000 7f37c6ffd640 1 -- 172.21.15.116:0/2013166011 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f37dc0c1170 con 0x556a15b59100 2024-03-19T01:20:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.263+0000 7f37f0625740 10 client.25739 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:20:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.263+0000 7f37f0625740 10 client.25739 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:20:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.263+0000 7f37f0625740 20 client.25739 populate_metadata read hostname 'smithi116' 2024-03-19T01:20:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.263+0000 7f37f0625740 1 --2- 172.21.15.116:0/2013166011 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556a15bdde90 0x556a15be02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.263+0000 7f37f0625740 4 client.25739 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:20:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.263+0000 7f37f0625740 1 -- 172.21.15.116:0/2013166011 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556a15ac5470 con 0x556a15bdde90 2024-03-19T01:20:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.263+0000 7f37e67fc640 1 --2- 172.21.15.116:0/2013166011 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556a15bdde90 0x556a15be02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:15.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.265+0000 7f37e67fc640 1 --2- 172.21.15.116:0/2013166011 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556a15bdde90 0x556a15be02b0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=531d7bb71cd5c3c0 server_cookie=79c5587baf54ccff in_seq=0 out_seq=0 2024-03-19T01:20:15.268 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.265+0000 7f37c6ffd640 10 client.25739 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:15.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.272+0000 7f37c6ffd640 1 -- 172.21.15.116:0/2013166011 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556a15ac5470 con 0x556a15bdde90 2024-03-19T01:20:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.272+0000 7f37c6ffd640 10 client.25739 handle_command_reply: tid=0 2024-03-19T01:20:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.273+0000 7f37f0625740 1 client.25739 shutdown 2024-03-19T01:20:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.273+0000 7f37f0625740 1 -- 172.21.15.116:0/2013166011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b59100 msgr2=0x556a15b6a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.273+0000 7f37f0625740 1 --2- 172.21.15.116:0/2013166011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b59100 0x556a15b6a730 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0a6820 tx=0x7f37dc0d61e0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.273+0000 7f37f0625740 1 -- 172.21.15.116:0/2013166011 shutdown_connections 2024-03-19T01:20:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.273+0000 7f37f0625740 1 --2- 172.21.15.116:0/2013166011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556a15b5ec60 0x556a15b6d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.273+0000 7f37f0625740 1 --2- 172.21.15.116:0/2013166011 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556a15bdde90 0x556a15be02b0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.273+0000 7f37f0625740 1 --2- 172.21.15.116:0/2013166011 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556a15b59af0 0x556a15b6cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.273+0000 7f37f0625740 1 --2- 172.21.15.116:0/2013166011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556a15b59100 0x556a15b6a730 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.273+0000 7f37c6ffd640 0 client.25739 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:15.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.273+0000 7f37f0625740 1 -- 172.21.15.116:0/2013166011 >> 172.21.15.116:0/2013166011 conn(0x556a15b54670 msgr2=0x556a15b674a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:15.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.273+0000 7f37f0625740 1 -- 172.21.15.116:0/2013166011 shutdown_connections 2024-03-19T01:20:15.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.273+0000 7f37f0625740 1 -- 172.21.15.116:0/2013166011 wait complete. 2024-03-19T01:20:15.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.273+0000 7f37f0625740 20 client.25739 trim_cache size 0 max 16384 2024-03-19T01:20:15.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:20:15.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:20:15.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:20:15.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:20:15.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.274+0000 7f37ed95b640 1 -- 172.21.15.116:0/2561675047 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37d00a4ab0 msgr2=0x7f37d00a6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:15.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.274+0000 7f37ed95b640 1 --2- 172.21.15.116:0/2561675047 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37d00a4ab0 0x7f37d00a6f70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f37e8196dc0 tx=0x7f37d8005f90 comp rx=0 tx=0).stop 2024-03-19T01:20:15.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.274+0000 7f37ed95b640 1 -- 172.21.15.116:0/2561675047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37e8100780 msgr2=0x7f37e81958b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:15.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.274+0000 7f37ed95b640 1 --2- 172.21.15.116:0/2561675047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37e8100780 0x7f37e81958b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f37dc000c00 tx=0x7f37dc02c6c0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.274+0000 7f37ed95b640 1 -- 172.21.15.116:0/2561675047 shutdown_connections 2024-03-19T01:20:15.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.274+0000 7f37ed95b640 1 --2- 172.21.15.116:0/2561675047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37e8102330 0x7f37e819ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.274+0000 7f37ed95b640 1 --2- 172.21.15.116:0/2561675047 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f37e8101970 0x7f37e8195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.274+0000 7f37ed95b640 1 --2- 172.21.15.116:0/2561675047 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f37d00a4ab0 0x7f37d00a6f70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.274+0000 7f37ed95b640 1 --2- 172.21.15.116:0/2561675047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37e8100780 0x7f37e81958b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:15.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.274+0000 7f37ed95b640 1 -- 172.21.15.116:0/2561675047 >> 172.21.15.116:0/2561675047 conn(0x7f37e80fbcf0 msgr2=0x7f37e8104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:15.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.275+0000 7f37ed95b640 1 -- 172.21.15.116:0/2561675047 shutdown_connections 2024-03-19T01:20:15.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:15.275+0000 7f37ed95b640 1 -- 172.21.15.116:0/2561675047 wait complete. 2024-03-19T01:20:15.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:15 smithi162 ceph-mon[67178]: pgmap v581: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.0 MiB/s wr, 682 op/s 2024-03-19T01:20:15.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:15 smithi116 ceph-mon[62701]: pgmap v581: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.0 MiB/s wr, 682 op/s 2024-03-19T01:20:15.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:15 smithi156 ceph-mon[66799]: pgmap v581: 97 pgs: 97 active+clean; 1.6 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 7.0 MiB/s wr, 682 op/s 2024-03-19T01:20:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 450.0 s, 2078.0 tps, lat 48.264 ms stddev 51.242 2024-03-19T01:20:17.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:17 smithi162 ceph-mon[67178]: pgmap v582: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 7.1 MiB/s wr, 687 op/s 2024-03-19T01:20:17.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:17 smithi116 ceph-mon[62701]: pgmap v582: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 7.1 MiB/s wr, 687 op/s 2024-03-19T01:20:17.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:17 smithi156 ceph-mon[66799]: pgmap v582: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 7.1 MiB/s wr, 687 op/s 2024-03-19T01:20:19.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:19 smithi162 ceph-mon[67178]: pgmap v583: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.0 MiB/s wr, 641 op/s 2024-03-19T01:20:19.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:19 smithi116 ceph-mon[62701]: pgmap v583: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.0 MiB/s wr, 641 op/s 2024-03-19T01:20:19.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:19 smithi156 ceph-mon[66799]: pgmap v583: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.0 MiB/s wr, 641 op/s 2024-03-19T01:20:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 455.0 s, 2475.8 tps, lat 40.399 ms stddev 46.163 2024-03-19T01:20:21.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:21 smithi162 ceph-mon[67178]: pgmap v584: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.4 MiB/s wr, 671 op/s 2024-03-19T01:20:21.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:21 smithi116 ceph-mon[62701]: pgmap v584: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.4 MiB/s wr, 671 op/s 2024-03-19T01:20:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:21 smithi156 ceph-mon[66799]: pgmap v584: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.4 MiB/s wr, 671 op/s 2024-03-19T01:20:23.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:23 smithi162 ceph-mon[67178]: pgmap v585: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 543 op/s 2024-03-19T01:20:23.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:23 smithi116 ceph-mon[62701]: pgmap v585: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 543 op/s 2024-03-19T01:20:23.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:23 smithi156 ceph-mon[66799]: pgmap v585: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.2 MiB/s wr, 543 op/s 2024-03-19T01:20:25.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:25 smithi162 ceph-mon[67178]: pgmap v586: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 748 op/s 2024-03-19T01:20:25.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:25 smithi116 ceph-mon[62701]: pgmap v586: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 748 op/s 2024-03-19T01:20:25.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:25 smithi156 ceph-mon[66799]: pgmap v586: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 748 op/s 2024-03-19T01:20:26.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 460.0 s, 2607.6 tps, lat 38.292 ms stddev 44.208 2024-03-19T01:20:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:27 smithi162 ceph-mon[67178]: pgmap v587: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.7 MiB/s wr, 675 op/s 2024-03-19T01:20:27.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:27 smithi116 ceph-mon[62701]: pgmap v587: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.7 MiB/s wr, 675 op/s 2024-03-19T01:20:27.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:27 smithi156 ceph-mon[66799]: pgmap v587: 97 pgs: 97 active+clean; 1.6 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 6.7 MiB/s wr, 675 op/s 2024-03-19T01:20:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.416+0000 7fb612b53640 1 -- 172.21.15.116:0/3131625325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60c101970 msgr2=0x7fb60c101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.416+0000 7fb612b53640 1 --2- 172.21.15.116:0/3131625325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60c101970 0x7fb60c101df0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00b9f0 tx=0x7fb5fc02d910 comp rx=0 tx=0).stop 2024-03-19T01:20:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.416+0000 7fb612b53640 1 -- 172.21.15.116:0/3131625325 shutdown_connections 2024-03-19T01:20:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.416+0000 7fb612b53640 1 --2- 172.21.15.116:0/3131625325 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb60c102330 0x7fb60c108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.416+0000 7fb612b53640 1 --2- 172.21.15.116:0/3131625325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60c101970 0x7fb60c101df0 secure :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00b9f0 tx=0x7fb5fc02d910 comp rx=0 tx=0).stop 2024-03-19T01:20:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.416+0000 7fb612b53640 1 --2- 172.21.15.116:0/3131625325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb60c100780 0x7fb60c100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.416+0000 7fb612b53640 1 -- 172.21.15.116:0/3131625325 >> 172.21.15.116:0/3131625325 conn(0x7fb60c0fbcf0 msgr2=0x7fb60c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.417+0000 7fb612b53640 1 -- 172.21.15.116:0/3131625325 shutdown_connections 2024-03-19T01:20:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.417+0000 7fb612b53640 1 -- 172.21.15.116:0/3131625325 wait complete. 2024-03-19T01:20:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.418+0000 7fb612b53640 1 Processor -- start 2024-03-19T01:20:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.418+0000 7fb612b53640 1 -- start start 2024-03-19T01:20:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.418+0000 7fb612b53640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb60c100780 0x7fb60c195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.418+0000 7fb612b53640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60c102330 0x7fb60c1960e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.418+0000 7fb612b53640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb60c1ac220 0x7fb60c1ae610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.418+0000 7fb60bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60c102330 0x7fb60c1960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.418+0000 7fb6108ef640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb60c100780 0x7fb60c195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.418+0000 7fb60bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60c102330 0x7fb60c1960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33462/0 (socket says 172.21.15.116:33462) 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.418+0000 7fb60bfff640 1 -- 172.21.15.116:0/3057876448 learned_addr learned my addr 172.21.15.116:0/3057876448 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.418+0000 7fb6108ef640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb60c100780 0x7fb60c195ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:40326/0 (socket says 172.21.15.116:40326) 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.418+0000 7fb612b53640 1 -- 172.21.15.116:0/3057876448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c10b350 con 0x7fb60c102330 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.418+0000 7fb6110f0640 1 --2- 172.21.15.116:0/3057876448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb60c1ac220 0x7fb60c1ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.418+0000 7fb612b53640 1 -- 172.21.15.116:0/3057876448 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c10b1d0 con 0x7fb60c100780 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.419+0000 7fb612b53640 1 -- 172.21.15.116:0/3057876448 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c10b4d0 con 0x7fb60c1ac220 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.419+0000 7fb60bfff640 1 -- 172.21.15.116:0/3057876448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb60c1ac220 msgr2=0x7fb60c1ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.419+0000 7fb60bfff640 1 --2- 172.21.15.116:0/3057876448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb60c1ac220 0x7fb60c1ae610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.419+0000 7fb60bfff640 1 -- 172.21.15.116:0/3057876448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb60c100780 msgr2=0x7fb60c195ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.419+0000 7fb60bfff640 1 --2- 172.21.15.116:0/3057876448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb60c100780 0x7fb60c195ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.419+0000 7fb60bfff640 1 -- 172.21.15.116:0/3057876448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60c1aeb80 con 0x7fb60c102330 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.419+0000 7fb60bfff640 1 --2- 172.21.15.116:0/3057876448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60c102330 0x7fb60c1960e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc009a70 tx=0x7fb5fc009470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:28.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.419+0000 7fb609ffb640 1 -- 172.21.15.116:0/3057876448 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc037050 con 0x7fb60c102330 2024-03-19T01:20:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.419+0000 7fb609ffb640 1 -- 172.21.15.116:0/3057876448 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5fc033b90 con 0x7fb60c102330 2024-03-19T01:20:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.420+0000 7fb612b53640 1 -- 172.21.15.116:0/3057876448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb60c1aee10 con 0x7fb60c102330 2024-03-19T01:20:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.420+0000 7fb609ffb640 1 -- 172.21.15.116:0/3057876448 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc03c940 con 0x7fb60c102330 2024-03-19T01:20:28.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.420+0000 7fb612b53640 1 -- 172.21.15.116:0/3057876448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb60c1af1c0 con 0x7fb60c102330 2024-03-19T01:20:28.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.423+0000 7fb61581a740 1 Processor -- start 2024-03-19T01:20:28.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.424+0000 7fb61581a740 1 -- start start 2024-03-19T01:20:28.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.424+0000 7fb61581a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4120 0x5572cb6f4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.424+0000 7fb61581a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f4b10 0x5572cb6f9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.424+0000 7fb61581a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f9c80 0x5572cb6fc110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.424+0000 7fb61581a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5572cb658be0 con 0x5572cb6f9c80 2024-03-19T01:20:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.424+0000 7fb61581a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5572cb601960 con 0x5572cb6f4120 2024-03-19T01:20:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.424+0000 7fb61581a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5572cb654c80 con 0x5572cb6f4b10 2024-03-19T01:20:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.424+0000 7fb6110f0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4120 0x5572cb6f4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.424+0000 7fb6110f0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4120 0x5572cb6f4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:40340/0 (socket says 172.21.15.116:40340) 2024-03-19T01:20:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.424+0000 7fb6110f0640 1 -- 172.21.15.116:0/4202508076 learned_addr learned my addr 172.21.15.116:0/4202508076 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.424+0000 7fb6108ef640 1 --2- 172.21.15.116:0/4202508076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f4b10 0x5572cb6f9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.424+0000 7fb609ffb640 1 -- 172.21.15.116:0/3057876448 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fb5fc03a020 con 0x7fb60c102330 2024-03-19T01:20:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.425+0000 7fb6110f0640 1 -- 172.21.15.116:0/4202508076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f4b10 msgr2=0x5572cb6f9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:28.426 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.425+0000 7fb6110f0640 1 --2- 172.21.15.116:0/4202508076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f4b10 0x5572cb6f9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.425+0000 7fb6110f0640 1 -- 172.21.15.116:0/4202508076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f9c80 msgr2=0x5572cb6fc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.425+0000 7fb6110f0640 1 --2- 172.21.15.116:0/4202508076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f9c80 0x5572cb6fc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.425+0000 7fb6110f0640 1 -- 172.21.15.116:0/4202508076 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572cb5f8d20 con 0x5572cb6f4120 2024-03-19T01:20:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.425+0000 7fb6110f0640 1 --2- 172.21.15.116:0/4202508076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4120 0x5572cb6f4540 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb604009e90 tx=0x7fb6040107e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc892c6d3166b554 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.425+0000 7fb5f2ffd640 1 -- 172.21.15.116:0/4202508076 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb604003b70 con 0x5572cb6f4120 2024-03-19T01:20:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.425+0000 7fb5f2ffd640 1 -- 172.21.15.116:0/4202508076 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb604003d10 con 0x5572cb6f4120 2024-03-19T01:20:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.425+0000 7fb5f2ffd640 1 -- 172.21.15.116:0/4202508076 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60400ebf0 con 0x5572cb6f4120 2024-03-19T01:20:28.427 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.425+0000 7fb609ffb640 1 --2- 172.21.15.116:0/3057876448 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb5f40a2970 0x7fb5f40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.425+0000 7fb6108ef640 1 --2- 172.21.15.116:0/3057876448 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb5f40a2970 0x7fb5f40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.425+0000 7fb609ffb640 1 -- 172.21.15.116:0/3057876448 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb5fc0f0d80 con 0x7fb60c102330 2024-03-19T01:20:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.426+0000 7fb61581a740 1 -- 172.21.15.116:0/4202508076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4120 msgr2=0x5572cb6f4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:28.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.426+0000 7fb6108ef640 1 --2- 172.21.15.116:0/3057876448 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb5f40a2970 0x7fb5f40a4e30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x5572cb6f4f10 tx=0x7fb60000a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.426+0000 7fb61581a740 1 --2- 172.21.15.116:0/4202508076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4120 0x5572cb6f4540 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb604009e90 tx=0x7fb6040107e0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.426+0000 7fb61581a740 1 -- 172.21.15.116:0/4202508076 shutdown_connections 2024-03-19T01:20:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.426+0000 7fb61581a740 1 --2- 172.21.15.116:0/4202508076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f9c80 0x5572cb6fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.426+0000 7fb61581a740 1 --2- 172.21.15.116:0/4202508076 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f4b10 0x5572cb6f9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.426+0000 7fb61581a740 1 --2- 172.21.15.116:0/4202508076 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4120 0x5572cb6f4540 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.426+0000 7fb61581a740 1 -- 172.21.15.116:0/4202508076 >> 172.21.15.116:0/4202508076 conn(0x5572cb6ef690 msgr2=0x5572cb6efa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.426+0000 7fb61581a740 1 -- 172.21.15.116:0/4202508076 shutdown_connections 2024-03-19T01:20:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.426+0000 7fb61581a740 1 -- 172.21.15.116:0/4202508076 wait complete. 2024-03-19T01:20:28.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.427+0000 7fb61581a740 1 -- start start 2024-03-19T01:20:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.427+0000 7fb61581a740 1 Processor -- start 2024-03-19T01:20:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.427+0000 7fb61581a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f4120 0x5572cb6fe170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.427+0000 7fb61581a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb6fe6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.427+0000 7fb61581a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f9c80 0x5572cb7056f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb61581a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5572cb6fcc30 con 0x5572cb6f4120 2024-03-19T01:20:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb61581a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5572cb614db0 con 0x5572cb6f4b10 2024-03-19T01:20:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb61581a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5572cb6fcdb0 con 0x5572cb6f9c80 2024-03-19T01:20:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb6108ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f4120 0x5572cb6fe170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb60bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb6fe6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb60bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb6fe6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:40356/0 (socket says 172.21.15.116:40356) 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb60bfff640 1 -- 172.21.15.116:0/4278669791 learned_addr learned my addr 172.21.15.116:0/4278669791 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb6108ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f4120 0x5572cb6fe170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33470/0 (socket says 172.21.15.116:33470) 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb61581a740 10 client.0 _ll_register_callbacks cb 0x5572cb619e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb6110f0640 1 --2- 172.21.15.116:0/4278669791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f9c80 0x5572cb7056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb60bfff640 1 -- 172.21.15.116:0/4278669791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f9c80 msgr2=0x5572cb7056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb60bfff640 1 --2- 172.21.15.116:0/4278669791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f9c80 0x5572cb7056f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb60bfff640 1 -- 172.21.15.116:0/4278669791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f4120 msgr2=0x5572cb6fe170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb60bfff640 1 --2- 172.21.15.116:0/4278669791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f4120 0x5572cb6fe170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb60bfff640 1 -- 172.21.15.116:0/4278669791 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572cb613c60 con 0x5572cb6f4b10 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb60bfff640 1 --2- 172.21.15.116:0/4278669791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb6fe6b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0c5910 tx=0x7fb5fc0bd1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa7198dce61202e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb5ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb5ebfff640 1 -- 172.21.15.116:0/4278669791 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0a80b0 con 0x5572cb6f4b10 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb5ebfff640 1 -- 172.21.15.116:0/4278669791 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb5fc0c59b0 con 0x5572cb6f4b10 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.428+0000 7fb61581a740 1 -- 172.21.15.116:0/4278669791 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb604003d10 con 0x5572cb6f4b10 2024-03-19T01:20:28.435 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.429+0000 7fb5ebfff640 1 -- 172.21.15.116:0/4278669791 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb5fc0bd6a0 con 0x5572cb6f4b10 2024-03-19T01:20:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.429+0000 7fb61581a740 10 client.25748 fetch_fsmap learned FSMap version 45 2024-03-19T01:20:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.429+0000 7fb61581a740 1 -- 172.21.15.116:0/4278669791 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5572cb36d210 con 0x5572cb6f4b10 2024-03-19T01:20:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.429+0000 7fb5ebfff640 1 -- 172.21.15.116:0/4278669791 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb604003d10 con 0x5572cb6f4b10 2024-03-19T01:20:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.429+0000 7fb5ebfff640 1 -- 172.21.15.116:0/4278669791 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fb5fc04f030 con 0x5572cb6f4b10 2024-03-19T01:20:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.429+0000 7fb61581a740 10 client.25748 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:20:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.429+0000 7fb61581a740 10 client.25748 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:20:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.429+0000 7fb61581a740 20 client.25748 populate_metadata read hostname 'smithi116' 2024-03-19T01:20:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.429+0000 7fb61581a740 1 --2- 172.21.15.116:0/4278669791 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5572cb709d30 0x5572cb70c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.429+0000 7fb61581a740 4 client.25748 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:20:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.430+0000 7fb61581a740 1 -- 172.21.15.116:0/4278669791 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572cb666120 con 0x5572cb709d30 2024-03-19T01:20:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.430+0000 7fb6108ef640 1 --2- 172.21.15.116:0/4278669791 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5572cb709d30 0x5572cb70c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.430+0000 7fb5ebfff640 10 client.25748 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:28.436 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.430+0000 7fb6108ef640 1 --2- 172.21.15.116:0/4278669791 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5572cb709d30 0x5572cb70c120 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c8fd6bbdc76a54b server_cookie=1e2d402d60ec3642 in_seq=0 out_seq=0 2024-03-19T01:20:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb5ebfff640 1 -- 172.21.15.116:0/4278669791 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5572cb666120 con 0x5572cb709d30 2024-03-19T01:20:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb5ebfff640 10 client.25748 handle_command_reply: tid=0 2024-03-19T01:20:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb61581a740 1 client.25748 shutdown 2024-03-19T01:20:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb61581a740 1 -- 172.21.15.116:0/4278669791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 msgr2=0x5572cb6fe6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb61581a740 1 --2- 172.21.15.116:0/4278669791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb6fe6b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc0c5910 tx=0x7fb5fc0bd1e0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb61581a740 1 -- 172.21.15.116:0/4278669791 shutdown_connections 2024-03-19T01:20:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb61581a740 1 --2- 172.21.15.116:0/4278669791 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5572cb709d30 0x5572cb70c120 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb61581a740 1 --2- 172.21.15.116:0/4278669791 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f9c80 0x5572cb7056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb5ebfff640 0 client.25748 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:28.437 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb61581a740 1 --2- 172.21.15.116:0/4278669791 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb6fe6b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb61581a740 1 --2- 172.21.15.116:0/4278669791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f4120 0x5572cb6fe170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb61581a740 1 -- 172.21.15.116:0/4278669791 >> 172.21.15.116:0/4278669791 conn(0x5572cb6ef690 msgr2=0x5572cb6f08e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb61581a740 1 -- 172.21.15.116:0/4278669791 shutdown_connections 2024-03-19T01:20:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb61581a740 1 -- 172.21.15.116:0/4278669791 wait complete. 2024-03-19T01:20:28.438 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.431+0000 7fb61581a740 20 client.25748 trim_cache size 0 max 16384 2024-03-19T01:20:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.455+0000 7fb61581a740 1 Processor -- start 2024-03-19T01:20:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb61581a740 1 -- start start 2024-03-19T01:20:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb61581a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f4120 0x5572cb6eeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb61581a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb7056b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb61581a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f9c80 0x5572cb6f3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb61581a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c1b8180 con 0x5572cb6f9c80 2024-03-19T01:20:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb61581a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c1b9cc0 con 0x5572cb6f4b10 2024-03-19T01:20:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb61581a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c1b8850 con 0x5572cb6f4120 2024-03-19T01:20:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb6110f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f9c80 0x5572cb6f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb60bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb7056b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb60bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb7056b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:40364/0 (socket says 172.21.15.116:40364) 2024-03-19T01:20:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb60bfff640 1 -- 172.21.15.116:0/3198749591 learned_addr learned my addr 172.21.15.116:0/3198749591 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb6110f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f9c80 0x5572cb6f3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33472/0 (socket says 172.21.15.116:33472) 2024-03-19T01:20:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb6108ef640 1 --2- 172.21.15.116:0/3198749591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f4120 0x5572cb6eeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb60bfff640 1 -- 172.21.15.116:0/3198749591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f4120 msgr2=0x5572cb6eeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb60bfff640 1 --2- 172.21.15.116:0/3198749591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f4120 0x5572cb6eeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb60bfff640 1 -- 172.21.15.116:0/3198749591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f9c80 msgr2=0x5572cb6f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb60bfff640 1 --2- 172.21.15.116:0/3198749591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f9c80 0x5572cb6f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.456+0000 7fb60bfff640 1 -- 172.21.15.116:0/3198749591 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572cb70c490 con 0x5572cb6f4b10 2024-03-19T01:20:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.457+0000 7fb60bfff640 1 --2- 172.21.15.116:0/3198749591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb7056b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc000c00 tx=0x7fb5fc0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb8703dacc02fbda server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.457+0000 7fb5f27fc640 1 -- 172.21.15.116:0/3198749591 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0a80b0 con 0x5572cb6f4b10 2024-03-19T01:20:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.457+0000 7fb5f27fc640 1 -- 172.21.15.116:0/3198749591 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5fc0c5e40 con 0x5572cb6f4b10 2024-03-19T01:20:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.457+0000 7fb5f27fc640 1 -- 172.21.15.116:0/3198749591 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0b96a0 con 0x5572cb6f4b10 2024-03-19T01:20:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.458+0000 7fb61581a740 1 -- 172.21.15.116:0/3198749591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 msgr2=0x5572cb7056b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.458+0000 7fb61581a740 1 --2- 172.21.15.116:0/3198749591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb7056b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc000c00 tx=0x7fb5fc0057f0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.458+0000 7fb61581a740 1 -- 172.21.15.116:0/3198749591 shutdown_connections 2024-03-19T01:20:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.459+0000 7fb61581a740 1 --2- 172.21.15.116:0/3198749591 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f9c80 0x5572cb6f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.459+0000 7fb61581a740 1 --2- 172.21.15.116:0/3198749591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb7056b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.459+0000 7fb61581a740 1 --2- 172.21.15.116:0/3198749591 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f4120 0x5572cb6eeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.459+0000 7fb61581a740 1 -- 172.21.15.116:0/3198749591 >> 172.21.15.116:0/3198749591 conn(0x5572cb6ef690 msgr2=0x5572cb6f28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.459+0000 7fb61581a740 1 -- 172.21.15.116:0/3198749591 shutdown_connections 2024-03-19T01:20:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.459+0000 7fb61581a740 1 -- 172.21.15.116:0/3198749591 wait complete. 2024-03-19T01:20:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.459+0000 7fb61581a740 1 -- start start 2024-03-19T01:20:28.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.459+0000 7fb61581a740 1 Processor -- start 2024-03-19T01:20:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.460+0000 7fb61581a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f4120 0x5572cb703560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.460+0000 7fb61581a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb703aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.460+0000 7fb61581a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f9c80 0x5572cb704a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.460+0000 7fb61581a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c1b95f0 con 0x5572cb6f4120 2024-03-19T01:20:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.460+0000 7fb61581a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c1b8f20 con 0x5572cb6f4b10 2024-03-19T01:20:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.460+0000 7fb61581a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb60c1b6160 con 0x5572cb6f9c80 2024-03-19T01:20:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.460+0000 7fb61581a740 10 client.0 _ll_register_callbacks cb 0x7fb5fc0b9940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:20:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.460+0000 7fb60bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb703aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.460+0000 7fb6108ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f4120 0x5572cb703560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.465 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.460+0000 7fb60bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb703aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:40368/0 (socket says 172.21.15.116:40368) 2024-03-19T01:20:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.460+0000 7fb60bfff640 1 -- 172.21.15.116:0/3806914133 learned_addr learned my addr 172.21.15.116:0/3806914133 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.460+0000 7fb6110f0640 1 --2- 172.21.15.116:0/3806914133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f9c80 0x5572cb704a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.460+0000 7fb60bfff640 1 -- 172.21.15.116:0/3806914133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f9c80 msgr2=0x5572cb704a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb60bfff640 1 --2- 172.21.15.116:0/3806914133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f9c80 0x5572cb704a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb60bfff640 1 -- 172.21.15.116:0/3806914133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f4120 msgr2=0x5572cb703560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb60bfff640 1 --2- 172.21.15.116:0/3806914133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f4120 0x5572cb703560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb60bfff640 1 -- 172.21.15.116:0/3806914133 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572cb439620 con 0x5572cb6f4b10 2024-03-19T01:20:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb60bfff640 1 --2- 172.21.15.116:0/3806914133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb703aa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00e090 tx=0x7fb5fc02d910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f107fe76c3ade1f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb5ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:20:28.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb5ebfff640 1 -- 172.21.15.116:0/3806914133 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0c17f0 con 0x5572cb6f4b10 2024-03-19T01:20:28.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb5ebfff640 1 -- 172.21.15.116:0/3806914133 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb5fc0b93f0 con 0x5572cb6f4b10 2024-03-19T01:20:28.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb61581a740 1 -- 172.21.15.116:0/3806914133 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5fc0c5e40 con 0x5572cb6f4b10 2024-03-19T01:20:28.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb5ebfff640 1 -- 172.21.15.116:0/3806914133 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb5fc0a80b0 con 0x5572cb6f4b10 2024-03-19T01:20:28.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb61581a740 10 client.25754 fetch_fsmap learned FSMap version 45 2024-03-19T01:20:28.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb61581a740 1 -- 172.21.15.116:0/3806914133 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5572cb439620 con 0x5572cb6f4b10 2024-03-19T01:20:28.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb5ebfff640 1 -- 172.21.15.116:0/3806914133 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5fc0c5e40 con 0x5572cb6f4b10 2024-03-19T01:20:28.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb5ebfff640 1 -- 172.21.15.116:0/3806914133 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fb5fc04f070 con 0x5572cb6f4b10 2024-03-19T01:20:28.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb61581a740 10 client.25754 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:20:28.467 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.461+0000 7fb61581a740 10 client.25754 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:20:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.462+0000 7fb61581a740 20 client.25754 populate_metadata read hostname 'smithi116' 2024-03-19T01:20:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.462+0000 7fb61581a740 1 --2- 172.21.15.116:0/3806914133 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5572cb709d30 0x5572cb772a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.462+0000 7fb61581a740 4 client.25754 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:20:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.462+0000 7fb61581a740 1 -- 172.21.15.116:0/3806914133 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5572cb660470 con 0x5572cb709d30 2024-03-19T01:20:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.462+0000 7fb6108ef640 1 --2- 172.21.15.116:0/3806914133 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5572cb709d30 0x5572cb772a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.464+0000 7fb6108ef640 1 --2- 172.21.15.116:0/3806914133 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5572cb709d30 0x5572cb772a50 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79e846c9195f580a server_cookie=fcbd3c02e46cb740 in_seq=0 out_seq=0 2024-03-19T01:20:28.468 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.464+0000 7fb5ebfff640 10 client.25754 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:28.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.472+0000 7fb5ebfff640 1 -- 172.21.15.116:0/3806914133 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5572cb660470 con 0x5572cb709d30 2024-03-19T01:20:28.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.472+0000 7fb5ebfff640 10 client.25754 handle_command_reply: tid=0 2024-03-19T01:20:28.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.472+0000 7fb61581a740 1 client.25754 shutdown 2024-03-19T01:20:28.475 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.472+0000 7fb61581a740 1 -- 172.21.15.116:0/3806914133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 msgr2=0x5572cb703aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:28.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.472+0000 7fb61581a740 1 --2- 172.21.15.116:0/3806914133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb703aa0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00e090 tx=0x7fb5fc02d910 comp rx=0 tx=0).stop 2024-03-19T01:20:28.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.472+0000 7fb61581a740 1 -- 172.21.15.116:0/3806914133 shutdown_connections 2024-03-19T01:20:28.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.472+0000 7fb61581a740 1 --2- 172.21.15.116:0/3806914133 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5572cb709d30 0x5572cb772a50 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.472+0000 7fb61581a740 1 --2- 172.21.15.116:0/3806914133 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5572cb6f9c80 0x5572cb704a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.476 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.472+0000 7fb61581a740 1 --2- 172.21.15.116:0/3806914133 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5572cb6f4b10 0x5572cb703aa0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.472+0000 7fb5ebfff640 0 client.25754 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:28.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.472+0000 7fb61581a740 1 --2- 172.21.15.116:0/3806914133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5572cb6f4120 0x5572cb703560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.472+0000 7fb61581a740 1 -- 172.21.15.116:0/3806914133 >> 172.21.15.116:0/3806914133 conn(0x5572cb6ef690 msgr2=0x5572cb6f3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:28.477 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.472+0000 7fb61581a740 1 -- 172.21.15.116:0/3806914133 shutdown_connections 2024-03-19T01:20:28.478 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.473+0000 7fb61581a740 1 -- 172.21.15.116:0/3806914133 wait complete. 2024-03-19T01:20:28.478 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.473+0000 7fb61581a740 20 client.25754 trim_cache size 0 max 16384 2024-03-19T01:20:28.478 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:20:28.478 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:20:28.479 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:20:28.480 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:20:28.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.474+0000 7fb612b53640 1 -- 172.21.15.116:0/3057876448 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb5f40a2970 msgr2=0x7fb5f40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:28.481 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.474+0000 7fb612b53640 1 --2- 172.21.15.116:0/3057876448 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb5f40a2970 0x7fb5f40a4e30 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x5572cb6f4f10 tx=0x7fb60000a2d0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.475+0000 7fb612b53640 1 -- 172.21.15.116:0/3057876448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60c102330 msgr2=0x7fb60c1960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.475+0000 7fb612b53640 1 --2- 172.21.15.116:0/3057876448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60c102330 0x7fb60c1960e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc009a70 tx=0x7fb5fc009470 comp rx=0 tx=0).stop 2024-03-19T01:20:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.475+0000 7fb612b53640 1 -- 172.21.15.116:0/3057876448 shutdown_connections 2024-03-19T01:20:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.475+0000 7fb612b53640 1 --2- 172.21.15.116:0/3057876448 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb60c1ac220 0x7fb60c1ae610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.475+0000 7fb612b53640 1 --2- 172.21.15.116:0/3057876448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb60c102330 0x7fb60c1960e0 secure :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc009a70 tx=0x7fb5fc009470 comp rx=0 tx=0).stop 2024-03-19T01:20:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.475+0000 7fb612b53640 1 --2- 172.21.15.116:0/3057876448 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb5f40a2970 0x7fb5f40a4e30 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.475+0000 7fb612b53640 1 --2- 172.21.15.116:0/3057876448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb60c100780 0x7fb60c195ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.475+0000 7fb612b53640 1 -- 172.21.15.116:0/3057876448 >> 172.21.15.116:0/3057876448 conn(0x7fb60c0fbcf0 msgr2=0x7fb60c104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.475+0000 7fb612b53640 1 -- 172.21.15.116:0/3057876448 shutdown_connections 2024-03-19T01:20:28.484 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:28.475+0000 7fb612b53640 1 -- 172.21.15.116:0/3057876448 wait complete. 2024-03-19T01:20:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:29 smithi162 ceph-mon[67178]: pgmap v588: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.2 MiB/s wr, 729 op/s 2024-03-19T01:20:29.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:29 smithi116 ceph-mon[62701]: pgmap v588: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.2 MiB/s wr, 729 op/s 2024-03-19T01:20:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:29 smithi156 ceph-mon[66799]: pgmap v588: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.2 MiB/s wr, 729 op/s 2024-03-19T01:20:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 465.0 s, 1945.8 tps, lat 50.883 ms stddev 56.273 2024-03-19T01:20:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:31 smithi162 ceph-mon[67178]: pgmap v589: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.9 MiB/s wr, 763 op/s 2024-03-19T01:20:31.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:31 smithi116 ceph-mon[62701]: pgmap v589: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.9 MiB/s wr, 763 op/s 2024-03-19T01:20:31.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:31 smithi156 ceph-mon[66799]: pgmap v589: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 7.9 MiB/s wr, 763 op/s 2024-03-19T01:20:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:33 smithi162 ceph-mon[67178]: pgmap v590: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.5 MiB/s wr, 624 op/s 2024-03-19T01:20:33.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:33 smithi116 ceph-mon[62701]: pgmap v590: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.5 MiB/s wr, 624 op/s 2024-03-19T01:20:33.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:33 smithi156 ceph-mon[66799]: pgmap v590: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.5 MiB/s wr, 624 op/s 2024-03-19T01:20:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:35 smithi162 ceph-mon[67178]: pgmap v591: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 775 op/s 2024-03-19T01:20:35.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:35 smithi116 ceph-mon[62701]: pgmap v591: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 775 op/s 2024-03-19T01:20:35.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:35 smithi156 ceph-mon[66799]: pgmap v591: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 775 op/s 2024-03-19T01:20:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 470.0 s, 1601.8 tps, lat 62.544 ms stddev 64.947 2024-03-19T01:20:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:37 smithi162 ceph-mon[67178]: pgmap v592: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.9 MiB/s wr, 666 op/s 2024-03-19T01:20:37.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:37 smithi116 ceph-mon[62701]: pgmap v592: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.9 MiB/s wr, 666 op/s 2024-03-19T01:20:37.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:37 smithi156 ceph-mon[66799]: pgmap v592: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.9 MiB/s wr, 666 op/s 2024-03-19T01:20:39.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:39 smithi162 ceph-mon[67178]: pgmap v593: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.1 MiB/s wr, 692 op/s 2024-03-19T01:20:39.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:39 smithi116 ceph-mon[62701]: pgmap v593: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.1 MiB/s wr, 692 op/s 2024-03-19T01:20:39.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:39 smithi156 ceph-mon[66799]: pgmap v593: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.1 MiB/s wr, 692 op/s 2024-03-19T01:20:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 475.0 s, 2049.2 tps, lat 49.280 ms stddev 54.665 2024-03-19T01:20:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.618+0000 7f02adfc6640 1 -- 172.21.15.116:0/1593413134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02a8102330 msgr2=0x7f02a8108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:41.622 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.618+0000 7f02adfc6640 1 --2- 172.21.15.116:0/1593413134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02a8102330 0x7f02a8108bc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f029001a910 tx=0x7f029003cca0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.626+0000 7f02adfc6640 1 -- 172.21.15.116:0/1593413134 shutdown_connections 2024-03-19T01:20:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.626+0000 7f02adfc6640 1 --2- 172.21.15.116:0/1593413134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02a8102330 0x7f02a8108bc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.626+0000 7f02adfc6640 1 --2- 172.21.15.116:0/1593413134 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02a8101970 0x7f02a8101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.626+0000 7f02adfc6640 1 --2- 172.21.15.116:0/1593413134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02a8100780 0x7f02a8100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.626+0000 7f02adfc6640 1 -- 172.21.15.116:0/1593413134 >> 172.21.15.116:0/1593413134 conn(0x7f02a80fbcf0 msgr2=0x7f02a80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:41.632 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.630+0000 7f02adfc6640 1 -- 172.21.15.116:0/1593413134 shutdown_connections 2024-03-19T01:20:41.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.633+0000 7f02adfc6640 1 -- 172.21.15.116:0/1593413134 wait complete. 2024-03-19T01:20:41.635 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.634+0000 7f02adfc6640 1 Processor -- start 2024-03-19T01:20:41.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.639+0000 7f02adfc6640 1 -- start start 2024-03-19T01:20:41.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.639+0000 7f02adfc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02a8100780 0x7f02a8107cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.640 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.639+0000 7f02adfc6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02a8101970 0x7f02a8108230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.639+0000 7f02adfc6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02a8102330 0x7f02a819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.639+0000 7f02adfc6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02a8071290 con 0x7f02a8100780 2024-03-19T01:20:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.639+0000 7f02adfc6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f02a8071110 con 0x7f02a8101970 2024-03-19T01:20:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.639+0000 7f02adfc6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02a8071410 con 0x7f02a8102330 2024-03-19T01:20:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.640+0000 7f02a6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02a8101970 0x7f02a8108230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.640+0000 7f02a6ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02a8101970 0x7f02a8108230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:51194/0 (socket says 172.21.15.116:51194) 2024-03-19T01:20:41.641 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.640+0000 7f02a6ffd640 1 -- 172.21.15.116:0/2660843239 learned_addr learned my addr 172.21.15.116:0/2660843239 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:41.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.640+0000 7f02a6ffd640 1 -- 172.21.15.116:0/2660843239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02a8102330 msgr2=0x7f02a819ce90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:20:41.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.640+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/2660843239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02a8102330 0x7f02a819ce90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.640+0000 7f02a6ffd640 1 -- 172.21.15.116:0/2660843239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02a8100780 msgr2=0x7f02a8107cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:41.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.640+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/2660843239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02a8100780 0x7f02a8107cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.642 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.640+0000 7f02a6ffd640 1 -- 172.21.15.116:0/2660843239 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a8109910 con 0x7f02a8101970 2024-03-19T01:20:41.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.641+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/2660843239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02a8101970 0x7f02a8108230 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f029800ba80 tx=0x7f029800bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:41.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.641+0000 7f02a4ff9640 1 -- 172.21.15.116:0/2660843239 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298019070 con 0x7f02a8101970 2024-03-19T01:20:41.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.641+0000 7f02a4ff9640 1 -- 172.21.15.116:0/2660843239 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f029800cc00 con 0x7f02a8101970 2024-03-19T01:20:41.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.641+0000 7f02adfc6640 1 -- 172.21.15.116:0/2660843239 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02a8109ba0 con 0x7f02a8101970 2024-03-19T01:20:41.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.642+0000 7f02adfc6640 1 -- 172.21.15.116:0/2660843239 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02a810a170 con 0x7f02a8101970 2024-03-19T01:20:41.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.642+0000 7f02a4ff9640 1 -- 172.21.15.116:0/2660843239 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298007650 con 0x7f02a8101970 2024-03-19T01:20:41.648 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.646+0000 7f02b0bb4740 1 Processor -- start 2024-03-19T01:20:41.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:41 smithi162 ceph-mon[67178]: pgmap v594: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.7 MiB/s wr, 669 op/s 2024-03-19T01:20:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.649+0000 7f02a4ff9640 1 -- 172.21.15.116:0/2660843239 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f029801c070 con 0x7f02a8101970 2024-03-19T01:20:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.649+0000 7f02a4ff9640 1 --2- 172.21.15.116:0/2660843239 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02940a2ad0 0x7f02940a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.650+0000 7f02a4ff9640 1 -- 172.21.15.116:0/2660843239 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f02980ce020 con 0x7f02a8101970 2024-03-19T01:20:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.650+0000 7f02b0bb4740 1 -- start start 2024-03-19T01:20:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02b0bb4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 0x559110be6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02a7fff640 1 --2- 172.21.15.116:0/2660843239 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02940a2ad0 0x7f02940a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02b0bb4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110beb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02b0bb4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110bebc60 0x559110bee0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02b0bb4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559110b4abe0 con 0x559110be6af0 2024-03-19T01:20:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02b0bb4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559110af3960 con 0x559110be6100 2024-03-19T01:20:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02b0bb4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559110b46c80 con 0x559110bebc60 2024-03-19T01:20:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110beb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110beb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49502/0 (socket says 172.21.15.116:49502) 2024-03-19T01:20:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02a6ffd640 1 -- 172.21.15.116:0/2252839955 learned_addr learned my addr 172.21.15.116:0/2252839955 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02a6ffd640 1 -- 172.21.15.116:0/2252839955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110bebc60 msgr2=0x559110bee0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:20:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/2252839955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110bebc60 0x559110bee0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02a6ffd640 1 -- 172.21.15.116:0/2252839955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 msgr2=0x559110be6520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:20:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/2252839955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 0x559110be6520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.651+0000 7f02a6ffd640 1 -- 172.21.15.116:0/2252839955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559110aead20 con 0x559110be6af0 2024-03-19T01:20:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.652+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/2252839955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110beb5f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0298097310 tx=0x7f029809a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5f7c97c05be0d66 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.652+0000 7f029dffb640 1 -- 172.21.15.116:0/2252839955 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02980b3070 con 0x559110be6af0 2024-03-19T01:20:41.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.652+0000 7f029dffb640 1 -- 172.21.15.116:0/2252839955 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02980a30e0 con 0x559110be6af0 2024-03-19T01:20:41.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.652+0000 7f029dffb640 1 -- 172.21.15.116:0/2252839955 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298099d90 con 0x559110be6af0 2024-03-19T01:20:41.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.653+0000 7f02b0bb4740 1 -- 172.21.15.116:0/2252839955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 msgr2=0x559110beb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:41.655 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.653+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/2252839955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110beb5f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0298097310 tx=0x7f029809a710 comp rx=0 tx=0).stop 2024-03-19T01:20:41.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:41 smithi116 ceph-mon[62701]: pgmap v594: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.7 MiB/s wr, 669 op/s 2024-03-19T01:20:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.654+0000 7f02a7fff640 1 --2- 172.21.15.116:0/2660843239 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02940a2ad0 0x7f02940a4f90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0290005ab0 tx=0x7f0290005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.654+0000 7f02b0bb4740 1 -- 172.21.15.116:0/2252839955 shutdown_connections 2024-03-19T01:20:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.654+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/2252839955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110bebc60 0x559110bee0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.654+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/2252839955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110beb5f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.654+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/2252839955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 0x559110be6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.654+0000 7f02b0bb4740 1 -- 172.21.15.116:0/2252839955 >> 172.21.15.116:0/2252839955 conn(0x559110be1670 msgr2=0x559110be1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:41.659 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.657+0000 7f02b0bb4740 1 -- 172.21.15.116:0/2252839955 shutdown_connections 2024-03-19T01:20:41.659 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.658+0000 7f02b0bb4740 1 -- 172.21.15.116:0/2252839955 wait complete. 2024-03-19T01:20:41.659 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.658+0000 7f02b0bb4740 1 -- start start 2024-03-19T01:20:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.658+0000 7f02b0bb4740 1 Processor -- start 2024-03-19T01:20:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.660+0000 7f02b0bb4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 0x559110bf0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.660+0000 7f02b0bb4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110be6af0 0x559110bf06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.660+0000 7f02b0bb4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110bebc60 0x559110bf7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.660+0000 7f02b0bb4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559110beec10 con 0x559110bebc60 2024-03-19T01:20:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.660+0000 7f02b0bb4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559110b06db0 con 0x559110be6100 2024-03-19T01:20:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.660+0000 7f02b0bb4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559110beed90 con 0x559110be6af0 2024-03-19T01:20:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.660+0000 7f02b0bb4740 10 client.0 _ll_register_callbacks cb 0x559110b0be70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:20:41.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.661+0000 7f02a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110bebc60 0x559110bf7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:41.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.661+0000 7f02a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110bebc60 0x559110bf7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49514/0 (socket says 172.21.15.116:49514) 2024-03-19T01:20:41.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.661+0000 7f02a7fff640 1 -- 172.21.15.116:0/3259852749 learned_addr learned my addr 172.21.15.116:0/3259852749 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:41.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.661+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/3259852749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110be6af0 0x559110bf06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:41.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.661+0000 7f02a7fff640 1 -- 172.21.15.116:0/3259852749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110be6af0 msgr2=0x559110bf06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:41.663 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.661+0000 7f02a7fff640 1 --2- 172.21.15.116:0/3259852749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110be6af0 0x559110bf06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.661+0000 7f02a7fff640 1 -- 172.21.15.116:0/3259852749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 msgr2=0x559110bf0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:41.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.661+0000 7f02a7fff640 1 --2- 172.21.15.116:0/3259852749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 0x559110bf0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.661+0000 7f02a7fff640 1 -- 172.21.15.116:0/3259852749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559110b05c60 con 0x559110bebc60 2024-03-19T01:20:41.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.662+0000 7f02a7fff640 1 --2- 172.21.15.116:0/3259852749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110bebc60 0x559110bf7710 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f029001a910 tx=0x7f029003cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d66fae8b22c2f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:41.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.662+0000 7f0286ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:20:41.664 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.663+0000 7f0286ffd640 1 -- 172.21.15.116:0/3259852749 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029005e020 con 0x559110bebc60 2024-03-19T01:20:41.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.663+0000 7f02b0bb4740 1 -- 172.21.15.116:0/3259852749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02980a30e0 con 0x559110bebc60 2024-03-19T01:20:41.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.663+0000 7f0286ffd640 1 -- 172.21.15.116:0/3259852749 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f029003dc50 con 0x559110bebc60 2024-03-19T01:20:41.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.664+0000 7f0286ffd640 1 -- 172.21.15.116:0/3259852749 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f0290069050 con 0x559110bebc60 2024-03-19T01:20:41.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.664+0000 7f02b0bb4740 10 client.15930 fetch_fsmap learned FSMap version 45 2024-03-19T01:20:41.665 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.664+0000 7f02b0bb4740 1 -- 172.21.15.116:0/3259852749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55911085f210 con 0x559110bebc60 2024-03-19T01:20:41.666 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.665+0000 7f0286ffd640 1 -- 172.21.15.116:0/3259852749 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02980a30e0 con 0x559110bebc60 2024-03-19T01:20:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.666+0000 7f0286ffd640 1 -- 172.21.15.116:0/3259852749 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f0290059030 con 0x559110bebc60 2024-03-19T01:20:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.666+0000 7f02b0bb4740 10 client.15930 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:20:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.666+0000 7f02b0bb4740 10 client.15930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:20:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.666+0000 7f02b0bb4740 20 client.15930 populate_metadata read hostname 'smithi116' 2024-03-19T01:20:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.666+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/3259852749 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559110bfc2b0 0x559110bfe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.666+0000 7f02b0bb4740 4 client.15930 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:20:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.666+0000 7f02b0bb4740 1 -- 172.21.15.116:0/3259852749 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559110b58120 con 0x559110bfc2b0 2024-03-19T01:20:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.666+0000 7f02a77fe640 1 --2- 172.21.15.116:0/3259852749 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559110bfc2b0 0x559110bfe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.667+0000 7f0286ffd640 10 client.15930 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:41.668 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.667+0000 7f02a77fe640 1 --2- 172.21.15.116:0/3259852749 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559110bfc2b0 0x559110bfe770 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efb80dba7a3e2f70 server_cookie=b9c9b20d6a6d2c2a in_seq=0 out_seq=0 2024-03-19T01:20:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.668+0000 7f0286ffd640 1 -- 172.21.15.116:0/3259852749 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559110b58120 con 0x559110bfc2b0 2024-03-19T01:20:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.668+0000 7f0286ffd640 10 client.15930 handle_command_reply: tid=0 2024-03-19T01:20:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.668+0000 7f02b0bb4740 1 client.15930 shutdown 2024-03-19T01:20:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.669+0000 7f02b0bb4740 1 -- 172.21.15.116:0/3259852749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110bebc60 msgr2=0x559110bf7710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.669+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/3259852749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110bebc60 0x559110bf7710 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f029001a910 tx=0x7f029003cca0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.669+0000 7f02b0bb4740 1 -- 172.21.15.116:0/3259852749 shutdown_connections 2024-03-19T01:20:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.669+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/3259852749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110bebc60 0x559110bf7710 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.669+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/3259852749 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559110bfc2b0 0x559110bfe770 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.669+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/3259852749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110be6af0 0x559110bf06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.669+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/3259852749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 0x559110bf0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.669+0000 7f02b0bb4740 1 -- 172.21.15.116:0/3259852749 >> 172.21.15.116:0/3259852749 conn(0x559110be1670 msgr2=0x559110be28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:41.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.669+0000 7f0286ffd640 0 client.15930 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:41.671 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.669+0000 7f02b0bb4740 1 -- 172.21.15.116:0/3259852749 shutdown_connections 2024-03-19T01:20:41.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.670+0000 7f02b0bb4740 1 -- 172.21.15.116:0/3259852749 wait complete. 2024-03-19T01:20:41.672 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.670+0000 7f02b0bb4740 20 client.15930 trim_cache size 0 max 16384 2024-03-19T01:20:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02b0bb4740 1 Processor -- start 2024-03-19T01:20:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02b0bb4740 1 -- start start 2024-03-19T01:20:41.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02b0bb4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110be6100 0x559110bfe2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02b0bb4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110bf76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02b0bb4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110bebc60 0x559110be7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02b0bb4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f028800e640 con 0x559110be6af0 2024-03-19T01:20:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110bf76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110bf76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49526/0 (socket says 172.21.15.116:49526) 2024-03-19T01:20:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02a6ffd640 1 -- 172.21.15.116:0/2421337442 learned_addr learned my addr 172.21.15.116:0/2421337442 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02b0bb4740 1 -- 172.21.15.116:0/2421337442 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0288010180 con 0x559110bebc60 2024-03-19T01:20:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02a77fe640 1 --2- 172.21.15.116:0/2421337442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110be6100 0x559110bfe2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02b0bb4740 1 -- 172.21.15.116:0/2421337442 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f028800ed10 con 0x559110be6100 2024-03-19T01:20:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02a7fff640 1 --2- 172.21.15.116:0/2421337442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110bebc60 0x559110be7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02a6ffd640 1 -- 172.21.15.116:0/2421337442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110be6100 msgr2=0x559110bfe2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/2421337442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110be6100 0x559110bfe2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.695 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02a6ffd640 1 -- 172.21.15.116:0/2421337442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110bebc60 msgr2=0x559110be7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/2421337442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110bebc60 0x559110be7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.692+0000 7f02a6ffd640 1 -- 172.21.15.116:0/2421337442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55911092b620 con 0x559110be6af0 2024-03-19T01:20:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.693+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/2421337442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110bf76d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0298096da0 tx=0x7f0298014d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8c14d245293e73 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.693+0000 7f029d7fa640 1 -- 172.21.15.116:0/2421337442 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02980b3040 con 0x559110be6af0 2024-03-19T01:20:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.693+0000 7f029d7fa640 1 -- 172.21.15.116:0/2421337442 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02980b8070 con 0x559110be6af0 2024-03-19T01:20:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.693+0000 7f029d7fa640 1 -- 172.21.15.116:0/2421337442 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298099d90 con 0x559110be6af0 2024-03-19T01:20:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.694+0000 7f02b0bb4740 1 -- 172.21.15.116:0/2421337442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 msgr2=0x559110bf76d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.694+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/2421337442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110bf76d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0298096da0 tx=0x7f0298014d20 comp rx=0 tx=0).stop 2024-03-19T01:20:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.694+0000 7f02b0bb4740 1 -- 172.21.15.116:0/2421337442 shutdown_connections 2024-03-19T01:20:41.696 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.694+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/2421337442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110bebc60 0x559110be7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.694+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/2421337442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110bf76d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.694+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/2421337442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110be6100 0x559110bfe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.694+0000 7f02b0bb4740 1 -- 172.21.15.116:0/2421337442 >> 172.21.15.116:0/2421337442 conn(0x559110be1670 msgr2=0x559110be0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.695+0000 7f02b0bb4740 1 -- 172.21.15.116:0/2421337442 shutdown_connections 2024-03-19T01:20:41.697 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.696+0000 7f02b0bb4740 1 -- 172.21.15.116:0/2421337442 wait complete. 2024-03-19T01:20:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.696+0000 7f02b0bb4740 1 -- start start 2024-03-19T01:20:41.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.696+0000 7f02b0bb4740 1 Processor -- start 2024-03-19T01:20:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02b0bb4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 0x559110bf5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02b0bb4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110bf59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02b0bb4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110bebc60 0x559110bf69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02b0bb4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f028800fab0 con 0x559110be6af0 2024-03-19T01:20:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02a77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 0x559110bf5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02b0bb4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f028800f3e0 con 0x559110be6100 2024-03-19T01:20:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02a77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 0x559110bf5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:51232/0 (socket says 172.21.15.116:51232) 2024-03-19T01:20:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02a77fe640 1 -- 172.21.15.116:0/1489599970 learned_addr learned my addr 172.21.15.116:0/1489599970 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110bf59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02b0bb4740 1 -- 172.21.15.116:0/1489599970 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f028800c620 con 0x559110bebc60 2024-03-19T01:20:41.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02b0bb4740 10 client.0 _ll_register_callbacks cb 0x7f029001a790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:20:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02a7fff640 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110bebc60 0x559110bf69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02a77fe640 1 -- 172.21.15.116:0/1489599970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110bebc60 msgr2=0x559110bf69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02a77fe640 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110bebc60 0x559110bf69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02a77fe640 1 -- 172.21.15.116:0/1489599970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 msgr2=0x559110bf59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02a77fe640 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110bf59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02a77fe640 1 -- 172.21.15.116:0/1489599970 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559110bf6fa0 con 0x559110be6100 2024-03-19T01:20:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110bf59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:20:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f02a77fe640 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 0x559110bf5460 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x559110be7350 tx=0x7f02a0016920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7e0f0a6a22e839d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.697+0000 7f0286ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:20:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.698+0000 7f0286ffd640 1 -- 172.21.15.116:0/1489599970 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a0023060 con 0x559110be6100 2024-03-19T01:20:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.698+0000 7f0286ffd640 1 -- 172.21.15.116:0/1489599970 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f02a001e430 con 0x559110be6100 2024-03-19T01:20:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.698+0000 7f02b0bb4740 1 -- 172.21.15.116:0/1489599970 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02980b8070 con 0x559110be6100 2024-03-19T01:20:41.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.698+0000 7f0286ffd640 1 -- 172.21.15.116:0/1489599970 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f02a001e630 con 0x559110be6100 2024-03-19T01:20:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.698+0000 7f02b0bb4740 10 client.25763 fetch_fsmap learned FSMap version 45 2024-03-19T01:20:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.698+0000 7f02b0bb4740 1 -- 172.21.15.116:0/1489599970 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x559110c62d00 con 0x559110be6100 2024-03-19T01:20:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.698+0000 7f0286ffd640 1 -- 172.21.15.116:0/1489599970 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02a001ecb0 con 0x559110be6100 2024-03-19T01:20:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.698+0000 7f0286ffd640 1 -- 172.21.15.116:0/1489599970 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f02a0029b50 con 0x559110be6100 2024-03-19T01:20:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.699+0000 7f02b0bb4740 10 client.25763 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:20:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.699+0000 7f02b0bb4740 10 client.25763 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:20:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.699+0000 7f02b0bb4740 20 client.25763 populate_metadata read hostname 'smithi116' 2024-03-19T01:20:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.699+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559110c63760 0x559110c65b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.699+0000 7f02b0bb4740 4 client.25763 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:20:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.699+0000 7f02b0bb4740 1 -- 172.21.15.116:0/1489599970 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f02980b8070 con 0x559110c63760 2024-03-19T01:20:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.699+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559110c63760 0x559110c65b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.699+0000 7f02a6ffd640 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559110c63760 0x559110c65b80 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb655a4d2884e261 server_cookie=a680119d2a0b265c in_seq=0 out_seq=0 2024-03-19T01:20:41.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.699+0000 7f0286ffd640 10 client.25763 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:41.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.706+0000 7f0286ffd640 1 -- 172.21.15.116:0/1489599970 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f02980b8070 con 0x559110c63760 2024-03-19T01:20:41.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.706+0000 7f0286ffd640 10 client.25763 handle_command_reply: tid=0 2024-03-19T01:20:41.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.706+0000 7f02b0bb4740 1 client.25763 shutdown 2024-03-19T01:20:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.707+0000 7f02b0bb4740 1 -- 172.21.15.116:0/1489599970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 msgr2=0x559110bf5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.707+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 0x559110bf5460 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x559110be7350 tx=0x7f02a0016920 comp rx=0 tx=0).stop 2024-03-19T01:20:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.707+0000 7f02b0bb4740 1 -- 172.21.15.116:0/1489599970 shutdown_connections 2024-03-19T01:20:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.707+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559110c63760 0x559110c65b80 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.707+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559110bebc60 0x559110bf69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.707+0000 7f0286ffd640 0 client.25763 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.707+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559110be6af0 0x559110bf59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.707+0000 7f02b0bb4740 1 --2- 172.21.15.116:0/1489599970 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559110be6100 0x559110bf5460 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.707+0000 7f02b0bb4740 1 -- 172.21.15.116:0/1489599970 >> 172.21.15.116:0/1489599970 conn(0x559110be1670 msgr2=0x559110be48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.707+0000 7f02b0bb4740 1 -- 172.21.15.116:0/1489599970 shutdown_connections 2024-03-19T01:20:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.707+0000 7f02b0bb4740 1 -- 172.21.15.116:0/1489599970 wait complete. 2024-03-19T01:20:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.707+0000 7f02b0bb4740 20 client.25763 trim_cache size 0 max 16384 2024-03-19T01:20:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:20:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:20:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:20:41.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:20:41.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.710+0000 7f02adfc6640 1 -- 172.21.15.116:0/2660843239 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02940a2ad0 msgr2=0x7f02940a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.710+0000 7f02adfc6640 1 --2- 172.21.15.116:0/2660843239 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02940a2ad0 0x7f02940a4f90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0290005ab0 tx=0x7f0290005fd0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.710+0000 7f02adfc6640 1 -- 172.21.15.116:0/2660843239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02a8101970 msgr2=0x7f02a8108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.710+0000 7f02adfc6640 1 --2- 172.21.15.116:0/2660843239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02a8101970 0x7f02a8108230 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f029800ba80 tx=0x7f029800bf40 comp rx=0 tx=0).stop 2024-03-19T01:20:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.710+0000 7f02adfc6640 1 -- 172.21.15.116:0/2660843239 shutdown_connections 2024-03-19T01:20:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.710+0000 7f02adfc6640 1 --2- 172.21.15.116:0/2660843239 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02a8102330 0x7f02a819ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.710+0000 7f02adfc6640 1 --2- 172.21.15.116:0/2660843239 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02940a2ad0 0x7f02940a4f90 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.710+0000 7f02adfc6640 1 --2- 172.21.15.116:0/2660843239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02a8101970 0x7f02a8108230 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.710+0000 7f02adfc6640 1 --2- 172.21.15.116:0/2660843239 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02a8100780 0x7f02a8107cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.710+0000 7f02adfc6640 1 -- 172.21.15.116:0/2660843239 >> 172.21.15.116:0/2660843239 conn(0x7f02a80fbcf0 msgr2=0x7f02a806b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.710+0000 7f02adfc6640 1 -- 172.21.15.116:0/2660843239 shutdown_connections 2024-03-19T01:20:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:41.710+0000 7f02adfc6640 1 -- 172.21.15.116:0/2660843239 wait complete. 2024-03-19T01:20:41.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:41 smithi156 ceph-mon[66799]: pgmap v594: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 6.7 MiB/s wr, 669 op/s 2024-03-19T01:20:43.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:43 smithi162 ceph-mon[67178]: pgmap v595: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.3 MiB/s wr, 546 op/s 2024-03-19T01:20:43.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:43 smithi116 ceph-mon[62701]: pgmap v595: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.3 MiB/s wr, 546 op/s 2024-03-19T01:20:43.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:43 smithi156 ceph-mon[66799]: pgmap v595: 97 pgs: 97 active+clean; 1.6 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.3 MiB/s wr, 546 op/s 2024-03-19T01:20:46.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:46 smithi162 ceph-mon[67178]: pgmap v596: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.7 MiB/s wr, 699 op/s 2024-03-19T01:20:46.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:46 smithi116 ceph-mon[62701]: pgmap v596: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.7 MiB/s wr, 699 op/s 2024-03-19T01:20:46.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:46 smithi156 ceph-mon[66799]: pgmap v596: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.7 MiB/s wr, 699 op/s 2024-03-19T01:20:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 480.0 s, 1815.4 tps, lat 54.886 ms stddev 65.936 2024-03-19T01:20:47.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:47 smithi162 ceph-mon[67178]: pgmap v597: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 607 op/s 2024-03-19T01:20:47.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:47 smithi116 ceph-mon[62701]: pgmap v597: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 607 op/s 2024-03-19T01:20:47.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:47 smithi156 ceph-mon[66799]: pgmap v597: 97 pgs: 97 active+clean; 1.6 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 MiB/s wr, 607 op/s 2024-03-19T01:20:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:49 smithi162 ceph-mon[67178]: pgmap v598: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.5 MiB/s wr, 610 op/s 2024-03-19T01:20:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:49 smithi116 ceph-mon[62701]: pgmap v598: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.5 MiB/s wr, 610 op/s 2024-03-19T01:20:49.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:49 smithi156 ceph-mon[66799]: pgmap v598: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.5 MiB/s wr, 610 op/s 2024-03-19T01:20:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 485.0 s, 1959.8 tps, lat 50.816 ms stddev 56.507 2024-03-19T01:20:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:51 smithi162 ceph-mon[67178]: pgmap v599: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.6 MiB/s wr, 692 op/s 2024-03-19T01:20:51.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:51 smithi116 ceph-mon[62701]: pgmap v599: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.6 MiB/s wr, 692 op/s 2024-03-19T01:20:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:51 smithi156 ceph-mon[66799]: pgmap v599: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.6 MiB/s wr, 692 op/s 2024-03-19T01:20:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:53 smithi162 ceph-mon[67178]: pgmap v600: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.0 MiB/s wr, 538 op/s 2024-03-19T01:20:53.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:53 smithi116 ceph-mon[62701]: pgmap v600: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.0 MiB/s wr, 538 op/s 2024-03-19T01:20:53.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:53 smithi156 ceph-mon[66799]: pgmap v600: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.0 MiB/s wr, 538 op/s 2024-03-19T01:20:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.849+0000 7f970de9a640 1 -- 172.21.15.116:0/2613464106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97081022c0 msgr2=0x7f9708108b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.849+0000 7f970de9a640 1 --2- 172.21.15.116:0/2613464106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97081022c0 0x7f9708108b50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9700009ca0 tx=0x7f970002c100 comp rx=0 tx=0).stop 2024-03-19T01:20:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.849+0000 7f970de9a640 1 -- 172.21.15.116:0/2613464106 shutdown_connections 2024-03-19T01:20:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.849+0000 7f970de9a640 1 --2- 172.21.15.116:0/2613464106 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97081022c0 0x7f9708108b50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.849+0000 7f970de9a640 1 --2- 172.21.15.116:0/2613464106 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9708101900 0x7f9708101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.849+0000 7f970de9a640 1 --2- 172.21.15.116:0/2613464106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9708100710 0x7f9708100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.849+0000 7f970de9a640 1 -- 172.21.15.116:0/2613464106 >> 172.21.15.116:0/2613464106 conn(0x7f97080fbc80 msgr2=0x7f97080fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.849+0000 7f970de9a640 1 -- 172.21.15.116:0/2613464106 shutdown_connections 2024-03-19T01:20:54.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.849+0000 7f970de9a640 1 -- 172.21.15.116:0/2613464106 wait complete. 2024-03-19T01:20:54.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.850+0000 7f970de9a640 1 Processor -- start 2024-03-19T01:20:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.850+0000 7f970de9a640 1 -- start start 2024-03-19T01:20:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.850+0000 7f970de9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9708100710 0x7f9708069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.850+0000 7f970de9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9708101900 0x7f970819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.850+0000 7f970de9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97081022c0 0x7f970819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.850+0000 7f970de9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f970810c370 con 0x7f9708100710 2024-03-19T01:20:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.850+0000 7f970de9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f970810c1f0 con 0x7f97081022c0 2024-03-19T01:20:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.850+0000 7f970de9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f970810c4f0 con 0x7f9708101900 2024-03-19T01:20:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.851+0000 7f9707fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97081022c0 0x7f970819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.851+0000 7f97077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9708100710 0x7f9708069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.851+0000 7f9707fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97081022c0 0x7f970819ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:51966/0 (socket says 172.21.15.116:51966) 2024-03-19T01:20:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.851+0000 7f9707fff640 1 -- 172.21.15.116:0/2798667806 learned_addr learned my addr 172.21.15.116:0/2798667806 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.851+0000 7f9706ffd640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9708101900 0x7f970819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.851+0000 7f9706ffd640 1 -- 172.21.15.116:0/2798667806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97081022c0 msgr2=0x7f970819ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:54.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.851+0000 7f9706ffd640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97081022c0 0x7f970819ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.851+0000 7f9706ffd640 1 -- 172.21.15.116:0/2798667806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9708100710 msgr2=0x7f9708069a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:54.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.851+0000 7f9706ffd640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9708100710 0x7f9708069a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.851+0000 7f9706ffd640 1 -- 172.21.15.116:0/2798667806 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9708196590 con 0x7f9708101900 2024-03-19T01:20:54.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.852+0000 7f9707fff640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97081022c0 0x7f970819ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:20:54.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.852+0000 7f97077fe640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9708100710 0x7f9708069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:20:54.854 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.852+0000 7f9706ffd640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9708101900 0x7f970819c740 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00d6d0 tx=0x7f96fc00db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.853+0000 7f9704ff9640 1 -- 172.21.15.116:0/2798667806 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc017070 con 0x7f9708101900 2024-03-19T01:20:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.853+0000 7f9704ff9640 1 -- 172.21.15.116:0/2798667806 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96fc009820 con 0x7f9708101900 2024-03-19T01:20:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.854+0000 7f970de9a640 1 -- 172.21.15.116:0/2798667806 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9708196820 con 0x7f9708101900 2024-03-19T01:20:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.854+0000 7f9704ff9640 1 -- 172.21.15.116:0/2798667806 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc004d00 con 0x7f9708101900 2024-03-19T01:20:54.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.854+0000 7f970de9a640 1 -- 172.21.15.116:0/2798667806 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9708196d70 con 0x7f9708101900 2024-03-19T01:20:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.857+0000 7f9704ff9640 1 -- 172.21.15.116:0/2798667806 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f96fc0099d0 con 0x7f9708101900 2024-03-19T01:20:54.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.857+0000 7f9710b66740 1 Processor -- start 2024-03-19T01:20:54.858 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.857+0000 7f9710b66740 1 -- start start 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.857+0000 7f9710b66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106100 0x55f665106520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.857+0000 7f9710b66740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106af0 0x55f66510b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.857+0000 7f9710b66740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 0x55f66510e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.857+0000 7f9710b66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f66506abe0 con 0x55f665106100 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.857+0000 7f9710b66740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f665013960 con 0x55f66510bc60 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.857+0000 7f9710b66740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f665066c80 con 0x55f665106af0 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.858+0000 7f97077fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106af0 0x55f66510b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.858+0000 7f9704ff9640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f96f00a2a10 0x7f96f00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.858+0000 7f97077fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106af0 0x55f66510b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58756/0 (socket says 172.21.15.116:58756) 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.858+0000 7f97077fe640 1 -- 172.21.15.116:0/3757668388 learned_addr learned my addr 172.21.15.116:0/3757668388 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.858+0000 7f9707fff640 1 --2- 172.21.15.116:0/3757668388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106100 0x55f665106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.858+0000 7f9704ff9640 1 -- 172.21.15.116:0/2798667806 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f96fc0ce480 con 0x7f9708101900 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.858+0000 7f9706ffd640 1 --2- 172.21.15.116:0/3757668388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 0x55f66510e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.860 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.859+0000 7f97077fe640 1 -- 172.21.15.116:0/3757668388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 msgr2=0x55f66510e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.859+0000 7f97077fe640 1 --2- 172.21.15.116:0/3757668388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 0x55f66510e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.859+0000 7f97077fe640 1 -- 172.21.15.116:0/3757668388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106100 msgr2=0x55f665106520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.859+0000 7f97077fe640 1 --2- 172.21.15.116:0/3757668388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106100 0x55f665106520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.859+0000 7f97077fe640 1 -- 172.21.15.116:0/3757668388 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f66500ad20 con 0x55f665106af0 2024-03-19T01:20:54.861 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.859+0000 7f97077fe640 1 --2- 172.21.15.116:0/3757668388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106af0 0x55f66510b5f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f96f80114c0 tx=0x7f96f8011980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21b199e29f1608d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.860+0000 7f96edffb640 1 -- 172.21.15.116:0/3757668388 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f8004020 con 0x55f665106af0 2024-03-19T01:20:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.860+0000 7f9707fff640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f96f00a2a10 0x7f96f00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.860+0000 7f96edffb640 1 -- 172.21.15.116:0/3757668388 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96f8005500 con 0x55f665106af0 2024-03-19T01:20:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.861+0000 7f96edffb640 1 -- 172.21.15.116:0/3757668388 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f80035f0 con 0x55f665106af0 2024-03-19T01:20:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.861+0000 7f9710b66740 1 -- 172.21.15.116:0/3757668388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106af0 msgr2=0x55f66510b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:54.862 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.861+0000 7f9710b66740 1 --2- 172.21.15.116:0/3757668388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106af0 0x55f66510b5f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f96f80114c0 tx=0x7f96f8011980 comp rx=0 tx=0).stop 2024-03-19T01:20:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.861+0000 7f9707fff640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f96f00a2a10 0x7f96f00a4ed0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x55f665107350 tx=0x7f970002e040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.861+0000 7f9710b66740 1 -- 172.21.15.116:0/3757668388 shutdown_connections 2024-03-19T01:20:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.861+0000 7f9710b66740 1 --2- 172.21.15.116:0/3757668388 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 0x55f66510e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.861+0000 7f9710b66740 1 --2- 172.21.15.116:0/3757668388 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106af0 0x55f66510b5f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.861+0000 7f9710b66740 1 --2- 172.21.15.116:0/3757668388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106100 0x55f665106520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.861+0000 7f9710b66740 1 -- 172.21.15.116:0/3757668388 >> 172.21.15.116:0/3757668388 conn(0x55f665101670 msgr2=0x55f665101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.861+0000 7f9710b66740 1 -- 172.21.15.116:0/3757668388 shutdown_connections 2024-03-19T01:20:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.861+0000 7f9710b66740 1 -- 172.21.15.116:0/3757668388 wait complete. 2024-03-19T01:20:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.861+0000 7f9710b66740 1 -- start start 2024-03-19T01:20:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.862+0000 7f9710b66740 1 Processor -- start 2024-03-19T01:20:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9710b66740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f665110180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9710b66740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f665106af0 0x55f6651106c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9710b66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f66510bc60 0x55f665117700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9710b66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f66510ec60 con 0x55f66510bc60 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9710b66740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f665026db0 con 0x55f665106af0 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9710b66740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f66510ede0 con 0x55f665106100 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9710b66740 10 client.0 _ll_register_callbacks cb 0x55f66502be70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f97077fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f665110180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9706ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f665106af0 0x55f6651106c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f97077fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f665110180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58772/0 (socket says 172.21.15.116:58772) 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f97077fe640 1 -- 172.21.15.116:0/117532342 learned_addr learned my addr 172.21.15.116:0/117532342 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9707fff640 1 --2- 172.21.15.116:0/117532342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f66510bc60 0x55f665117700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9706ffd640 1 -- 172.21.15.116:0/117532342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 msgr2=0x55f665110180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9706ffd640 1 --2- 172.21.15.116:0/117532342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f665110180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9706ffd640 1 -- 172.21.15.116:0/117532342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f66510bc60 msgr2=0x55f665117700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9706ffd640 1 --2- 172.21.15.116:0/117532342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f66510bc60 0x55f665117700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.863+0000 7f9706ffd640 1 -- 172.21.15.116:0/117532342 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f665025c60 con 0x55f665106af0 2024-03-19T01:20:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.864+0000 7f9706ffd640 1 --2- 172.21.15.116:0/117532342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f665106af0 0x55f6651106c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0cbfb0 tx=0x7f96fc09aff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3926a2f74a907ff7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.864+0000 7f96e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:20:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.864+0000 7f96e6ffd640 1 -- 172.21.15.116:0/117532342 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc020040 con 0x55f665106af0 2024-03-19T01:20:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.864+0000 7f96e6ffd640 1 -- 172.21.15.116:0/117532342 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f96fc097520 con 0x55f665106af0 2024-03-19T01:20:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.864+0000 7f9710b66740 1 -- 172.21.15.116:0/117532342 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96f8005500 con 0x55f665106af0 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.864+0000 7f96e6ffd640 1 -- 172.21.15.116:0/117532342 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f96fc099270 con 0x55f665106af0 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.864+0000 7f9710b66740 10 client.25775 fetch_fsmap learned FSMap version 45 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.864+0000 7f9710b66740 1 -- 172.21.15.116:0/117532342 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55f664d7f210 con 0x55f665106af0 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.864+0000 7f96e6ffd640 1 -- 172.21.15.116:0/117532342 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96fc0ce0b0 con 0x55f665106af0 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.864+0000 7f96e6ffd640 1 -- 172.21.15.116:0/117532342 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f96fc099d70 con 0x55f665106af0 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.865+0000 7f9710b66740 10 client.25775 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.865+0000 7f9710b66740 10 client.25775 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.865+0000 7f9710b66740 20 client.25775 populate_metadata read hostname 'smithi116' 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.865+0000 7f9710b66740 1 --2- 172.21.15.116:0/117532342 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f66511bf80 0x55f66511e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.865+0000 7f9710b66740 4 client.25775 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.865+0000 7f9710b66740 1 -- 172.21.15.116:0/117532342 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f96f8005500 con 0x55f66511bf80 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.865+0000 7f97077fe640 1 --2- 172.21.15.116:0/117532342 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f66511bf80 0x55f66511e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.866+0000 7f97077fe640 1 --2- 172.21.15.116:0/117532342 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f66511bf80 0x55f66511e3b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14819c4f4074342b server_cookie=a64d169d04dd9828 in_seq=0 out_seq=0 2024-03-19T01:20:54.867 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.866+0000 7f96e6ffd640 10 client.25775 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f96e6ffd640 1 -- 172.21.15.116:0/117532342 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f96f8005500 con 0x55f66511bf80 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f96e6ffd640 10 client.25775 handle_command_reply: tid=0 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f9710b66740 1 client.25775 shutdown 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f9710b66740 1 -- 172.21.15.116:0/117532342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f665106af0 msgr2=0x55f6651106c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f9710b66740 1 --2- 172.21.15.116:0/117532342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f665106af0 0x55f6651106c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0cbfb0 tx=0x7f96fc09aff0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f9710b66740 1 -- 172.21.15.116:0/117532342 shutdown_connections 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f9710b66740 1 --2- 172.21.15.116:0/117532342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f66510bc60 0x55f665117700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f9710b66740 1 --2- 172.21.15.116:0/117532342 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f665106af0 0x55f6651106c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f9710b66740 1 --2- 172.21.15.116:0/117532342 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f66511bf80 0x55f66511e3b0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f9710b66740 1 --2- 172.21.15.116:0/117532342 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f665110180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f9710b66740 1 -- 172.21.15.116:0/117532342 >> 172.21.15.116:0/117532342 conn(0x55f665101670 msgr2=0x55f6651028c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f96e6ffd640 0 client.25775 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f9710b66740 1 -- 172.21.15.116:0/117532342 shutdown_connections 2024-03-19T01:20:54.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f9710b66740 1 -- 172.21.15.116:0/117532342 wait complete. 2024-03-19T01:20:54.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.867+0000 7f9710b66740 20 client.25775 trim_cache size 0 max 16384 2024-03-19T01:20:54.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.888+0000 7f9710b66740 1 Processor -- start 2024-03-19T01:20:54.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.894+0000 7f9710b66740 1 -- start start 2024-03-19T01:20:54.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.894+0000 7f9710b66740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f665100ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.894+0000 7f9710b66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106af0 0x55f6651176c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.895+0000 7f9710b66740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 0x55f665104fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.895+0000 7f9710b66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f97081a6a20 con 0x55f665106af0 2024-03-19T01:20:54.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.895+0000 7f9710b66740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f97081a8560 con 0x55f66510bc60 2024-03-19T01:20:54.895 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.895+0000 7f9710b66740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f97081a70f0 con 0x55f665106100 2024-03-19T01:20:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.897+0000 7f9706ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106af0 0x55f6651176c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.897+0000 7f9706ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106af0 0x55f6651176c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40374/0 (socket says 172.21.15.116:40374) 2024-03-19T01:20:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.897+0000 7f9706ffd640 1 -- 172.21.15.116:0/1265283665 learned_addr learned my addr 172.21.15.116:0/1265283665 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.898+0000 7f9707fff640 1 --2- 172.21.15.116:0/1265283665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 0x55f665104fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.899 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.898+0000 7f97077fe640 1 --2- 172.21.15.116:0/1265283665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f665100ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.906+0000 7f9706ffd640 1 -- 172.21.15.116:0/1265283665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 msgr2=0x55f665100ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:54.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.906+0000 7f9706ffd640 1 --2- 172.21.15.116:0/1265283665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f665100ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.906+0000 7f9706ffd640 1 -- 172.21.15.116:0/1265283665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 msgr2=0x55f665104fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:54.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.906+0000 7f9706ffd640 1 --2- 172.21.15.116:0/1265283665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 0x55f665104fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.906+0000 7f9706ffd640 1 -- 172.21.15.116:0/1265283665 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f664e4b620 con 0x55f665106af0 2024-03-19T01:20:54.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.912+0000 7f9706ffd640 1 --2- 172.21.15.116:0/1265283665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106af0 0x55f6651176c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f96fc097880 tx=0x7f96fc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa27281c58dc0078 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:54.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.918+0000 7f96ed7fa640 1 -- 172.21.15.116:0/1265283665 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc020070 con 0x55f665106af0 2024-03-19T01:20:54.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.918+0000 7f96ed7fa640 1 -- 172.21.15.116:0/1265283665 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96fc099d70 con 0x55f665106af0 2024-03-19T01:20:54.918 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.918+0000 7f96ed7fa640 1 -- 172.21.15.116:0/1265283665 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc09a300 con 0x55f665106af0 2024-03-19T01:20:54.919 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.919+0000 7f9710b66740 1 -- 172.21.15.116:0/1265283665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106af0 msgr2=0x55f6651176c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:54.920 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.919+0000 7f9710b66740 1 --2- 172.21.15.116:0/1265283665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106af0 0x55f6651176c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f96fc097880 tx=0x7f96fc009f90 comp rx=0 tx=0).stop 2024-03-19T01:20:54.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.925+0000 7f9710b66740 1 -- 172.21.15.116:0/1265283665 shutdown_connections 2024-03-19T01:20:54.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.925+0000 7f9710b66740 1 --2- 172.21.15.116:0/1265283665 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 0x55f665104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.925+0000 7f9710b66740 1 --2- 172.21.15.116:0/1265283665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106af0 0x55f6651176c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.925+0000 7f9710b66740 1 --2- 172.21.15.116:0/1265283665 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f665100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.925 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.925+0000 7f9710b66740 1 -- 172.21.15.116:0/1265283665 >> 172.21.15.116:0/1265283665 conn(0x55f665101670 msgr2=0x55f6651048d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:54.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.926+0000 7f9710b66740 1 -- 172.21.15.116:0/1265283665 shutdown_connections 2024-03-19T01:20:54.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.926+0000 7f9710b66740 1 -- 172.21.15.116:0/1265283665 wait complete. 2024-03-19T01:20:54.926 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.926+0000 7f9710b66740 1 -- start start 2024-03-19T01:20:54.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.927+0000 7f9710b66740 1 Processor -- start 2024-03-19T01:20:54.927 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.927+0000 7f9710b66740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f6651150a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.927+0000 7f9710b66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106af0 0x55f6651155e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.928+0000 7f97077fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f6651150a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.928+0000 7f97077fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f6651150a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:58778/0 (socket says 172.21.15.116:58778) 2024-03-19T01:20:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.928+0000 7f9706ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106af0 0x55f6651155e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.928+0000 7f9706ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106af0 0x55f6651155e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40376/0 (socket says 172.21.15.116:40376) 2024-03-19T01:20:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.928+0000 7f97077fe640 1 -- 172.21.15.116:0/3826736365 learned_addr learned my addr 172.21.15.116:0/3826736365 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:20:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.928+0000 7f9710b66740 1 --2- 172.21.15.116:0/3826736365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 0x55f6651165f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.928+0000 7f9710b66740 1 -- 172.21.15.116:0/3826736365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f97081a7e90 con 0x55f665106af0 2024-03-19T01:20:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.928+0000 7f9710b66740 1 -- 172.21.15.116:0/3826736365 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f97081a77c0 con 0x55f66510bc60 2024-03-19T01:20:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.928+0000 7f9710b66740 1 -- 172.21.15.116:0/3826736365 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f97081a4a00 con 0x55f665106100 2024-03-19T01:20:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.928+0000 7f9710b66740 10 client.0 _ll_register_callbacks cb 0x7f96fc005810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:20:54.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.928+0000 7f9707fff640 1 --2- 172.21.15.116:0/3826736365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 0x55f6651165f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.929+0000 7f97077fe640 1 -- 172.21.15.116:0/3826736365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 msgr2=0x55f6651165f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.929+0000 7f97077fe640 1 --2- 172.21.15.116:0/3826736365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 0x55f6651165f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.929+0000 7f97077fe640 1 -- 172.21.15.116:0/3826736365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106af0 msgr2=0x55f6651155e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:20:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.929+0000 7f97077fe640 1 --2- 172.21.15.116:0/3826736365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106af0 0x55f6651155e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.930 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.929+0000 7f97077fe640 1 -- 172.21.15.116:0/3826736365 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f665116be0 con 0x55f665106100 2024-03-19T01:20:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.929+0000 7f97077fe640 1 --2- 172.21.15.116:0/3826736365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f6651150a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f96f8009250 tx=0x7f96f8005b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12bbb1d0f63fae48 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:20:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.929+0000 7f96e6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:20:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.929+0000 7f96e6ffd640 1 -- 172.21.15.116:0/3826736365 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f801d040 con 0x55f665106100 2024-03-19T01:20:54.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.929+0000 7f96e6ffd640 1 -- 172.21.15.116:0/3826736365 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f96f8009b50 con 0x55f665106100 2024-03-19T01:20:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.929+0000 7f9710b66740 1 -- 172.21.15.116:0/3826736365 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96fc099d70 con 0x55f665106100 2024-03-19T01:20:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.930+0000 7f96e6ffd640 1 -- 172.21.15.116:0/3826736365 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f96f800f5c0 con 0x55f665106100 2024-03-19T01:20:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.930+0000 7f9710b66740 10 client.25771 fetch_fsmap learned FSMap version 45 2024-03-19T01:20:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.930+0000 7f9710b66740 1 -- 172.21.15.116:0/3826736365 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55f6651825f0 con 0x55f665106100 2024-03-19T01:20:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.930+0000 7f96e6ffd640 1 -- 172.21.15.116:0/3826736365 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96fc099d70 con 0x55f665106100 2024-03-19T01:20:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.930+0000 7f96e6ffd640 1 -- 172.21.15.116:0/3826736365 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f96f8033920 con 0x55f665106100 2024-03-19T01:20:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.931+0000 7f9710b66740 10 client.25771 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:20:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.931+0000 7f9710b66740 10 client.25771 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:20:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.931+0000 7f9710b66740 20 client.25771 populate_metadata read hostname 'smithi116' 2024-03-19T01:20:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.931+0000 7f9710b66740 1 --2- 172.21.15.116:0/3826736365 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f665182e60 0x55f665185280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:20:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.931+0000 7f9710b66740 4 client.25771 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:20:54.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.931+0000 7f9710b66740 1 -- 172.21.15.116:0/3826736365 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f665078120 con 0x55f665182e60 2024-03-19T01:20:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.931+0000 7f9706ffd640 1 --2- 172.21.15.116:0/3826736365 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f665182e60 0x55f665185280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:20:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.932+0000 7f9706ffd640 1 --2- 172.21.15.116:0/3826736365 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f665182e60 0x55f665185280 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=265644be3a1dc38b server_cookie=4d4bce3822719a12 in_seq=0 out_seq=0 2024-03-19T01:20:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.932+0000 7f96e6ffd640 10 client.25771 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.939+0000 7f96e6ffd640 1 -- 172.21.15.116:0/3826736365 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f665078120 con 0x55f665182e60 2024-03-19T01:20:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.939+0000 7f96e6ffd640 10 client.25771 handle_command_reply: tid=0 2024-03-19T01:20:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.939+0000 7f9710b66740 1 client.25771 shutdown 2024-03-19T01:20:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.939+0000 7f9710b66740 1 -- 172.21.15.116:0/3826736365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 msgr2=0x55f6651150a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.939+0000 7f9710b66740 1 --2- 172.21.15.116:0/3826736365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f6651150a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f96f8009250 tx=0x7f96f8005b40 comp rx=0 tx=0).stop 2024-03-19T01:20:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.939+0000 7f9710b66740 1 -- 172.21.15.116:0/3826736365 shutdown_connections 2024-03-19T01:20:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.939+0000 7f9710b66740 1 --2- 172.21.15.116:0/3826736365 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f66510bc60 0x55f6651165f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.939+0000 7f9710b66740 1 --2- 172.21.15.116:0/3826736365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f665106af0 0x55f6651155e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.939+0000 7f9710b66740 1 --2- 172.21.15.116:0/3826736365 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f665182e60 0x55f665185280 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.939+0000 7f9710b66740 1 --2- 172.21.15.116:0/3826736365 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f665106100 0x55f6651150a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.939+0000 7f9710b66740 1 -- 172.21.15.116:0/3826736365 >> 172.21.15.116:0/3826736365 conn(0x55f665101670 msgr2=0x55f665105160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:54.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.940+0000 7f96e6ffd640 0 client.25771 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:20:54.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.940+0000 7f9710b66740 1 -- 172.21.15.116:0/3826736365 shutdown_connections 2024-03-19T01:20:54.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.940+0000 7f9710b66740 1 -- 172.21.15.116:0/3826736365 wait complete. 2024-03-19T01:20:54.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.941+0000 7f9710b66740 20 client.25771 trim_cache size 0 max 16384 2024-03-19T01:20:54.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:20:54.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:20:54.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:20:54.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:20:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.942+0000 7f970de9a640 1 -- 172.21.15.116:0/2798667806 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f96f00a2a10 msgr2=0x7f96f00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.942+0000 7f970de9a640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f96f00a2a10 0x7f96f00a4ed0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x55f665107350 tx=0x7f970002e040 comp rx=0 tx=0).stop 2024-03-19T01:20:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.942+0000 7f970de9a640 1 -- 172.21.15.116:0/2798667806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9708101900 msgr2=0x7f970819c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:20:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.942+0000 7f970de9a640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9708101900 0x7f970819c740 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f96fc00d6d0 tx=0x7f96fc00db90 comp rx=0 tx=0).stop 2024-03-19T01:20:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.942+0000 7f970de9a640 1 -- 172.21.15.116:0/2798667806 shutdown_connections 2024-03-19T01:20:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.942+0000 7f970de9a640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f96f00a2a10 0x7f96f00a4ed0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.942+0000 7f970de9a640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97081022c0 0x7f970819ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.942+0000 7f970de9a640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9708101900 0x7f970819c740 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.942+0000 7f970de9a640 1 --2- 172.21.15.116:0/2798667806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9708100710 0x7f9708069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:20:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.942+0000 7f970de9a640 1 -- 172.21.15.116:0/2798667806 >> 172.21.15.116:0/2798667806 conn(0x7f97080fbc80 msgr2=0x7f9708104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:20:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.942+0000 7f970de9a640 1 -- 172.21.15.116:0/2798667806 shutdown_connections 2024-03-19T01:20:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:20:54.942+0000 7f970de9a640 1 -- 172.21.15.116:0/2798667806 wait complete. 2024-03-19T01:20:55.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:55 smithi162 ceph-mon[67178]: pgmap v601: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.1 MiB/s wr, 671 op/s 2024-03-19T01:20:55.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:55 smithi116 ceph-mon[62701]: pgmap v601: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.1 MiB/s wr, 671 op/s 2024-03-19T01:20:55.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:55 smithi156 ceph-mon[66799]: pgmap v601: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.1 MiB/s wr, 671 op/s 2024-03-19T01:20:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 490.0 s, 2469.8 tps, lat 40.933 ms stddev 48.521 2024-03-19T01:20:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:57 smithi162 ceph-mon[67178]: pgmap v602: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.5 MiB/s wr, 602 op/s 2024-03-19T01:20:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:57 smithi116 ceph-mon[62701]: pgmap v602: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.5 MiB/s wr, 602 op/s 2024-03-19T01:20:57.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:57 smithi156 ceph-mon[66799]: pgmap v602: 97 pgs: 97 active+clean; 1.6 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 5.5 MiB/s wr, 602 op/s 2024-03-19T01:20:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:20:59 smithi162 ceph-mon[67178]: pgmap v603: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.8 MiB/s wr, 695 op/s 2024-03-19T01:20:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:20:59 smithi116 ceph-mon[62701]: pgmap v603: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.8 MiB/s wr, 695 op/s 2024-03-19T01:20:59.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:20:59 smithi156 ceph-mon[66799]: pgmap v603: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.8 MiB/s wr, 695 op/s 2024-03-19T01:21:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 495.0 s, 2426.4 tps, lat 40.742 ms stddev 48.474 2024-03-19T01:21:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:01 smithi162 ceph-mon[67178]: pgmap v604: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.5 MiB/s wr, 746 op/s 2024-03-19T01:21:01.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:01 smithi116 ceph-mon[62701]: pgmap v604: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.5 MiB/s wr, 746 op/s 2024-03-19T01:21:01.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:01 smithi156 ceph-mon[66799]: pgmap v604: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.5 MiB/s wr, 746 op/s 2024-03-19T01:21:03.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:03 smithi162 ceph-mon[67178]: pgmap v605: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.7 MiB/s wr, 579 op/s 2024-03-19T01:21:03.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:03 smithi116 ceph-mon[62701]: pgmap v605: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.7 MiB/s wr, 579 op/s 2024-03-19T01:21:03.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:03 smithi156 ceph-mon[66799]: pgmap v605: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.7 MiB/s wr, 579 op/s 2024-03-19T01:21:05.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:04 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:21:05.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:04 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:21:05.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:04 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:21:06.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:05 smithi162 ceph-mon[67178]: pgmap v606: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.8 MiB/s wr, 778 op/s 2024-03-19T01:21:06.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:21:06.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:21:06.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:05 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:21:06.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:05 smithi116 ceph-mon[62701]: pgmap v606: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.8 MiB/s wr, 778 op/s 2024-03-19T01:21:06.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:21:06.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:21:06.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:05 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:21:06.223 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:05 smithi156 ceph-mon[66799]: pgmap v606: 97 pgs: 97 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 7.8 MiB/s wr, 778 op/s 2024-03-19T01:21:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:21:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:21:06.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:05 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:21:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 500.0 s, 2489.2 tps, lat 40.563 ms stddev 44.448 2024-03-19T01:21:07.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:07 smithi162 ceph-mon[67178]: pgmap v607: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.2 MiB/s wr, 704 op/s 2024-03-19T01:21:07.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:07 smithi116 ceph-mon[62701]: pgmap v607: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.2 MiB/s wr, 704 op/s 2024-03-19T01:21:07.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:07 smithi156 ceph-mon[66799]: pgmap v607: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.2 MiB/s wr, 704 op/s 2024-03-19T01:21:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.061+0000 7f861c4dc640 1 -- 172.21.15.116:0/1000150635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8614103490 msgr2=0x7f8614109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.061+0000 7f861c4dc640 1 --2- 172.21.15.116:0/1000150635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8614103490 0x7f8614109d20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f860c009be0 tx=0x7f860c02bef0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.061+0000 7f861c4dc640 1 -- 172.21.15.116:0/1000150635 shutdown_connections 2024-03-19T01:21:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.061+0000 7f861c4dc640 1 --2- 172.21.15.116:0/1000150635 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8614103490 0x7f8614109d20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.061+0000 7f861c4dc640 1 --2- 172.21.15.116:0/1000150635 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8614102ad0 0x7f8614102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.061+0000 7f861c4dc640 1 --2- 172.21.15.116:0/1000150635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86140ffb00 0x7f86140fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.061+0000 7f861c4dc640 1 -- 172.21.15.116:0/1000150635 >> 172.21.15.116:0/1000150635 conn(0x7f86140fb470 msgr2=0x7f86140fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.061+0000 7f861c4dc640 1 -- 172.21.15.116:0/1000150635 shutdown_connections 2024-03-19T01:21:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.061+0000 7f861c4dc640 1 -- 172.21.15.116:0/1000150635 wait complete. 2024-03-19T01:21:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861c4dc640 1 Processor -- start 2024-03-19T01:21:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861c4dc640 1 -- start start 2024-03-19T01:21:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861c4dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86140ffb00 0x7f8614197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861c4dc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8614102ad0 0x7f8614198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861c4dc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8614103490 0x7f861419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861c4dc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f861410d4e0 con 0x7f86140ffb00 2024-03-19T01:21:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861c4dc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f861410d360 con 0x7f8614102ad0 2024-03-19T01:21:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861c4dc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f861410d660 con 0x7f8614103490 2024-03-19T01:21:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f8619a77640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8614102ad0 0x7f8614198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861aa79640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8614103490 0x7f861419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f8619a77640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8614102ad0 0x7f8614198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:47660/0 (socket says 172.21.15.116:47660) 2024-03-19T01:21:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f8619a77640 1 -- 172.21.15.116:0/2248709070 learned_addr learned my addr 172.21.15.116:0/2248709070 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861a278640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86140ffb00 0x7f8614197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861a278640 1 -- 172.21.15.116:0/2248709070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8614103490 msgr2=0x7f861419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861a278640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8614103490 0x7f861419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861a278640 1 -- 172.21.15.116:0/2248709070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8614102ad0 msgr2=0x7f8614198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861a278640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8614102ad0 0x7f8614198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861a278640 1 -- 172.21.15.116:0/2248709070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861419f690 con 0x7f86140ffb00 2024-03-19T01:21:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.062+0000 7f861aa79640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8614103490 0x7f861419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:21:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.063+0000 7f861a278640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86140ffb00 0x7f8614197ac0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8614068ca0 tx=0x7f861000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.063+0000 7f860b7fe640 1 -- 172.21.15.116:0/2248709070 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8610004440 con 0x7f86140ffb00 2024-03-19T01:21:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.063+0000 7f860b7fe640 1 -- 172.21.15.116:0/2248709070 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f861000cda0 con 0x7f86140ffb00 2024-03-19T01:21:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.063+0000 7f861c4dc640 1 -- 172.21.15.116:0/2248709070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f861419f980 con 0x7f86140ffb00 2024-03-19T01:21:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.063+0000 7f860b7fe640 1 -- 172.21.15.116:0/2248709070 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86100196e0 con 0x7f86140ffb00 2024-03-19T01:21:08.066 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.063+0000 7f861c4dc640 1 -- 172.21.15.116:0/2248709070 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f861419fed0 con 0x7f86140ffb00 2024-03-19T01:21:08.067 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.067+0000 7f861f1a4740 1 Processor -- start 2024-03-19T01:21:08.068 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.067+0000 7f861f1a4740 1 -- start start 2024-03-19T01:21:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861f1a4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80188520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861f1a4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 0x559e8018d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861f1a4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 0x559e801900f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861f1a4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559e800ecbe0 con 0x559e80188100 2024-03-19T01:21:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861f1a4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559e80095960 con 0x559e8018dc60 2024-03-19T01:21:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861f1a4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559e800e8c80 con 0x559e80188af0 2024-03-19T01:21:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861aa79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80188520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861aa79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80188520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55828/0 (socket says 172.21.15.116:55828) 2024-03-19T01:21:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861aa79640 1 -- 172.21.15.116:0/2558080781 learned_addr learned my addr 172.21.15.116:0/2558080781 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f8619a77640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 0x559e801900f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f860b7fe640 1 -- 172.21.15.116:0/2248709070 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f8610003780 con 0x7f86140ffb00 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861aa79640 1 -- 172.21.15.116:0/2558080781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 msgr2=0x559e8018d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861aa79640 1 --2- 172.21.15.116:0/2558080781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 0x559e8018d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861aa79640 1 -- 172.21.15.116:0/2558080781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 msgr2=0x559e801900f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861aa79640 1 --2- 172.21.15.116:0/2558080781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 0x559e801900f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861aa79640 1 -- 172.21.15.116:0/2558080781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e8008cd20 con 0x559e80188100 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f8619a77640 1 --2- 172.21.15.116:0/2558080781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 0x559e801900f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.068+0000 7f861aa79640 1 --2- 172.21.15.116:0/2558080781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80188520 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f860c009be0 tx=0x7f860c005ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8471300ab44f95ee server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.069+0000 7f8608bfe640 1 -- 172.21.15.116:0/2558080781 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c043030 con 0x559e80188100 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.069+0000 7f8608bfe640 1 -- 172.21.15.116:0/2558080781 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f860c003970 con 0x559e80188100 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.069+0000 7f860b7fe640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f86000a28a0 0x7f86000a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.069+0000 7f8608bfe640 1 -- 172.21.15.116:0/2558080781 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c02cd30 con 0x559e80188100 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.069+0000 7f861a278640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f86000a28a0 0x7f86000a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.069+0000 7f860b7fe640 1 -- 172.21.15.116:0/2248709070 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f86100cee40 con 0x7f86140ffb00 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.070+0000 7f861f1a4740 1 -- 172.21.15.116:0/2558080781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 msgr2=0x559e80188520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.070+0000 7f861a278640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f86000a28a0 0x7f86000a4d60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f86100cc970 tx=0x7f861006b2e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.070+0000 7f861f1a4740 1 --2- 172.21.15.116:0/2558080781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80188520 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f860c009be0 tx=0x7f860c005ec0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.070+0000 7f861f1a4740 1 -- 172.21.15.116:0/2558080781 shutdown_connections 2024-03-19T01:21:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.070+0000 7f861f1a4740 1 --2- 172.21.15.116:0/2558080781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 0x559e801900f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.070+0000 7f861f1a4740 1 --2- 172.21.15.116:0/2558080781 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 0x559e8018d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.070+0000 7f861f1a4740 1 --2- 172.21.15.116:0/2558080781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80188520 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.070+0000 7f861f1a4740 1 -- 172.21.15.116:0/2558080781 >> 172.21.15.116:0/2558080781 conn(0x559e80183670 msgr2=0x559e80183a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.070+0000 7f861f1a4740 1 -- 172.21.15.116:0/2558080781 shutdown_connections 2024-03-19T01:21:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.070+0000 7f861f1a4740 1 -- 172.21.15.116:0/2558080781 wait complete. 2024-03-19T01:21:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861f1a4740 1 -- start start 2024-03-19T01:21:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861f1a4740 1 Processor -- start 2024-03-19T01:21:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861f1a4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188100 0x559e801964b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861f1a4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e80188af0 0x559e801969f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861f1a4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e8018dc60 0x559e8019da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861f1a4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559e80194fc0 con 0x559e8018dc60 2024-03-19T01:21:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861f1a4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559e800a8db0 con 0x559e80188af0 2024-03-19T01:21:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861f1a4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559e80195140 con 0x559e80188100 2024-03-19T01:21:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861f1a4740 10 client.0 _ll_register_callbacks cb 0x559e800ade70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:21:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861aa79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e8018dc60 0x559e8019da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861aa79640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188100 0x559e801964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f8619a77640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e80188af0 0x559e801969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861aa79640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188100 0x559e801964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:50146/0 (socket says 172.21.15.116:50146) 2024-03-19T01:21:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861aa79640 1 -- 172.21.15.116:0/4041840585 learned_addr learned my addr 172.21.15.116:0/4041840585 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f8619a77640 1 -- 172.21.15.116:0/4041840585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188100 msgr2=0x559e801964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f8619a77640 1 --2- 172.21.15.116:0/4041840585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188100 0x559e801964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f8619a77640 1 -- 172.21.15.116:0/4041840585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e8018dc60 msgr2=0x559e8019da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f8619a77640 1 --2- 172.21.15.116:0/4041840585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e8018dc60 0x559e8019da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f8619a77640 1 -- 172.21.15.116:0/4041840585 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e800a7c60 con 0x559e80188af0 2024-03-19T01:21:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.071+0000 7f861aa79640 1 --2- 172.21.15.116:0/4041840585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188100 0x559e801964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:21:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f8619a77640 1 --2- 172.21.15.116:0/4041840585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e80188af0 0x559e801969f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8604009640 tx=0x7f8604012c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9641898f785897a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f85fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:21:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f85fdffb640 1 -- 172.21.15.116:0/4041840585 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860400f8f0 con 0x559e80188af0 2024-03-19T01:21:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f861f1a4740 1 -- 172.21.15.116:0/4041840585 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f860c003970 con 0x559e80188af0 2024-03-19T01:21:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f85fdffb640 1 -- 172.21.15.116:0/4041840585 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f860400fa90 con 0x559e80188af0 2024-03-19T01:21:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f85fdffb640 1 -- 172.21.15.116:0/4041840585 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f860400fc90 con 0x559e80188af0 2024-03-19T01:21:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f861f1a4740 10 client.25793 fetch_fsmap learned FSMap version 45 2024-03-19T01:21:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f861f1a4740 1 -- 172.21.15.116:0/4041840585 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x559e7fe01210 con 0x559e80188af0 2024-03-19T01:21:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f85fdffb640 1 -- 172.21.15.116:0/4041840585 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f860c003970 con 0x559e80188af0 2024-03-19T01:21:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f85fdffb640 1 -- 172.21.15.116:0/4041840585 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f86040264b0 con 0x559e80188af0 2024-03-19T01:21:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f861f1a4740 10 client.25793 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:21:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f861f1a4740 10 client.25793 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:21:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f861f1a4740 20 client.25793 populate_metadata read hostname 'smithi116' 2024-03-19T01:21:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f861f1a4740 1 --2- 172.21.15.116:0/4041840585 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e801a2410 0x559e801a4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f861f1a4740 4 client.25793 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:21:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.072+0000 7f861f1a4740 1 -- 172.21.15.116:0/4041840585 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e800fa120 con 0x559e801a2410 2024-03-19T01:21:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.073+0000 7f861aa79640 1 --2- 172.21.15.116:0/4041840585 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e801a2410 0x559e801a4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.073+0000 7f85fdffb640 10 client.25793 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.073+0000 7f861aa79640 1 --2- 172.21.15.116:0/4041840585 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e801a2410 0x559e801a4850 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcb3d1b5504df1d7 server_cookie=7b327ad5717488d4 in_seq=0 out_seq=0 2024-03-19T01:21:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f85fdffb640 1 -- 172.21.15.116:0/4041840585 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559e800fa120 con 0x559e801a2410 2024-03-19T01:21:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f85fdffb640 10 client.25793 handle_command_reply: tid=0 2024-03-19T01:21:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f861f1a4740 1 client.25793 shutdown 2024-03-19T01:21:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f861f1a4740 1 -- 172.21.15.116:0/4041840585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e80188af0 msgr2=0x559e801969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f861f1a4740 1 --2- 172.21.15.116:0/4041840585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e80188af0 0x559e801969f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8604009640 tx=0x7f8604012c30 comp rx=0 tx=0).stop 2024-03-19T01:21:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f861f1a4740 1 -- 172.21.15.116:0/4041840585 shutdown_connections 2024-03-19T01:21:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f861f1a4740 1 --2- 172.21.15.116:0/4041840585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e8018dc60 0x559e8019da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f861f1a4740 1 --2- 172.21.15.116:0/4041840585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e80188af0 0x559e801969f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f861f1a4740 1 --2- 172.21.15.116:0/4041840585 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e801a2410 0x559e801a4850 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f861f1a4740 1 --2- 172.21.15.116:0/4041840585 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188100 0x559e801964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f861f1a4740 1 -- 172.21.15.116:0/4041840585 >> 172.21.15.116:0/4041840585 conn(0x559e80183670 msgr2=0x559e801848c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f85fdffb640 0 client.25793 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f861f1a4740 1 -- 172.21.15.116:0/4041840585 shutdown_connections 2024-03-19T01:21:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f861f1a4740 1 -- 172.21.15.116:0/4041840585 wait complete. 2024-03-19T01:21:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.074+0000 7f861f1a4740 20 client.25793 trim_cache size 0 max 16384 2024-03-19T01:21:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.093+0000 7f861f1a4740 1 Processor -- start 2024-03-19T01:21:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.093+0000 7f861f1a4740 1 -- start start 2024-03-19T01:21:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.093+0000 7f861f1a4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.093+0000 7f861f1a4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 0x559e80182930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.093+0000 7f861f1a4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 0x559e80189590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.093+0000 7f861f1a4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f86141a6990 con 0x559e80188100 2024-03-19T01:21:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.093+0000 7f861f1a4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f86141a84d0 con 0x559e8018dc60 2024-03-19T01:21:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.093+0000 7f861f1a4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f86141a7060 con 0x559e80188af0 2024-03-19T01:21:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.094+0000 7f861aa79640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 0x559e80189590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.094+0000 7f861aa79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.094+0000 7f861aa79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80195dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55848/0 (socket says 172.21.15.116:55848) 2024-03-19T01:21:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.094+0000 7f861aa79640 1 -- 172.21.15.116:0/479040510 learned_addr learned my addr 172.21.15.116:0/479040510 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.094+0000 7f8619a77640 1 --2- 172.21.15.116:0/479040510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 0x559e80182930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.094+0000 7f8619a77640 1 -- 172.21.15.116:0/479040510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 msgr2=0x559e80189590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.094+0000 7f8619a77640 1 --2- 172.21.15.116:0/479040510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 0x559e80189590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.094+0000 7f8619a77640 1 -- 172.21.15.116:0/479040510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 msgr2=0x559e80195dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.094+0000 7f8619a77640 1 --2- 172.21.15.116:0/479040510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80195dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.094+0000 7f8619a77640 1 -- 172.21.15.116:0/479040510 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e7fecd620 con 0x559e80188af0 2024-03-19T01:21:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.095+0000 7f8619a77640 1 --2- 172.21.15.116:0/479040510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 0x559e80182930 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8604026ac0 tx=0x7f860400f810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60b30ad2124c4223 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.095+0000 7f85fffff640 1 -- 172.21.15.116:0/479040510 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604026b40 con 0x559e80188af0 2024-03-19T01:21:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.095+0000 7f85fffff640 1 -- 172.21.15.116:0/479040510 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8604026ce0 con 0x559e80188af0 2024-03-19T01:21:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.095+0000 7f85fffff640 1 -- 172.21.15.116:0/479040510 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604026660 con 0x559e80188af0 2024-03-19T01:21:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.096+0000 7f861f1a4740 1 -- 172.21.15.116:0/479040510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 msgr2=0x559e80182930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.096+0000 7f861f1a4740 1 --2- 172.21.15.116:0/479040510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 0x559e80182930 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8604026ac0 tx=0x7f860400f810 comp rx=0 tx=0).stop 2024-03-19T01:21:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.096+0000 7f861f1a4740 1 -- 172.21.15.116:0/479040510 shutdown_connections 2024-03-19T01:21:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.096+0000 7f861f1a4740 1 --2- 172.21.15.116:0/479040510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 0x559e80189590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.096+0000 7f861f1a4740 1 --2- 172.21.15.116:0/479040510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 0x559e80182930 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.096+0000 7f861f1a4740 1 --2- 172.21.15.116:0/479040510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.096+0000 7f861f1a4740 1 -- 172.21.15.116:0/479040510 >> 172.21.15.116:0/479040510 conn(0x559e80183670 msgr2=0x559e80182220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.096+0000 7f861f1a4740 1 -- 172.21.15.116:0/479040510 shutdown_connections 2024-03-19T01:21:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.096+0000 7f861f1a4740 1 -- 172.21.15.116:0/479040510 wait complete. 2024-03-19T01:21:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.096+0000 7f861f1a4740 1 -- start start 2024-03-19T01:21:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.096+0000 7f861f1a4740 1 Processor -- start 2024-03-19T01:21:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f861f1a4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80193a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f861f1a4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 0x559e80193f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f861f1a4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 0x559e8020af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f861f1a4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f86141a7e00 con 0x559e80188100 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f861f1a4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f86141a7730 con 0x559e8018dc60 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f861f1a4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f86141a4970 con 0x559e80188af0 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f861f1a4740 10 client.0 _ll_register_callbacks cb 0x7f8604009960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f861aa79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80193a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f8619a77640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 0x559e80193f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f861aa79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80193a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55862/0 (socket says 172.21.15.116:55862) 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f861aa79640 1 -- 172.21.15.116:0/2367762090 learned_addr learned my addr 172.21.15.116:0/2367762090 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f861aa79640 1 --2- 172.21.15.116:0/2367762090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 0x559e8020af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f8619a77640 1 -- 172.21.15.116:0/2367762090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 msgr2=0x559e8020af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f8619a77640 1 --2- 172.21.15.116:0/2367762090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 0x559e8020af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f8619a77640 1 -- 172.21.15.116:0/2367762090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 msgr2=0x559e80193a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f8619a77640 1 --2- 172.21.15.116:0/2367762090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80193a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f8619a77640 1 -- 172.21.15.116:0/2367762090 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e8019dbd0 con 0x559e80188af0 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f861aa79640 1 --2- 172.21.15.116:0/2367762090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80193a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f8619a77640 1 --2- 172.21.15.116:0/2367762090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 0x559e80193f90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8604026ac0 tx=0x7f8604009330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b409f09d2102aa3f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.097+0000 7f85fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f85fdffb640 1 -- 172.21.15.116:0/2367762090 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8604026af0 con 0x559e80188af0 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f85fdffb640 1 -- 172.21.15.116:0/2367762090 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8604005250 con 0x559e80188af0 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f861f1a4740 1 -- 172.21.15.116:0/2367762090 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8604026ce0 con 0x559e80188af0 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f85fdffb640 1 -- 172.21.15.116:0/2367762090 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f8604005450 con 0x559e80188af0 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f861f1a4740 10 client.25786 fetch_fsmap learned FSMap version 45 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f861f1a4740 1 -- 172.21.15.116:0/2367762090 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x559e8020c370 con 0x559e80188af0 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f85fdffb640 1 -- 172.21.15.116:0/2367762090 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8604026ce0 con 0x559e80188af0 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f85fdffb640 1 -- 172.21.15.116:0/2367762090 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f8604010430 con 0x559e80188af0 2024-03-19T01:21:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f861f1a4740 10 client.25786 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:21:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f861f1a4740 10 client.25786 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:21:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f861f1a4740 20 client.25786 populate_metadata read hostname 'smithi116' 2024-03-19T01:21:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f861f1a4740 1 --2- 172.21.15.116:0/2367762090 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e8020ccc0 0x559e8020f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f861f1a4740 4 client.25786 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:21:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f861f1a4740 1 -- 172.21.15.116:0/2367762090 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559e800f4470 con 0x559e8020ccc0 2024-03-19T01:21:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.098+0000 7f861aa79640 1 --2- 172.21.15.116:0/2367762090 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e8020ccc0 0x559e8020f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.099+0000 7f85fdffb640 10 client.25786 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.099+0000 7f861aa79640 1 --2- 172.21.15.116:0/2367762090 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e8020ccc0 0x559e8020f0e0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49d82a0275fcbe91 server_cookie=c5d1e5bc973f3964 in_seq=0 out_seq=0 2024-03-19T01:21:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.106+0000 7f85fdffb640 1 -- 172.21.15.116:0/2367762090 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559e800f4470 con 0x559e8020ccc0 2024-03-19T01:21:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.106+0000 7f85fdffb640 10 client.25786 handle_command_reply: tid=0 2024-03-19T01:21:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.106+0000 7f861f1a4740 1 client.25786 shutdown 2024-03-19T01:21:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861f1a4740 1 -- 172.21.15.116:0/2367762090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 msgr2=0x559e80193f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861f1a4740 1 --2- 172.21.15.116:0/2367762090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 0x559e80193f90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8604026ac0 tx=0x7f8604009330 comp rx=0 tx=0).stop 2024-03-19T01:21:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861f1a4740 1 -- 172.21.15.116:0/2367762090 shutdown_connections 2024-03-19T01:21:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861f1a4740 1 --2- 172.21.15.116:0/2367762090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e8018dc60 0x559e8020af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861f1a4740 1 --2- 172.21.15.116:0/2367762090 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e8020ccc0 0x559e8020f0e0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861f1a4740 1 --2- 172.21.15.116:0/2367762090 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e80188af0 0x559e80193f90 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f85fdffb640 0 client.25786 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861f1a4740 1 --2- 172.21.15.116:0/2367762090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e80188100 0x559e80193a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861f1a4740 1 -- 172.21.15.116:0/2367762090 >> 172.21.15.116:0/2367762090 conn(0x559e80183670 msgr2=0x559e801868d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861f1a4740 1 -- 172.21.15.116:0/2367762090 shutdown_connections 2024-03-19T01:21:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861f1a4740 1 -- 172.21.15.116:0/2367762090 wait complete. 2024-03-19T01:21:08.108 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861f1a4740 20 client.25786 trim_cache size 0 max 16384 2024-03-19T01:21:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:21:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:21:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:21:08.109 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:21:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861c4dc640 1 -- 172.21.15.116:0/2248709070 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f86000a28a0 msgr2=0x7f86000a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861c4dc640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f86000a28a0 0x7f86000a4d60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f86100cc970 tx=0x7f861006b2e0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861c4dc640 1 -- 172.21.15.116:0/2248709070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86140ffb00 msgr2=0x7f8614197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.107+0000 7f861c4dc640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86140ffb00 0x7f8614197ac0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8614068ca0 tx=0x7f861000ee50 comp rx=0 tx=0).stop 2024-03-19T01:21:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.108+0000 7f861c4dc640 1 -- 172.21.15.116:0/2248709070 shutdown_connections 2024-03-19T01:21:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.108+0000 7f861c4dc640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8614103490 0x7f861419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.108+0000 7f861c4dc640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f86000a28a0 0x7f86000a4d60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.108+0000 7f861c4dc640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8614102ad0 0x7f8614198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.108+0000 7f861c4dc640 1 --2- 172.21.15.116:0/2248709070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86140ffb00 0x7f8614197ac0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.108+0000 7f861c4dc640 1 -- 172.21.15.116:0/2248709070 >> 172.21.15.116:0/2248709070 conn(0x7f86140fb470 msgr2=0x7f8614105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.108+0000 7f861c4dc640 1 -- 172.21.15.116:0/2248709070 shutdown_connections 2024-03-19T01:21:08.110 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:08.108+0000 7f861c4dc640 1 -- 172.21.15.116:0/2248709070 wait complete. 2024-03-19T01:21:09.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:09 smithi162 ceph-mon[67178]: pgmap v608: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.4 MiB/s wr, 706 op/s 2024-03-19T01:21:09.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:09 smithi116 ceph-mon[62701]: pgmap v608: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.4 MiB/s wr, 706 op/s 2024-03-19T01:21:09.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:09 smithi156 ceph-mon[66799]: pgmap v608: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.4 MiB/s wr, 706 op/s 2024-03-19T01:21:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 505.0 s, 2727.6 tps, lat 36.682 ms stddev 43.787 2024-03-19T01:21:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:11 smithi162 ceph-mon[67178]: pgmap v609: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.6 MiB/s wr, 732 op/s 2024-03-19T01:21:11.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:11 smithi116 ceph-mon[62701]: pgmap v609: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.6 MiB/s wr, 732 op/s 2024-03-19T01:21:11.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:11 smithi156 ceph-mon[66799]: pgmap v609: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 7.6 MiB/s wr, 732 op/s 2024-03-19T01:21:13.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:13 smithi162 ceph-mon[67178]: pgmap v610: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.0 MiB/s wr, 582 op/s 2024-03-19T01:21:13.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:13 smithi116 ceph-mon[62701]: pgmap v610: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.0 MiB/s wr, 582 op/s 2024-03-19T01:21:13.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:13 smithi156 ceph-mon[66799]: pgmap v610: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 6.0 MiB/s wr, 582 op/s 2024-03-19T01:21:15.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:15 smithi162 ceph-mon[67178]: pgmap v611: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.0 MiB/s wr, 771 op/s 2024-03-19T01:21:15.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:15 smithi116 ceph-mon[62701]: pgmap v611: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.0 MiB/s wr, 771 op/s 2024-03-19T01:21:15.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:15 smithi156 ceph-mon[66799]: pgmap v611: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 8.0 MiB/s wr, 771 op/s 2024-03-19T01:21:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 510.0 s, 2502.4 tps, lat 39.869 ms stddev 44.534 2024-03-19T01:21:17.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:17 smithi162 ceph-mon[67178]: pgmap v612: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.3 MiB/s wr, 627 op/s 2024-03-19T01:21:17.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:17 smithi116 ceph-mon[62701]: pgmap v612: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.3 MiB/s wr, 627 op/s 2024-03-19T01:21:17.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:17 smithi156 ceph-mon[66799]: pgmap v612: 97 pgs: 97 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 6.3 MiB/s wr, 627 op/s 2024-03-19T01:21:19.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:19 smithi162 ceph-mon[67178]: pgmap v613: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 722 op/s 2024-03-19T01:21:19.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:19 smithi116 ceph-mon[62701]: pgmap v613: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 722 op/s 2024-03-19T01:21:19.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:19 smithi156 ceph-mon[66799]: pgmap v613: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 MiB/s wr, 722 op/s 2024-03-19T01:21:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.232+0000 7f0d80368640 1 -- 172.21.15.116:0/4162972802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d78101970 msgr2=0x7f0d78101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:21.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.232+0000 7f0d80368640 1 --2- 172.21.15.116:0/4162972802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d78101970 0x7f0d78101df0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0d68009900 tx=0x7f0d6802bc30 comp rx=0 tx=0).stop 2024-03-19T01:21:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.233+0000 7f0d80368640 1 -- 172.21.15.116:0/4162972802 shutdown_connections 2024-03-19T01:21:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.233+0000 7f0d80368640 1 --2- 172.21.15.116:0/4162972802 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d78102330 0x7f0d78108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.233+0000 7f0d80368640 1 --2- 172.21.15.116:0/4162972802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d78101970 0x7f0d78101df0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.233+0000 7f0d80368640 1 --2- 172.21.15.116:0/4162972802 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d78100780 0x7f0d78100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.233+0000 7f0d80368640 1 -- 172.21.15.116:0/4162972802 >> 172.21.15.116:0/4162972802 conn(0x7f0d780fbcf0 msgr2=0x7f0d780fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.233+0000 7f0d80368640 1 -- 172.21.15.116:0/4162972802 shutdown_connections 2024-03-19T01:21:21.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.233+0000 7f0d80368640 1 -- 172.21.15.116:0/4162972802 wait complete. 2024-03-19T01:21:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.234+0000 7f0d80368640 1 Processor -- start 2024-03-19T01:21:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.234+0000 7f0d80368640 1 -- start start 2024-03-19T01:21:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.234+0000 7f0d80368640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d78100780 0x7f0d78197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.234+0000 7f0d80368640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d78101970 0x7f0d78197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.234+0000 7f0d80368640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d78102330 0x7f0d7819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.234+0000 7f0d80368640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7810c430 con 0x7f0d78100780 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.234+0000 7f0d80368640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7810c2b0 con 0x7f0d78102330 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.234+0000 7f0d80368640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7810c5b0 con 0x7f0d78101970 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.234+0000 7f0d7e104640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d78100780 0x7f0d78197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.234+0000 7f0d7d903640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d78101970 0x7f0d78197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.234+0000 7f0d7e104640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d78100780 0x7f0d78197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41436/0 (socket says 172.21.15.116:41436) 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.235+0000 7f0d7e104640 1 -- 172.21.15.116:0/2694415769 learned_addr learned my addr 172.21.15.116:0/2694415769 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.235+0000 7f0d7e905640 1 --2- 172.21.15.116:0/2694415769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d78102330 0x7f0d7819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.235+0000 7f0d7d903640 1 -- 172.21.15.116:0/2694415769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d78102330 msgr2=0x7f0d7819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.235+0000 7f0d7d903640 1 --2- 172.21.15.116:0/2694415769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d78102330 0x7f0d7819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.235+0000 7f0d7d903640 1 -- 172.21.15.116:0/2694415769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d78100780 msgr2=0x7f0d78197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.235+0000 7f0d7d903640 1 --2- 172.21.15.116:0/2694415769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d78100780 0x7f0d78197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.235+0000 7f0d7d903640 1 -- 172.21.15.116:0/2694415769 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7819f6d0 con 0x7f0d78101970 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.235+0000 7f0d7d903640 1 --2- 172.21.15.116:0/2694415769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d78101970 0x7f0d78197ff0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0d68036040 tx=0x7f0d6802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:21.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.235+0000 7f0d6f7fe640 1 -- 172.21.15.116:0/2694415769 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68043020 con 0x7f0d78101970 2024-03-19T01:21:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.235+0000 7f0d6f7fe640 1 -- 172.21.15.116:0/2694415769 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d680054e0 con 0x7f0d78101970 2024-03-19T01:21:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.235+0000 7f0d6f7fe640 1 -- 172.21.15.116:0/2694415769 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d68003c10 con 0x7f0d78101970 2024-03-19T01:21:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.235+0000 7f0d80368640 1 -- 172.21.15.116:0/2694415769 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d7819f960 con 0x7f0d78101970 2024-03-19T01:21:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.235+0000 7f0d80368640 1 -- 172.21.15.116:0/2694415769 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d7819fe30 con 0x7f0d78101970 2024-03-19T01:21:21.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.238+0000 7f0d6f7fe640 1 -- 172.21.15.116:0/2694415769 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f0d68046020 con 0x7f0d78101970 2024-03-19T01:21:21.239 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.238+0000 7f0d8302a740 1 Processor -- start 2024-03-19T01:21:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.239+0000 7f0d8302a740 1 -- start start 2024-03-19T01:21:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.239+0000 7f0d8302a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab914510 0x5633ab914930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.239+0000 7f0d8302a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914f00 0x5633ab919a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.239+0000 7f0d8302a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab91a070 0x5633ab91c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.239+0000 7f0d8302a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633ab874be0 con 0x5633ab914510 2024-03-19T01:21:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.239+0000 7f0d8302a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5633ab81d960 con 0x5633ab91a070 2024-03-19T01:21:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.239+0000 7f0d8302a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5633ab870c80 con 0x5633ab914f00 2024-03-19T01:21:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.239+0000 7f0d6f7fe640 1 --2- 172.21.15.116:0/2694415769 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0d640a2a10 0x7f0d640a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.239+0000 7f0d7d903640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab91a070 0x5633ab91c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.239+0000 7f0d7d903640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab91a070 0x5633ab91c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:48224/0 (socket says 172.21.15.116:48224) 2024-03-19T01:21:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.239+0000 7f0d7d903640 1 -- 172.21.15.116:0/3832062004 learned_addr learned my addr 172.21.15.116:0/3832062004 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.239+0000 7f0d7e104640 1 --2- 172.21.15.116:0/3832062004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914f00 0x5633ab919a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.239+0000 7f0d6f7fe640 1 -- 172.21.15.116:0/2694415769 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0d680f0980 con 0x7f0d78101970 2024-03-19T01:21:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.240+0000 7f0d7d903640 1 -- 172.21.15.116:0/3832062004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914f00 msgr2=0x5633ab919a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.240+0000 7f0d7d903640 1 --2- 172.21.15.116:0/3832062004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914f00 0x5633ab919a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.240+0000 7f0d7d903640 1 -- 172.21.15.116:0/3832062004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab914510 msgr2=0x5633ab914930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:21:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.240+0000 7f0d7d903640 1 --2- 172.21.15.116:0/3832062004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab914510 0x5633ab914930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.240+0000 7f0d7d903640 1 -- 172.21.15.116:0/3832062004 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633ab814d20 con 0x5633ab91a070 2024-03-19T01:21:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.240+0000 7f0d7d903640 1 --2- 172.21.15.116:0/3832062004 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab91a070 0x5633ab91c500 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0d6802ceb0 tx=0x7f0d6802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf5a91cd28cd1f60 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.241+0000 7f0d7e905640 1 --2- 172.21.15.116:0/2694415769 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0d640a2a10 0x7f0d640a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.241+0000 7f0d6cbfe640 1 -- 172.21.15.116:0/3832062004 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d680bcc00 con 0x5633ab91a070 2024-03-19T01:21:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.241+0000 7f0d6cbfe640 1 -- 172.21.15.116:0/3832062004 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d680035f0 con 0x5633ab91a070 2024-03-19T01:21:21.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.241+0000 7f0d6cbfe640 1 -- 172.21.15.116:0/3832062004 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d680bb340 con 0x5633ab91a070 2024-03-19T01:21:21.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.243+0000 7f0d8302a740 1 -- 172.21.15.116:0/3832062004 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab91a070 msgr2=0x5633ab91c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:21.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.243+0000 7f0d7e905640 1 --2- 172.21.15.116:0/2694415769 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0d640a2a10 0x7f0d640a4ed0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x5633ab915760 tx=0x7f0d70009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:21.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.243+0000 7f0d8302a740 1 --2- 172.21.15.116:0/3832062004 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab91a070 0x5633ab91c500 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0d6802ceb0 tx=0x7f0d6802bc30 comp rx=0 tx=0).stop 2024-03-19T01:21:21.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.243+0000 7f0d8302a740 1 -- 172.21.15.116:0/3832062004 shutdown_connections 2024-03-19T01:21:21.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.243+0000 7f0d8302a740 1 --2- 172.21.15.116:0/3832062004 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab91a070 0x5633ab91c500 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.243+0000 7f0d8302a740 1 --2- 172.21.15.116:0/3832062004 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914f00 0x5633ab919a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.243+0000 7f0d8302a740 1 --2- 172.21.15.116:0/3832062004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab914510 0x5633ab914930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.243+0000 7f0d8302a740 1 -- 172.21.15.116:0/3832062004 >> 172.21.15.116:0/3832062004 conn(0x5633ab90b670 msgr2=0x5633ab90ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.243+0000 7f0d8302a740 1 -- 172.21.15.116:0/3832062004 shutdown_connections 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.243+0000 7f0d8302a740 1 -- 172.21.15.116:0/3832062004 wait complete. 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.243+0000 7f0d8302a740 1 -- start start 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.243+0000 7f0d8302a740 1 Processor -- start 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d8302a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914510 0x5633ab91e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d8302a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab914f00 0x5633ab91e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d8302a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab91a070 0x5633ab925810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d8302a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5633ab91cde0 con 0x5633ab914f00 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d8302a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5633ab830db0 con 0x5633ab91a070 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d8302a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5633ab91cf60 con 0x5633ab914510 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d7d903640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab914f00 0x5633ab91e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d7d903640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab914f00 0x5633ab91e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41444/0 (socket says 172.21.15.116:41444) 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d7d903640 1 -- 172.21.15.116:0/2488766332 learned_addr learned my addr 172.21.15.116:0/2488766332 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d8302a740 10 client.0 _ll_register_callbacks cb 0x5633ab835e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d7e104640 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914510 0x5633ab91e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d7e905640 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab91a070 0x5633ab925810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d7d903640 1 -- 172.21.15.116:0/2488766332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914510 msgr2=0x5633ab91e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d7d903640 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914510 0x5633ab91e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d7d903640 1 -- 172.21.15.116:0/2488766332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab91a070 msgr2=0x5633ab925810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d7d903640 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab91a070 0x5633ab925810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d7d903640 1 -- 172.21.15.116:0/2488766332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633ab82fc60 con 0x5633ab914f00 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d7e905640 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab91a070 0x5633ab925810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d7d903640 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab914f00 0x5633ab91e840 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0d680b9700 tx=0x7f0d680bcc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3816880cbe22b797 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d61ffb640 1 -- 172.21.15.116:0/2488766332 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d680d5070 con 0x5633ab914f00 2024-03-19T01:21:21.248 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d61ffb640 1 -- 172.21.15.116:0/2488766332 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0d680bb5a0 con 0x5633ab914f00 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.244+0000 7f0d8302a740 1 -- 172.21.15.116:0/2488766332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5633ab882120 con 0x5633ab914f00 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.245+0000 7f0d61ffb640 1 -- 172.21.15.116:0/2488766332 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f0d680bd2a0 con 0x5633ab914f00 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.245+0000 7f0d8302a740 10 client.15981 fetch_fsmap learned FSMap version 45 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.245+0000 7f0d8302a740 1 -- 172.21.15.116:0/2488766332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5633ab589210 con 0x5633ab914f00 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.245+0000 7f0d61ffb640 1 -- 172.21.15.116:0/2488766332 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5633ab882120 con 0x5633ab914f00 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.245+0000 7f0d61ffb640 1 -- 172.21.15.116:0/2488766332 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f0d680bd500 con 0x5633ab914f00 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.245+0000 7f0d8302a740 10 client.15981 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.245+0000 7f0d8302a740 10 client.15981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.245+0000 7f0d8302a740 20 client.15981 populate_metadata read hostname 'smithi116' 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.245+0000 7f0d8302a740 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5633ab929d00 0x5633ab92c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.245+0000 7f0d8302a740 4 client.15981 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.245+0000 7f0d8302a740 1 -- 172.21.15.116:0/2488766332 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633ab87c470 con 0x5633ab929d00 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.245+0000 7f0d7e104640 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5633ab929d00 0x5633ab92c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.249 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d7e104640 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5633ab929d00 0x5633ab92c0f0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e85645294e6e08c server_cookie=18158c8efd2828b1 in_seq=0 out_seq=0 2024-03-19T01:21:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d61ffb640 10 client.15981 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:21.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d61ffb640 1 -- 172.21.15.116:0/2488766332 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5633ab87c470 con 0x5633ab929d00 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d61ffb640 10 client.15981 handle_command_reply: tid=0 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d8302a740 1 client.15981 shutdown 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d8302a740 1 -- 172.21.15.116:0/2488766332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab914f00 msgr2=0x5633ab91e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d8302a740 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab914f00 0x5633ab91e840 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0d680b9700 tx=0x7f0d680bcc00 comp rx=0 tx=0).stop 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d8302a740 1 -- 172.21.15.116:0/2488766332 shutdown_connections 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d8302a740 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab91a070 0x5633ab925810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d8302a740 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab914f00 0x5633ab91e840 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d8302a740 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5633ab929d00 0x5633ab92c0f0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d8302a740 1 --2- 172.21.15.116:0/2488766332 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914510 0x5633ab91e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d8302a740 1 -- 172.21.15.116:0/2488766332 >> 172.21.15.116:0/2488766332 conn(0x5633ab90b670 msgr2=0x5633ab90c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.246+0000 7f0d61ffb640 0 client.15981 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.247+0000 7f0d8302a740 1 -- 172.21.15.116:0/2488766332 shutdown_connections 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.247+0000 7f0d8302a740 1 -- 172.21.15.116:0/2488766332 wait complete. 2024-03-19T01:21:21.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.247+0000 7f0d8302a740 20 client.15981 trim_cache size 0 max 16384 2024-03-19T01:21:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.270+0000 7f0d8302a740 1 Processor -- start 2024-03-19T01:21:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.270+0000 7f0d8302a740 1 -- start start 2024-03-19T01:21:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.270+0000 7f0d8302a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab914510 0x5633ab90efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.270+0000 7f0d8302a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914f00 0x5633ab9257d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.270+0000 7f0d8302a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 0x5633ab930980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d8302a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0d781a6a70 con 0x5633ab91a070 2024-03-19T01:21:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d8302a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0d781a85b0 con 0x5633ab914510 2024-03-19T01:21:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d8302a740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0d781a7140 con 0x5633ab914f00 2024-03-19T01:21:21.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d7e905640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 0x5633ab930980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d7d903640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914f00 0x5633ab9257d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d7e104640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab914510 0x5633ab90efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d7e905640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 0x5633ab930980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41446/0 (socket says 172.21.15.116:41446) 2024-03-19T01:21:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d7e905640 1 -- 172.21.15.116:0/1294842054 learned_addr learned my addr 172.21.15.116:0/1294842054 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d7e905640 1 -- 172.21.15.116:0/1294842054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914f00 msgr2=0x5633ab9257d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d7e905640 1 --2- 172.21.15.116:0/1294842054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914f00 0x5633ab9257d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d7e905640 1 -- 172.21.15.116:0/1294842054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab914510 msgr2=0x5633ab90efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d7e905640 1 --2- 172.21.15.116:0/1294842054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab914510 0x5633ab90efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d7e905640 1 -- 172.21.15.116:0/1294842054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633ab655620 con 0x5633ab91a070 2024-03-19T01:21:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.271+0000 7f0d7e905640 1 --2- 172.21.15.116:0/1294842054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 0x5633ab930980 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0d70003e60 tx=0x7f0d70014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5be346ede07905 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.272+0000 7f0d63fff640 1 -- 172.21.15.116:0/1294842054 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7001a450 con 0x5633ab91a070 2024-03-19T01:21:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.272+0000 7f0d63fff640 1 -- 172.21.15.116:0/1294842054 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d7001aad0 con 0x5633ab91a070 2024-03-19T01:21:21.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.272+0000 7f0d63fff640 1 -- 172.21.15.116:0/1294842054 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7001bc00 con 0x5633ab91a070 2024-03-19T01:21:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.272+0000 7f0d8302a740 1 -- 172.21.15.116:0/1294842054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 msgr2=0x5633ab930980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.272+0000 7f0d8302a740 1 --2- 172.21.15.116:0/1294842054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 0x5633ab930980 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0d70003e60 tx=0x7f0d70014000 comp rx=0 tx=0).stop 2024-03-19T01:21:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.272+0000 7f0d8302a740 1 -- 172.21.15.116:0/1294842054 shutdown_connections 2024-03-19T01:21:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.272+0000 7f0d8302a740 1 --2- 172.21.15.116:0/1294842054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 0x5633ab930980 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.272+0000 7f0d8302a740 1 --2- 172.21.15.116:0/1294842054 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914f00 0x5633ab9257d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.277 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.272+0000 7f0d8302a740 1 --2- 172.21.15.116:0/1294842054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab914510 0x5633ab90efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.272+0000 7f0d8302a740 1 -- 172.21.15.116:0/1294842054 >> 172.21.15.116:0/1294842054 conn(0x5633ab90b670 msgr2=0x5633ab90e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d8302a740 1 -- 172.21.15.116:0/1294842054 shutdown_connections 2024-03-19T01:21:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d8302a740 1 -- 172.21.15.116:0/1294842054 wait complete. 2024-03-19T01:21:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d8302a740 1 -- start start 2024-03-19T01:21:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d8302a740 1 Processor -- start 2024-03-19T01:21:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d8302a740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914510 0x5633ab9233d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d8302a740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab914f00 0x5633ab923910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d8302a740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 0x5633ab924920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d8302a740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0d781a7ee0 con 0x5633ab91a070 2024-03-19T01:21:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d8302a740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0d781a7810 con 0x5633ab914f00 2024-03-19T01:21:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d7e905640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 0x5633ab924920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d7d903640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab914f00 0x5633ab923910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.278 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d7e905640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 0x5633ab924920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41462/0 (socket says 172.21.15.116:41462) 2024-03-19T01:21:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d7e104640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914510 0x5633ab9233d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d7e905640 1 -- 172.21.15.116:0/3921034787 learned_addr learned my addr 172.21.15.116:0/3921034787 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.273+0000 7f0d8302a740 1 -- 172.21.15.116:0/3921034787 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0d781a4a50 con 0x5633ab914510 2024-03-19T01:21:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d8302a740 10 client.0 _ll_register_callbacks cb 0x7f0d680b8eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:21:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d7e905640 1 -- 172.21.15.116:0/3921034787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914510 msgr2=0x5633ab9233d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d7e905640 1 --2- 172.21.15.116:0/3921034787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914510 0x5633ab9233d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d7e905640 1 -- 172.21.15.116:0/3921034787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab914f00 msgr2=0x5633ab923910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d7e905640 1 --2- 172.21.15.116:0/3921034787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab914f00 0x5633ab923910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d7e905640 1 -- 172.21.15.116:0/3921034787 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633ab924f10 con 0x5633ab91a070 2024-03-19T01:21:21.279 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d7e905640 1 --2- 172.21.15.116:0/3921034787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 0x5633ab924920 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0d70003c50 tx=0x7f0d7001b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500f71a20b9e178c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d61ffb640 1 -- 172.21.15.116:0/3921034787 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7001bc90 con 0x5633ab91a070 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d8302a740 1 -- 172.21.15.116:0/3921034787 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d7001aad0 con 0x5633ab91a070 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d61ffb640 1 -- 172.21.15.116:0/3921034787 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0d70022040 con 0x5633ab91a070 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d61ffb640 1 -- 172.21.15.116:0/3921034787 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f0d7001be30 con 0x5633ab91a070 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d8302a740 10 client.15987 fetch_fsmap learned FSMap version 45 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.274+0000 7f0d8302a740 1 -- 172.21.15.116:0/3921034787 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5633ab990480 con 0x5633ab91a070 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.275+0000 7f0d61ffb640 1 -- 172.21.15.116:0/3921034787 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d7001aad0 con 0x5633ab91a070 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.275+0000 7f0d61ffb640 1 -- 172.21.15.116:0/3921034787 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f0d70018040 con 0x5633ab91a070 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.275+0000 7f0d8302a740 10 client.15987 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.275+0000 7f0d8302a740 10 client.15987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.275+0000 7f0d8302a740 20 client.15987 populate_metadata read hostname 'smithi116' 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.275+0000 7f0d8302a740 1 --2- 172.21.15.116:0/3921034787 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5633ab990d70 0x5633ab993190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:21.280 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.275+0000 7f0d8302a740 4 client.15987 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:21:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.275+0000 7f0d8302a740 1 -- 172.21.15.116:0/3921034787 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5633ab7fa770 con 0x5633ab990d70 2024-03-19T01:21:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.275+0000 7f0d7e104640 1 --2- 172.21.15.116:0/3921034787 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5633ab990d70 0x5633ab993190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.276+0000 7f0d7e104640 1 --2- 172.21.15.116:0/3921034787 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5633ab990d70 0x5633ab993190 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53910cb361702b0 server_cookie=17bc863c5531a6f in_seq=0 out_seq=0 2024-03-19T01:21:21.281 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.276+0000 7f0d61ffb640 10 client.15987 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.325+0000 7f0d61ffb640 1 -- 172.21.15.116:0/3921034787 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5633ab7fa770 con 0x5633ab990d70 2024-03-19T01:21:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.325+0000 7f0d61ffb640 10 client.15987 handle_command_reply: tid=0 2024-03-19T01:21:21.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.326+0000 7f0d8302a740 1 client.15987 shutdown 2024-03-19T01:21:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.327+0000 7f0d8302a740 1 -- 172.21.15.116:0/3921034787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 msgr2=0x5633ab924920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:21.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.327+0000 7f0d8302a740 1 --2- 172.21.15.116:0/3921034787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 0x5633ab924920 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0d70003c50 tx=0x7f0d7001b5a0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.328+0000 7f0d8302a740 1 -- 172.21.15.116:0/3921034787 shutdown_connections 2024-03-19T01:21:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.328+0000 7f0d8302a740 1 --2- 172.21.15.116:0/3921034787 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5633ab91a070 0x5633ab924920 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.328+0000 7f0d8302a740 1 --2- 172.21.15.116:0/3921034787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5633ab914f00 0x5633ab923910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.328+0000 7f0d8302a740 1 --2- 172.21.15.116:0/3921034787 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5633ab990d70 0x5633ab993190 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.328+0000 7f0d8302a740 1 --2- 172.21.15.116:0/3921034787 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5633ab914510 0x5633ab9233d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.328+0000 7f0d61ffb640 0 client.15987 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:21.329 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.328+0000 7f0d8302a740 1 -- 172.21.15.116:0/3921034787 >> 172.21.15.116:0/3921034787 conn(0x5633ab90b670 msgr2=0x5633ab90e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.329+0000 7f0d8302a740 1 -- 172.21.15.116:0/3921034787 shutdown_connections 2024-03-19T01:21:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.329+0000 7f0d8302a740 1 -- 172.21.15.116:0/3921034787 wait complete. 2024-03-19T01:21:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.329+0000 7f0d8302a740 20 client.15987 trim_cache size 0 max 16384 2024-03-19T01:21:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:21:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:21:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:21:21.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:21:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.330+0000 7f0d80368640 1 -- 172.21.15.116:0/2694415769 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0d640a2a10 msgr2=0x7f0d640a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.330+0000 7f0d80368640 1 --2- 172.21.15.116:0/2694415769 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0d640a2a10 0x7f0d640a4ed0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x5633ab915760 tx=0x7f0d70009500 comp rx=0 tx=0).stop 2024-03-19T01:21:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.330+0000 7f0d80368640 1 -- 172.21.15.116:0/2694415769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d78101970 msgr2=0x7f0d78197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:21.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.330+0000 7f0d80368640 1 --2- 172.21.15.116:0/2694415769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d78101970 0x7f0d78197ff0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0d68036040 tx=0x7f0d6802c650 comp rx=0 tx=0).stop 2024-03-19T01:21:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.332+0000 7f0d80368640 1 -- 172.21.15.116:0/2694415769 shutdown_connections 2024-03-19T01:21:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.332+0000 7f0d80368640 1 --2- 172.21.15.116:0/2694415769 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f0d640a2a10 0x7f0d640a4ed0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.332+0000 7f0d80368640 1 --2- 172.21.15.116:0/2694415769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d78102330 0x7f0d7819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.332+0000 7f0d80368640 1 --2- 172.21.15.116:0/2694415769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f0d78101970 0x7f0d78197ff0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.332+0000 7f0d80368640 1 --2- 172.21.15.116:0/2694415769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d78100780 0x7f0d78197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:21.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.332+0000 7f0d80368640 1 -- 172.21.15.116:0/2694415769 >> 172.21.15.116:0/2694415769 conn(0x7f0d780fbcf0 msgr2=0x7f0d78069fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.332+0000 7f0d80368640 1 -- 172.21.15.116:0/2694415769 shutdown_connections 2024-03-19T01:21:21.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:21.332+0000 7f0d80368640 1 -- 172.21.15.116:0/2694415769 wait complete. 2024-03-19T01:21:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 515.0 s, 2185.6 tps, lat 45.182 ms stddev 51.276 2024-03-19T01:21:21.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:21 smithi162 ceph-mon[67178]: pgmap v614: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.7 MiB/s wr, 775 op/s 2024-03-19T01:21:21.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:21 smithi116 ceph-mon[62701]: pgmap v614: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.7 MiB/s wr, 775 op/s 2024-03-19T01:21:21.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:21 smithi156 ceph-mon[66799]: pgmap v614: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.7 MiB/s wr, 775 op/s 2024-03-19T01:21:23.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:23 smithi162 ceph-mon[67178]: pgmap v615: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 642 op/s 2024-03-19T01:21:23.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:23 smithi116 ceph-mon[62701]: pgmap v615: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 642 op/s 2024-03-19T01:21:23.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:23 smithi156 ceph-mon[66799]: pgmap v615: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 642 op/s 2024-03-19T01:21:25.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:25 smithi162 ceph-mon[67178]: pgmap v616: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 777 op/s 2024-03-19T01:21:25.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:25 smithi116 ceph-mon[62701]: pgmap v616: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 777 op/s 2024-03-19T01:21:25.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:25 smithi156 ceph-mon[66799]: pgmap v616: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 777 op/s 2024-03-19T01:21:26.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 520.0 s, 1305.8 tps, lat 76.280 ms stddev 86.012 2024-03-19T01:21:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:27 smithi162 ceph-mon[67178]: pgmap v617: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.8 MiB/s wr, 673 op/s 2024-03-19T01:21:27.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:27 smithi116 ceph-mon[62701]: pgmap v617: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.8 MiB/s wr, 673 op/s 2024-03-19T01:21:27.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:27 smithi156 ceph-mon[66799]: pgmap v617: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.8 MiB/s wr, 673 op/s 2024-03-19T01:21:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:29 smithi162 ceph-mon[67178]: pgmap v618: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.1 MiB/s wr, 769 op/s 2024-03-19T01:21:29.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:29 smithi116 ceph-mon[62701]: pgmap v618: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.1 MiB/s wr, 769 op/s 2024-03-19T01:21:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:29 smithi156 ceph-mon[66799]: pgmap v618: 97 pgs: 97 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 9.1 MiB/s wr, 769 op/s 2024-03-19T01:21:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 525.0 s, 1490.2 tps, lat 67.950 ms stddev 77.274 2024-03-19T01:21:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:31 smithi162 ceph-mon[67178]: pgmap v619: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 759 op/s 2024-03-19T01:21:31.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:31 smithi116 ceph-mon[62701]: pgmap v619: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 759 op/s 2024-03-19T01:21:31.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:31 smithi156 ceph-mon[66799]: pgmap v619: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.3 MiB/s wr, 759 op/s 2024-03-19T01:21:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:33 smithi162 ceph-mon[67178]: pgmap v620: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 648 op/s 2024-03-19T01:21:33.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:33 smithi116 ceph-mon[62701]: pgmap v620: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 648 op/s 2024-03-19T01:21:33.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:33 smithi156 ceph-mon[66799]: pgmap v620: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 648 op/s 2024-03-19T01:21:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.443+0000 7f02b8c7e640 1 -- 172.21.15.116:0/1424930982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02b4102a70 msgr2=0x7f02b4102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.443+0000 7f02b8c7e640 1 --2- 172.21.15.116:0/1424930982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02b4102a70 0x7f02b4102ef0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f02a400fb10 tx=0x7f02a402fdb0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.443+0000 7f02b8c7e640 1 -- 172.21.15.116:0/1424930982 shutdown_connections 2024-03-19T01:21:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.443+0000 7f02b8c7e640 1 --2- 172.21.15.116:0/1424930982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02b4103430 0x7f02b4109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.443+0000 7f02b8c7e640 1 --2- 172.21.15.116:0/1424930982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02b4102a70 0x7f02b4102ef0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.443+0000 7f02b8c7e640 1 --2- 172.21.15.116:0/1424930982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02b4101880 0x7f02b4101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.444 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.443+0000 7f02b8c7e640 1 -- 172.21.15.116:0/1424930982 >> 172.21.15.116:0/1424930982 conn(0x7f02b40fcdf0 msgr2=0x7f02b40ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.443+0000 7f02b8c7e640 1 -- 172.21.15.116:0/1424930982 shutdown_connections 2024-03-19T01:21:34.445 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.443+0000 7f02b8c7e640 1 -- 172.21.15.116:0/1424930982 wait complete. 2024-03-19T01:21:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.444+0000 7f02b8c7e640 1 Processor -- start 2024-03-19T01:21:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.444+0000 7f02b8c7e640 1 -- start start 2024-03-19T01:21:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.444+0000 7f02b8c7e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02b4101880 0x7f02b4195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.444+0000 7f02b8c7e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02b4102a70 0x7f02b4195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.444+0000 7f02b8c7e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02b4103430 0x7f02b419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.444+0000 7f02b8c7e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02b410b320 con 0x7f02b4101880 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.444+0000 7f02b8c7e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f02b410b1a0 con 0x7f02b4103430 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.444+0000 7f02b8c7e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02b410b4a0 con 0x7f02b4102a70 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.444+0000 7f02b259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02b4101880 0x7f02b4195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.444+0000 7f02b259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02b4101880 0x7f02b4195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47618/0 (socket says 172.21.15.116:47618) 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.444+0000 7f02b259c640 1 -- 172.21.15.116:0/2685175314 learned_addr learned my addr 172.21.15.116:0/2685175314 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.444+0000 7f02b1d9b640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02b4102a70 0x7f02b4195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.444+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02b4103430 0x7f02b419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.445+0000 7f02b259c640 1 -- 172.21.15.116:0/2685175314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02b4102a70 msgr2=0x7f02b4195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.445+0000 7f02b259c640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02b4102a70 0x7f02b4195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.445+0000 7f02b259c640 1 -- 172.21.15.116:0/2685175314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02b4103430 msgr2=0x7f02b419ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.445+0000 7f02b259c640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02b4103430 0x7f02b419ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.445+0000 7f02b259c640 1 -- 172.21.15.116:0/2685175314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b419d500 con 0x7f02b4101880 2024-03-19T01:21:34.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.445+0000 7f02b259c640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02b4101880 0x7f02b4195930 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f029c00e990 tx=0x7f029c00ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.445+0000 7f02ab7fe640 1 -- 172.21.15.116:0/2685175314 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c00cc60 con 0x7f02b4101880 2024-03-19T01:21:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.445+0000 7f02ab7fe640 1 -- 172.21.15.116:0/2685175314 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f029c00ce00 con 0x7f02b4101880 2024-03-19T01:21:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.445+0000 7f02b8c7e640 1 -- 172.21.15.116:0/2685175314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b419d7f0 con 0x7f02b4101880 2024-03-19T01:21:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.445+0000 7f02ab7fe640 1 -- 172.21.15.116:0/2685175314 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c011470 con 0x7f02b4101880 2024-03-19T01:21:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.445+0000 7f02b8c7e640 1 -- 172.21.15.116:0/2685175314 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b419dd10 con 0x7f02b4101880 2024-03-19T01:21:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.448+0000 7f02bb867740 1 Processor -- start 2024-03-19T01:21:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.448+0000 7f02bb867740 1 -- start start 2024-03-19T01:21:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02bb867740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd749520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02bb867740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 0x5561cd74e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02bb867740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd7510f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.451 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02bb867740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5561cd6adbe0 con 0x5561cd749100 2024-03-19T01:21:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02bb867740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5561cd656960 con 0x5561cd749af0 2024-03-19T01:21:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02bb867740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5561cd6a9c80 con 0x5561cd74ec60 2024-03-19T01:21:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd749520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd749520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47620/0 (socket says 172.21.15.116:47620) 2024-03-19T01:21:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02b2d9d640 1 -- 172.21.15.116:0/1372610530 learned_addr learned my addr 172.21.15.116:0/1372610530 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02ab7fe640 1 -- 172.21.15.116:0/2685175314 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f029c004000 con 0x7f02b4101880 2024-03-19T01:21:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02b1d9b640 1 --2- 172.21.15.116:0/1372610530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd7510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02b259c640 1 --2- 172.21.15.116:0/1372610530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 0x5561cd74e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02b2d9d640 1 -- 172.21.15.116:0/1372610530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 msgr2=0x5561cd7510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/1372610530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd7510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02b2d9d640 1 -- 172.21.15.116:0/1372610530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 msgr2=0x5561cd74e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/1372610530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 0x5561cd74e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.452 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02b2d9d640 1 -- 172.21.15.116:0/1372610530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561cd64dd20 con 0x5561cd749100 2024-03-19T01:21:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.449+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/1372610530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd749520 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f02ac009f60 tx=0x7f02ac009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a819f315df0b26d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.450+0000 7f02a8ff9640 1 -- 172.21.15.116:0/1372610530 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac01e070 con 0x5561cd749100 2024-03-19T01:21:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.450+0000 7f02a8ff9640 1 -- 172.21.15.116:0/1372610530 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02ac003c30 con 0x5561cd749100 2024-03-19T01:21:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.450+0000 7f02a8ff9640 1 -- 172.21.15.116:0/1372610530 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac00f980 con 0x5561cd749100 2024-03-19T01:21:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.450+0000 7f02ab7fe640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02a00a28a0 0x7f02a00a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.450+0000 7f02ab7fe640 1 -- 172.21.15.116:0/2685175314 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f029c0ceec0 con 0x7f02b4101880 2024-03-19T01:21:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.451+0000 7f02b259c640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02a00a28a0 0x7f02a00a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.451+0000 7f02bb867740 1 -- 172.21.15.116:0/1372610530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 msgr2=0x5561cd749520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:34.453 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.451+0000 7f02bb867740 1 --2- 172.21.15.116:0/1372610530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd749520 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f02ac009f60 tx=0x7f02ac009f90 comp rx=0 tx=0).stop 2024-03-19T01:21:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.451+0000 7f02bb867740 1 -- 172.21.15.116:0/1372610530 shutdown_connections 2024-03-19T01:21:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.451+0000 7f02bb867740 1 --2- 172.21.15.116:0/1372610530 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd7510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.451+0000 7f02bb867740 1 --2- 172.21.15.116:0/1372610530 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 0x5561cd74e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.451+0000 7f02bb867740 1 --2- 172.21.15.116:0/1372610530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd749520 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.451+0000 7f02bb867740 1 -- 172.21.15.116:0/1372610530 >> 172.21.15.116:0/1372610530 conn(0x5561cd744670 msgr2=0x5561cd744a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.451+0000 7f02bb867740 1 -- 172.21.15.116:0/1372610530 shutdown_connections 2024-03-19T01:21:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02b259c640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02a00a28a0 0x7f02a00a4d60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f029c00e960 tx=0x7f029c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:34.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02bb867740 1 -- 172.21.15.116:0/1372610530 wait complete. 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02bb867740 1 -- start start 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02bb867740 1 Processor -- start 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02bb867740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd7530e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02bb867740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 0x5561cd753620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd7530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd7530e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47636/0 (socket says 172.21.15.116:47636) 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02b2d9d640 1 -- 172.21.15.116:0/572304577 learned_addr learned my addr 172.21.15.116:0/572304577 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02bb867740 1 --2- 172.21.15.116:0/572304577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd75a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02bb867740 1 -- 172.21.15.116:0/572304577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5561cd751c10 con 0x5561cd749100 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02bb867740 1 -- 172.21.15.116:0/572304577 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5561cd669db0 con 0x5561cd749af0 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02bb867740 1 -- 172.21.15.116:0/572304577 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5561cd751d90 con 0x5561cd74ec60 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.452+0000 7f02b1d9b640 1 --2- 172.21.15.116:0/572304577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 0x5561cd753620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.453+0000 7f02bb867740 10 client.0 _ll_register_callbacks cb 0x5561cd66ee70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:21:34.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.453+0000 7f02b2d9d640 1 -- 172.21.15.116:0/572304577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 msgr2=0x5561cd75a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.453+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/572304577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd75a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.453+0000 7f02b2d9d640 1 -- 172.21.15.116:0/572304577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 msgr2=0x5561cd753620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.453+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/572304577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 0x5561cd753620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.453+0000 7f02b2d9d640 1 -- 172.21.15.116:0/572304577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561cd668c60 con 0x5561cd749100 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.454+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/572304577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd7530e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f02ac011d60 tx=0x7f02ac004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a4c14068863f15 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.454+0000 7f0299ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.454+0000 7f0299ffb640 1 -- 172.21.15.116:0/572304577 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac025b90 con 0x5561cd749100 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.454+0000 7f0299ffb640 1 -- 172.21.15.116:0/572304577 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f02ac025d30 con 0x5561cd749100 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.454+0000 7f02bb867740 1 -- 172.21.15.116:0/572304577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02ac003c30 con 0x5561cd749100 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.454+0000 7f0299ffb640 1 -- 172.21.15.116:0/572304577 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f02ac004050 con 0x5561cd749100 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.454+0000 7f02bb867740 10 client.15999 fetch_fsmap learned FSMap version 45 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.454+0000 7f02bb867740 1 -- 172.21.15.116:0/572304577 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5561cd3c2210 con 0x5561cd749100 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.455+0000 7f0299ffb640 1 -- 172.21.15.116:0/572304577 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02ac003c30 con 0x5561cd749100 2024-03-19T01:21:34.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.455+0000 7f0299ffb640 1 -- 172.21.15.116:0/572304577 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f02ac005040 con 0x5561cd749100 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.455+0000 7f02bb867740 10 client.15999 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.455+0000 7f02bb867740 10 client.15999 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.455+0000 7f02bb867740 20 client.15999 populate_metadata read hostname 'smithi116' 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.455+0000 7f02bb867740 1 --2- 172.21.15.116:0/572304577 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5561cd75ec40 0x5561cd761070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.455+0000 7f02bb867740 4 client.15999 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.455+0000 7f02bb867740 1 -- 172.21.15.116:0/572304577 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561cd6bb120 con 0x5561cd75ec40 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.455+0000 7f02b1d9b640 1 --2- 172.21.15.116:0/572304577 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5561cd75ec40 0x5561cd761070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.456+0000 7f02b1d9b640 1 --2- 172.21.15.116:0/572304577 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5561cd75ec40 0x5561cd761070 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16ab8d8957049f27 server_cookie=a90994bb43e369a2 in_seq=0 out_seq=0 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.456+0000 7f0299ffb640 10 client.15999 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f0299ffb640 1 -- 172.21.15.116:0/572304577 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5561cd6bb120 con 0x5561cd75ec40 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f0299ffb640 10 client.15999 handle_command_reply: tid=0 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f02bb867740 1 client.15999 shutdown 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f02bb867740 1 -- 172.21.15.116:0/572304577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 msgr2=0x5561cd7530e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f02bb867740 1 --2- 172.21.15.116:0/572304577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd7530e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f02ac011d60 tx=0x7f02ac004f60 comp rx=0 tx=0).stop 2024-03-19T01:21:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f02bb867740 1 -- 172.21.15.116:0/572304577 shutdown_connections 2024-03-19T01:21:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f02bb867740 1 --2- 172.21.15.116:0/572304577 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5561cd75ec40 0x5561cd761070 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f02bb867740 1 --2- 172.21.15.116:0/572304577 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd75a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f02bb867740 1 --2- 172.21.15.116:0/572304577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 0x5561cd753620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f02bb867740 1 --2- 172.21.15.116:0/572304577 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd7530e0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f02bb867740 1 -- 172.21.15.116:0/572304577 >> 172.21.15.116:0/572304577 conn(0x5561cd744670 msgr2=0x5561cd7458c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f0299ffb640 0 client.15999 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f02bb867740 1 -- 172.21.15.116:0/572304577 shutdown_connections 2024-03-19T01:21:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f02bb867740 1 -- 172.21.15.116:0/572304577 wait complete. 2024-03-19T01:21:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.457+0000 7f02bb867740 20 client.15999 trim_cache size 0 max 16384 2024-03-19T01:21:34.514 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.514+0000 7f02bb867740 1 Processor -- start 2024-03-19T01:21:34.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.516+0000 7f02bb867740 1 -- start start 2024-03-19T01:21:34.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.517+0000 7f02bb867740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd760c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.517+0000 7f02bb867740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 0x5561cd75a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.517+0000 7f02bb867740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd753400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.517+0000 7f02bb867740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02b41a6bb0 con 0x5561cd749100 2024-03-19T01:21:34.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.517+0000 7f02bb867740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f02b41a86f0 con 0x5561cd749af0 2024-03-19T01:21:34.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.517+0000 7f02bb867740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02b41a7280 con 0x5561cd74ec60 2024-03-19T01:21:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.517+0000 7f02b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd760c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.517+0000 7f02b2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd760c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47652/0 (socket says 172.21.15.116:47652) 2024-03-19T01:21:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.517+0000 7f02b2d9d640 1 -- 172.21.15.116:0/1592840125 learned_addr learned my addr 172.21.15.116:0/1592840125 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:34.518 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.517+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/1592840125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd753400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.518+0000 7f02b1d9b640 1 --2- 172.21.15.116:0/1592840125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 0x5561cd75a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.518+0000 7f02b2d9d640 1 -- 172.21.15.116:0/1592840125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 msgr2=0x5561cd753400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.518+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/1592840125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd753400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.519 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.518+0000 7f02b2d9d640 1 -- 172.21.15.116:0/1592840125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 msgr2=0x5561cd75a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.518+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/1592840125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 0x5561cd75a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.520 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.519+0000 7f02b2d9d640 1 -- 172.21.15.116:0/1592840125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561cd48e620 con 0x5561cd749100 2024-03-19T01:21:34.522 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.521+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/1592840125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd760c50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f02ac005d70 tx=0x7f02ac009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc6bc0d23aeba94 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:34.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.523+0000 7f029bfff640 1 -- 172.21.15.116:0/1592840125 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac01e030 con 0x5561cd749100 2024-03-19T01:21:34.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.523+0000 7f029bfff640 1 -- 172.21.15.116:0/1592840125 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02ac009e30 con 0x5561cd749100 2024-03-19T01:21:34.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.523+0000 7f029bfff640 1 -- 172.21.15.116:0/1592840125 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac005230 con 0x5561cd749100 2024-03-19T01:21:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.525+0000 7f02bb867740 1 -- 172.21.15.116:0/1592840125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 msgr2=0x5561cd760c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:34.525 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.525+0000 7f02bb867740 1 --2- 172.21.15.116:0/1592840125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd760c50 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f02ac005d70 tx=0x7f02ac009500 comp rx=0 tx=0).stop 2024-03-19T01:21:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.526+0000 7f02bb867740 1 -- 172.21.15.116:0/1592840125 shutdown_connections 2024-03-19T01:21:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.526+0000 7f02bb867740 1 --2- 172.21.15.116:0/1592840125 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd753400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.526+0000 7f02bb867740 1 --2- 172.21.15.116:0/1592840125 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749af0 0x5561cd75a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.526+0000 7f02bb867740 1 --2- 172.21.15.116:0/1592840125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749100 0x5561cd760c50 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.528 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.526+0000 7f02bb867740 1 -- 172.21.15.116:0/1592840125 >> 172.21.15.116:0/1592840125 conn(0x5561cd744670 msgr2=0x5561cd752cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.531+0000 7f02bb867740 1 -- 172.21.15.116:0/1592840125 shutdown_connections 2024-03-19T01:21:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.534+0000 7f02bb867740 1 -- 172.21.15.116:0/1592840125 wait complete. 2024-03-19T01:21:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.534+0000 7f02bb867740 1 -- start start 2024-03-19T01:21:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.535+0000 7f02bb867740 1 Processor -- start 2024-03-19T01:21:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.535+0000 7f02bb867740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749100 0x5561cd7561c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.535+0000 7f02bb867740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749af0 0x5561cd756700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.535+0000 7f02bb867740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd759720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.535+0000 7f02bb867740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02b41a8020 con 0x5561cd749af0 2024-03-19T01:21:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.535+0000 7f02bb867740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f02b41a7950 con 0x5561cd749100 2024-03-19T01:21:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.535+0000 7f02bb867740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02b41a4b90 con 0x5561cd74ec60 2024-03-19T01:21:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.535+0000 7f02b1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749af0 0x5561cd756700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.535+0000 7f02b2d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749100 0x5561cd7561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.535+0000 7f02b2d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749100 0x5561cd7561c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:49206/0 (socket says 172.21.15.116:49206) 2024-03-19T01:21:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.535+0000 7f02b2d9d640 1 -- 172.21.15.116:0/2839338600 learned_addr learned my addr 172.21.15.116:0/2839338600 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.535+0000 7f02bb867740 10 client.0 _ll_register_callbacks cb 0x7f02ac01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:21:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.535+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/2839338600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd759720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.536+0000 7f02b1d9b640 1 -- 172.21.15.116:0/2839338600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 msgr2=0x5561cd759720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.536+0000 7f02b1d9b640 1 --2- 172.21.15.116:0/2839338600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd759720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.536+0000 7f02b1d9b640 1 -- 172.21.15.116:0/2839338600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749100 msgr2=0x5561cd7561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.536+0000 7f02b1d9b640 1 --2- 172.21.15.116:0/2839338600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749100 0x5561cd7561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.536+0000 7f02b1d9b640 1 -- 172.21.15.116:0/2839338600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561cd759d10 con 0x5561cd749af0 2024-03-19T01:21:34.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.536+0000 7f02b1d9b640 1 --2- 172.21.15.116:0/2839338600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749af0 0x5561cd756700 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f02a4006010 tx=0x7f02a4005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac0d588a906a7f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:34.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.537+0000 7f0299ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:21:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.539+0000 7f0299ffb640 1 -- 172.21.15.116:0/2839338600 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4051060 con 0x5561cd749af0 2024-03-19T01:21:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.539+0000 7f0299ffb640 1 -- 172.21.15.116:0/2839338600 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f02a400f1d0 con 0x5561cd749af0 2024-03-19T01:21:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.539+0000 7f02bb867740 1 -- 172.21.15.116:0/2839338600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02ac009e30 con 0x5561cd749af0 2024-03-19T01:21:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.540+0000 7f0299ffb640 1 -- 172.21.15.116:0/2839338600 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f02a400f3d0 con 0x5561cd749af0 2024-03-19T01:21:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.540+0000 7f02bb867740 10 client.16005 fetch_fsmap learned FSMap version 45 2024-03-19T01:21:34.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.540+0000 7f02bb867740 1 -- 172.21.15.116:0/2839338600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5561cd7c14f0 con 0x5561cd749af0 2024-03-19T01:21:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.542+0000 7f0299ffb640 1 -- 172.21.15.116:0/2839338600 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02ac009e30 con 0x5561cd749af0 2024-03-19T01:21:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.542+0000 7f0299ffb640 1 -- 172.21.15.116:0/2839338600 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f02a4044a40 con 0x5561cd749af0 2024-03-19T01:21:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.544+0000 7f02bb867740 10 client.16005 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:21:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.544+0000 7f02bb867740 10 client.16005 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:21:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.544+0000 7f02bb867740 20 client.16005 populate_metadata read hostname 'smithi116' 2024-03-19T01:21:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.544+0000 7f02bb867740 1 --2- 172.21.15.116:0/2839338600 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5561cd7c1e50 0x5561cd7c4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.544+0000 7f02bb867740 4 client.16005 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:21:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.544+0000 7f02bb867740 1 -- 172.21.15.116:0/2839338600 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5561cd6b5470 con 0x5561cd7c1e50 2024-03-19T01:21:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.549+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/2839338600 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5561cd7c1e50 0x5561cd7c4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.550+0000 7f02b2d9d640 1 --2- 172.21.15.116:0/2839338600 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5561cd7c1e50 0x5561cd7c4270 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc15772d7a341d7b server_cookie=ed52296624f2ef0d in_seq=0 out_seq=0 2024-03-19T01:21:34.550 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.550+0000 7f0299ffb640 10 client.16005 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.572+0000 7f0299ffb640 1 -- 172.21.15.116:0/2839338600 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5561cd6b5470 con 0x5561cd7c1e50 2024-03-19T01:21:34.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.573+0000 7f0299ffb640 10 client.16005 handle_command_reply: tid=0 2024-03-19T01:21:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.573+0000 7f02bb867740 1 client.16005 shutdown 2024-03-19T01:21:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.573+0000 7f02bb867740 1 -- 172.21.15.116:0/2839338600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749af0 msgr2=0x5561cd756700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:34.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.573+0000 7f02bb867740 1 --2- 172.21.15.116:0/2839338600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749af0 0x5561cd756700 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f02a4006010 tx=0x7f02a4005e60 comp rx=0 tx=0).stop 2024-03-19T01:21:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.573+0000 7f02bb867740 1 -- 172.21.15.116:0/2839338600 shutdown_connections 2024-03-19T01:21:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.573+0000 7f02bb867740 1 --2- 172.21.15.116:0/2839338600 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5561cd7c1e50 0x5561cd7c4270 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.573+0000 7f02bb867740 1 --2- 172.21.15.116:0/2839338600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5561cd74ec60 0x5561cd759720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.573+0000 7f02bb867740 1 --2- 172.21.15.116:0/2839338600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5561cd749af0 0x5561cd756700 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.573+0000 7f02bb867740 1 --2- 172.21.15.116:0/2839338600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5561cd749100 0x5561cd7561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.573+0000 7f0299ffb640 0 client.16005 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:34.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.573+0000 7f02bb867740 1 -- 172.21.15.116:0/2839338600 >> 172.21.15.116:0/2839338600 conn(0x5561cd744670 msgr2=0x5561cd74a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02bb867740 1 -- 172.21.15.116:0/2839338600 shutdown_connections 2024-03-19T01:21:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02bb867740 1 -- 172.21.15.116:0/2839338600 wait complete. 2024-03-19T01:21:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02bb867740 20 client.16005 trim_cache size 0 max 16384 2024-03-19T01:21:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:21:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:21:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:21:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:21:34.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02b8c7e640 1 -- 172.21.15.116:0/2685175314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02a00a28a0 msgr2=0x7f02a00a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02b8c7e640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02a00a28a0 0x7f02a00a4d60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f029c00e960 tx=0x7f029c000bc0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02b8c7e640 1 -- 172.21.15.116:0/2685175314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02b4101880 msgr2=0x7f02b4195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02b8c7e640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02b4101880 0x7f02b4195930 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f029c00e990 tx=0x7f029c00ed90 comp rx=0 tx=0).stop 2024-03-19T01:21:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02b8c7e640 1 -- 172.21.15.116:0/2685175314 shutdown_connections 2024-03-19T01:21:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02b8c7e640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02a00a28a0 0x7f02a00a4d60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02b8c7e640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02b4103430 0x7f02b419ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02b8c7e640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02b4102a70 0x7f02b4195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02b8c7e640 1 --2- 172.21.15.116:0/2685175314 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02b4101880 0x7f02b4195930 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02b8c7e640 1 -- 172.21.15.116:0/2685175314 >> 172.21.15.116:0/2685175314 conn(0x7f02b40fcdf0 msgr2=0x7f02b4105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02b8c7e640 1 -- 172.21.15.116:0/2685175314 shutdown_connections 2024-03-19T01:21:34.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:34.574+0000 7f02b8c7e640 1 -- 172.21.15.116:0/2685175314 wait complete. 2024-03-19T01:21:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:35 smithi162 ceph-mon[67178]: pgmap v621: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 751 op/s 2024-03-19T01:21:35.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:35 smithi116 ceph-mon[62701]: pgmap v621: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 751 op/s 2024-03-19T01:21:35.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:35 smithi156 ceph-mon[66799]: pgmap v621: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 751 op/s 2024-03-19T01:21:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 530.0 s, 1934.4 tps, lat 51.856 ms stddev 55.133 2024-03-19T01:21:38.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:38 smithi162 ceph-mon[67178]: pgmap v622: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.7 MiB/s wr, 672 op/s 2024-03-19T01:21:38.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:38 smithi116 ceph-mon[62701]: pgmap v622: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.7 MiB/s wr, 672 op/s 2024-03-19T01:21:38.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:38 smithi156 ceph-mon[66799]: pgmap v622: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 7.7 MiB/s wr, 672 op/s 2024-03-19T01:21:40.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:40 smithi162 ceph-mon[67178]: pgmap v623: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.0 MiB/s wr, 688 op/s 2024-03-19T01:21:40.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:40 smithi116 ceph-mon[62701]: pgmap v623: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.0 MiB/s wr, 688 op/s 2024-03-19T01:21:40.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:40 smithi156 ceph-mon[66799]: pgmap v623: 97 pgs: 97 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.0 MiB/s wr, 688 op/s 2024-03-19T01:21:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 535.0 s, 2038.4 tps, lat 48.718 ms stddev 58.870 2024-03-19T01:21:42.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:42 smithi162 ceph-mon[67178]: pgmap v624: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.7 MiB/s wr, 687 op/s 2024-03-19T01:21:42.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:42 smithi116 ceph-mon[62701]: pgmap v624: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.7 MiB/s wr, 687 op/s 2024-03-19T01:21:42.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:42 smithi156 ceph-mon[66799]: pgmap v624: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 6.7 MiB/s wr, 687 op/s 2024-03-19T01:21:44.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:44 smithi162 ceph-mon[67178]: pgmap v625: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.3 MiB/s wr, 579 op/s 2024-03-19T01:21:44.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:44 smithi116 ceph-mon[62701]: pgmap v625: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.3 MiB/s wr, 579 op/s 2024-03-19T01:21:44.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:44 smithi156 ceph-mon[66799]: pgmap v625: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.3 MiB/s wr, 579 op/s 2024-03-19T01:21:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 540.0 s, 1195.6 tps, lat 82.643 ms stddev 87.128 2024-03-19T01:21:46.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:46 smithi162 ceph-mon[67178]: pgmap v626: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 MiB/s wr, 675 op/s 2024-03-19T01:21:46.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:46 smithi116 ceph-mon[62701]: pgmap v626: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 MiB/s wr, 675 op/s 2024-03-19T01:21:46.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:46 smithi156 ceph-mon[66799]: pgmap v626: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 MiB/s wr, 675 op/s 2024-03-19T01:21:47.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.703+0000 7f8390e88640 1 -- 172.21.15.116:0/3763356105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f838c072ad0 msgr2=0x7f838c072ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:47.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.703+0000 7f8390e88640 1 --2- 172.21.15.116:0/3763356105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f838c072ad0 0x7f838c072ef0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f837c01a910 tx=0x7f837c03cc80 comp rx=0 tx=0).stop 2024-03-19T01:21:47.707 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.706+0000 7f8390e88640 1 -- 172.21.15.116:0/3763356105 shutdown_connections 2024-03-19T01:21:47.707 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.706+0000 7f8390e88640 1 --2- 172.21.15.116:0/3763356105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838c109510 0x7f838c10e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.707 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.706+0000 7f8390e88640 1 --2- 172.21.15.116:0/3763356105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f838c073430 0x7f838c108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.707 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.706+0000 7f8390e88640 1 --2- 172.21.15.116:0/3763356105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f838c072ad0 0x7f838c072ef0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.708 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.706+0000 7f8390e88640 1 -- 172.21.15.116:0/3763356105 >> 172.21.15.116:0/3763356105 conn(0x7f838c06e690 msgr2=0x7f838c070ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:47.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.709+0000 7f8390e88640 1 -- 172.21.15.116:0/3763356105 shutdown_connections 2024-03-19T01:21:47.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.711+0000 7f8390e88640 1 -- 172.21.15.116:0/3763356105 wait complete. 2024-03-19T01:21:47.714 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.712+0000 7f8390e88640 1 Processor -- start 2024-03-19T01:21:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.713+0000 7f8390e88640 1 -- start start 2024-03-19T01:21:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.713+0000 7f8390e88640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f838c072ad0 0x7f838c19ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.713+0000 7f8390e88640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f838c073430 0x7f838c19e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.713+0000 7f8390e88640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838c109510 0x7f838c1a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.713+0000 7f8390e88640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f838c111990 con 0x7f838c073430 2024-03-19T01:21:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.713+0000 7f8390e88640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f838c111810 con 0x7f838c109510 2024-03-19T01:21:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.713+0000 7f8390e88640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f838c111b10 con 0x7f838c072ad0 2024-03-19T01:21:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.714+0000 7f838bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838c109510 0x7f838c1a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.714+0000 7f838bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838c109510 0x7f838c1a53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:58696/0 (socket says 172.21.15.116:58696) 2024-03-19T01:21:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.714+0000 7f838bfff640 1 -- 172.21.15.116:0/2498565325 learned_addr learned my addr 172.21.15.116:0/2498565325 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.714+0000 7f838bfff640 1 -- 172.21.15.116:0/2498565325 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f838c072ad0 msgr2=0x7f838c19ddf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:21:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.714+0000 7f838bfff640 1 --2- 172.21.15.116:0/2498565325 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f838c072ad0 0x7f838c19ddf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.714+0000 7f838bfff640 1 -- 172.21.15.116:0/2498565325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f838c073430 msgr2=0x7f838c19e330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:21:47.717 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.714+0000 7f838bfff640 1 --2- 172.21.15.116:0/2498565325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f838c073430 0x7f838c19e330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.714+0000 7f838bfff640 1 -- 172.21.15.116:0/2498565325 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838c1a5a10 con 0x7f838c109510 2024-03-19T01:21:47.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.714+0000 7f838bfff640 1 --2- 172.21.15.116:0/2498565325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838c109510 0x7f838c1a53a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f838400eb10 tx=0x7f838400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:47.718 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.715+0000 7f8388ff9640 1 -- 172.21.15.116:0/2498565325 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8384004b20 con 0x7f838c109510 2024-03-19T01:21:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.715+0000 7f8388ff9640 1 -- 172.21.15.116:0/2498565325 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8384004cc0 con 0x7f838c109510 2024-03-19T01:21:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.715+0000 7f8388ff9640 1 -- 172.21.15.116:0/2498565325 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8384010790 con 0x7f838c109510 2024-03-19T01:21:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.716+0000 7f8390e88640 1 -- 172.21.15.116:0/2498565325 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f838c1a5d00 con 0x7f838c109510 2024-03-19T01:21:47.719 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.716+0000 7f8390e88640 1 -- 172.21.15.116:0/2498565325 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f838c1a6220 con 0x7f838c109510 2024-03-19T01:21:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.719+0000 7f8388ff9640 1 -- 172.21.15.116:0/2498565325 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f8384005000 con 0x7f838c109510 2024-03-19T01:21:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.720+0000 7f8394db4740 1 Processor -- start 2024-03-19T01:21:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.720+0000 7f8394db4740 1 -- start start 2024-03-19T01:21:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.720+0000 7f8388ff9640 1 --2- 172.21.15.116:0/2498565325 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f83740a2a00 0x7f83740a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.720+0000 7f8394db4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d5e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.720+0000 7f8394db4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d5ecd0 0x559e93d637d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.720+0000 7f8394db4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d63e40 0x559e93d662d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.720+0000 7f8394db4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559e93cc2be0 con 0x559e93d63e40 2024-03-19T01:21:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.720+0000 7f8394db4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559e93c6b960 con 0x559e93d5ecd0 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.720+0000 7f8394db4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559e93cbec80 con 0x559e93d5e2e0 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.720+0000 7f838affd640 1 --2- 172.21.15.116:0/2498565325 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f83740a2a00 0x7f83740a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.720+0000 7f8388ff9640 1 -- 172.21.15.116:0/2498565325 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f83840cf140 con 0x7f838c109510 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.720+0000 7f838affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d5ecd0 0x559e93d637d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.720+0000 7f838bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d63e40 0x559e93d662d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.721+0000 7f838bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d63e40 0x559e93d662d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39820/0 (socket says 172.21.15.116:39820) 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.721+0000 7f838affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d5ecd0 0x559e93d637d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:58700/0 (socket says 172.21.15.116:58700) 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.721+0000 7f838b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d5e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.721+0000 7f838bfff640 1 -- 172.21.15.116:0/2894313235 learned_addr learned my addr 172.21.15.116:0/2894313235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.721+0000 7f838bfff640 1 -- 172.21.15.116:0/2894313235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 msgr2=0x559e93d5e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.721+0000 7f838bfff640 1 --2- 172.21.15.116:0/2894313235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d5e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.721+0000 7f838bfff640 1 -- 172.21.15.116:0/2894313235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d5ecd0 msgr2=0x559e93d637d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.721+0000 7f838bfff640 1 --2- 172.21.15.116:0/2894313235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d5ecd0 0x559e93d637d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.721+0000 7f838bfff640 1 -- 172.21.15.116:0/2894313235 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e93c62d20 con 0x559e93d63e40 2024-03-19T01:21:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.721+0000 7f838affd640 1 --2- 172.21.15.116:0/2498565325 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f83740a2a00 0x7f83740a4ec0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f838c19f300 tx=0x7f83800092d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.721+0000 7f838bfff640 1 --2- 172.21.15.116:0/2894313235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d63e40 0x559e93d662d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8384097ec0 tx=0x7f838404a700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35ede86654deb8f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.721+0000 7f8371ffb640 1 -- 172.21.15.116:0/2894313235 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838409b830 con 0x559e93d63e40 2024-03-19T01:21:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.721+0000 7f8371ffb640 1 -- 172.21.15.116:0/2894313235 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f838409ba60 con 0x559e93d63e40 2024-03-19T01:21:47.728 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.722+0000 7f8371ffb640 1 -- 172.21.15.116:0/2894313235 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838404a8e0 con 0x559e93d63e40 2024-03-19T01:21:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.722+0000 7f8394db4740 1 -- 172.21.15.116:0/2894313235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d63e40 msgr2=0x559e93d662d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.722+0000 7f8394db4740 1 --2- 172.21.15.116:0/2894313235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d63e40 0x559e93d662d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8384097ec0 tx=0x7f838404a700 comp rx=0 tx=0).stop 2024-03-19T01:21:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.722+0000 7f8394db4740 1 -- 172.21.15.116:0/2894313235 shutdown_connections 2024-03-19T01:21:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.722+0000 7f8394db4740 1 --2- 172.21.15.116:0/2894313235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d63e40 0x559e93d662d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.722+0000 7f8394db4740 1 --2- 172.21.15.116:0/2894313235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d5ecd0 0x559e93d637d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.722+0000 7f8394db4740 1 --2- 172.21.15.116:0/2894313235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d5e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.722+0000 7f8394db4740 1 -- 172.21.15.116:0/2894313235 >> 172.21.15.116:0/2894313235 conn(0x559e93d59850 msgr2=0x559e93d59c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.723+0000 7f8394db4740 1 -- 172.21.15.116:0/2894313235 shutdown_connections 2024-03-19T01:21:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.723+0000 7f8394db4740 1 -- 172.21.15.116:0/2894313235 wait complete. 2024-03-19T01:21:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.723+0000 7f8394db4740 1 -- start start 2024-03-19T01:21:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.723+0000 7f8394db4740 1 Processor -- start 2024-03-19T01:21:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.723+0000 7f8394db4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d5e2e0 0x559e93d682f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.723+0000 7f8394db4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 0x559e93d68830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f8394db4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d63e40 0x559e93d6f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f838affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 0x559e93d68830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f838b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d5e2e0 0x559e93d682f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f838b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d5e2e0 0x559e93d682f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:58708/0 (socket says 172.21.15.116:58708) 2024-03-19T01:21:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f838b7fe640 1 -- 172.21.15.116:0/2429757686 learned_addr learned my addr 172.21.15.116:0/2429757686 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f8394db4740 1 -- 172.21.15.116:0/2429757686 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559e93d66e90 con 0x559e93d5ecd0 2024-03-19T01:21:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f8394db4740 1 -- 172.21.15.116:0/2429757686 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559e93c7edb0 con 0x559e93d5e2e0 2024-03-19T01:21:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f8394db4740 1 -- 172.21.15.116:0/2429757686 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559e93d67010 con 0x559e93d63e40 2024-03-19T01:21:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f838bfff640 1 --2- 172.21.15.116:0/2429757686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d63e40 0x559e93d6f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f8394db4740 10 client.0 _ll_register_callbacks cb 0x559e93c83e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:21:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f838b7fe640 1 -- 172.21.15.116:0/2429757686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d63e40 msgr2=0x559e93d6f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f838b7fe640 1 --2- 172.21.15.116:0/2429757686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d63e40 0x559e93d6f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f838b7fe640 1 -- 172.21.15.116:0/2429757686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 msgr2=0x559e93d68830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f838b7fe640 1 --2- 172.21.15.116:0/2429757686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 0x559e93d68830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f838b7fe640 1 -- 172.21.15.116:0/2429757686 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e93c7dc60 con 0x559e93d5e2e0 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f838affd640 1 --2- 172.21.15.116:0/2429757686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 0x559e93d68830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f838b7fe640 1 --2- 172.21.15.116:0/2429757686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d5e2e0 0x559e93d682f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f837c01a910 tx=0x7f837c03ddd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2092f4309daac5c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.724+0000 7f836affd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f836affd640 1 -- 172.21.15.116:0/2429757686 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c01a530 con 0x559e93d5e2e0 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f8394db4740 1 -- 172.21.15.116:0/2429757686 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f838409ba60 con 0x559e93d5e2e0 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f836affd640 1 -- 172.21.15.116:0/2429757686 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f837c003800 con 0x559e93d5e2e0 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f836affd640 1 -- 172.21.15.116:0/2429757686 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f837c0046c0 con 0x559e93d5e2e0 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f8394db4740 10 client.25844 fetch_fsmap learned FSMap version 45 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f8394db4740 1 -- 172.21.15.116:0/2429757686 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x559e939d7210 con 0x559e93d5e2e0 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f836affd640 1 -- 172.21.15.116:0/2429757686 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f838409ba60 con 0x559e93d5e2e0 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f836affd640 1 -- 172.21.15.116:0/2429757686 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f837c045c70 con 0x559e93d5e2e0 2024-03-19T01:21:47.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f8394db4740 10 client.25844 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:21:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f8394db4740 10 client.25844 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:21:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f8394db4740 20 client.25844 populate_metadata read hostname 'smithi116' 2024-03-19T01:21:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f8394db4740 1 --2- 172.21.15.116:0/2429757686 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e93d740d0 0x559e93d76560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f8394db4740 4 client.25844 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:21:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.725+0000 7f8394db4740 1 -- 172.21.15.116:0/2429757686 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559e93cd0120 con 0x559e93d740d0 2024-03-19T01:21:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.726+0000 7f838affd640 1 --2- 172.21.15.116:0/2429757686 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e93d740d0 0x559e93d76560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.727+0000 7f836affd640 10 client.25844 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.727+0000 7f838affd640 1 --2- 172.21.15.116:0/2429757686 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e93d740d0 0x559e93d76560 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3573f51bca2acfa server_cookie=b95834c3b68257f7 in_seq=0 out_seq=0 2024-03-19T01:21:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.728+0000 7f836affd640 1 -- 172.21.15.116:0/2429757686 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559e93cd0120 con 0x559e93d740d0 2024-03-19T01:21:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.728+0000 7f836affd640 10 client.25844 handle_command_reply: tid=0 2024-03-19T01:21:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.728+0000 7f8394db4740 1 client.25844 shutdown 2024-03-19T01:21:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.729+0000 7f8394db4740 1 -- 172.21.15.116:0/2429757686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d5e2e0 msgr2=0x559e93d682f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.729+0000 7f8394db4740 1 --2- 172.21.15.116:0/2429757686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d5e2e0 0x559e93d682f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f837c01a910 tx=0x7f837c03ddd0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.729+0000 7f8394db4740 1 -- 172.21.15.116:0/2429757686 shutdown_connections 2024-03-19T01:21:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.729+0000 7f8394db4740 1 --2- 172.21.15.116:0/2429757686 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e93d740d0 0x559e93d76560 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.729+0000 7f8394db4740 1 --2- 172.21.15.116:0/2429757686 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d63e40 0x559e93d6f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.729+0000 7f8394db4740 1 --2- 172.21.15.116:0/2429757686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 0x559e93d68830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.729+0000 7f8394db4740 1 --2- 172.21.15.116:0/2429757686 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d5e2e0 0x559e93d682f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.734 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.729+0000 7f836affd640 0 client.25844 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.729+0000 7f8394db4740 1 -- 172.21.15.116:0/2429757686 >> 172.21.15.116:0/2429757686 conn(0x559e93d59850 msgr2=0x559e93d58ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.729+0000 7f8394db4740 1 -- 172.21.15.116:0/2429757686 shutdown_connections 2024-03-19T01:21:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.729+0000 7f8394db4740 1 -- 172.21.15.116:0/2429757686 wait complete. 2024-03-19T01:21:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.729+0000 7f8394db4740 20 client.25844 trim_cache size 0 max 16384 2024-03-19T01:21:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.750+0000 7f8394db4740 1 Processor -- start 2024-03-19T01:21:47.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.751+0000 7f8394db4740 1 -- start start 2024-03-19T01:21:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.751+0000 7f8394db4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d5f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.751+0000 7f8394db4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 0x559e93d6f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.751+0000 7f8394db4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d63e40 0x559e93d5fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.751+0000 7f8394db4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f838c1af070 con 0x559e93d5ecd0 2024-03-19T01:21:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.751+0000 7f8394db4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f838c1b0bb0 con 0x559e93d63e40 2024-03-19T01:21:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.751+0000 7f8394db4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f838c1af740 con 0x559e93d5e2e0 2024-03-19T01:21:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.751+0000 7f838b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d5f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.751+0000 7f838b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d5f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53642/0 (socket says 172.21.15.116:53642) 2024-03-19T01:21:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.751+0000 7f838b7fe640 1 -- 172.21.15.116:0/1352560706 learned_addr learned my addr 172.21.15.116:0/1352560706 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.751+0000 7f838bfff640 1 --2- 172.21.15.116:0/1352560706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d63e40 0x559e93d5fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.752+0000 7f838affd640 1 --2- 172.21.15.116:0/1352560706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 0x559e93d6f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.752+0000 7f838bfff640 1 -- 172.21.15.116:0/1352560706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 msgr2=0x559e93d5f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.752+0000 7f838bfff640 1 --2- 172.21.15.116:0/1352560706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d5f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.752+0000 7f838bfff640 1 -- 172.21.15.116:0/1352560706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 msgr2=0x559e93d6f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.752+0000 7f838bfff640 1 --2- 172.21.15.116:0/1352560706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 0x559e93d6f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.752+0000 7f838bfff640 1 -- 172.21.15.116:0/1352560706 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559e93aa3620 con 0x559e93d63e40 2024-03-19T01:21:47.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.752+0000 7f838b7fe640 1 --2- 172.21.15.116:0/1352560706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d5f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:21:47.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.752+0000 7f838affd640 1 --2- 172.21.15.116:0/1352560706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 0x559e93d6f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:21:47.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.752+0000 7f838bfff640 1 --2- 172.21.15.116:0/1352560706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d63e40 0x559e93d5fcb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f83840a3cd0 tx=0x7f838409b630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9affb195a5cb81e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:47.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.752+0000 7f83717fa640 1 -- 172.21.15.116:0/1352560706 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83840b9070 con 0x559e93d63e40 2024-03-19T01:21:47.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.752+0000 7f83717fa640 1 -- 172.21.15.116:0/1352560706 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f838409ba60 con 0x559e93d63e40 2024-03-19T01:21:47.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.753+0000 7f83717fa640 1 -- 172.21.15.116:0/1352560706 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8384049070 con 0x559e93d63e40 2024-03-19T01:21:47.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.753+0000 7f8394db4740 1 -- 172.21.15.116:0/1352560706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d63e40 msgr2=0x559e93d5fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.754+0000 7f8394db4740 1 --2- 172.21.15.116:0/1352560706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d63e40 0x559e93d5fcb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f83840a3cd0 tx=0x7f838409b630 comp rx=0 tx=0).stop 2024-03-19T01:21:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.754+0000 7f8394db4740 1 -- 172.21.15.116:0/1352560706 shutdown_connections 2024-03-19T01:21:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.754+0000 7f8394db4740 1 --2- 172.21.15.116:0/1352560706 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d63e40 0x559e93d5fcb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.754+0000 7f8394db4740 1 --2- 172.21.15.116:0/1352560706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 0x559e93d6f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.754+0000 7f8394db4740 1 --2- 172.21.15.116:0/1352560706 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d5f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.754+0000 7f8394db4740 1 -- 172.21.15.116:0/1352560706 >> 172.21.15.116:0/1352560706 conn(0x559e93d59850 msgr2=0x559e93d58ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.754+0000 7f8394db4740 1 -- 172.21.15.116:0/1352560706 shutdown_connections 2024-03-19T01:21:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.754+0000 7f8394db4740 1 -- 172.21.15.116:0/1352560706 wait complete. 2024-03-19T01:21:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.754+0000 7f8394db4740 1 -- start start 2024-03-19T01:21:47.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.754+0000 7f8394db4740 1 Processor -- start 2024-03-19T01:21:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.755+0000 7f8394db4740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d6d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.755+0000 7f8394db4740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 0x559e93d6da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.755+0000 7f8394db4740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d63e40 0x559e93d6ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.755+0000 7f8394db4740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f838c1b04e0 con 0x559e93d5ecd0 2024-03-19T01:21:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.755+0000 7f8394db4740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f838c1afe10 con 0x559e93d63e40 2024-03-19T01:21:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.755+0000 7f8394db4740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f838c1ad050 con 0x559e93d5e2e0 2024-03-19T01:21:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.755+0000 7f8394db4740 10 client.0 _ll_register_callbacks cb 0x7f837c01a790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:21:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.755+0000 7f838b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d6d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.755+0000 7f838b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d6d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:53646/0 (socket says 172.21.15.116:53646) 2024-03-19T01:21:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.755+0000 7f838b7fe640 1 -- 172.21.15.116:0/3675185865 learned_addr learned my addr 172.21.15.116:0/3675185865 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:21:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.755+0000 7f838bfff640 1 --2- 172.21.15.116:0/3675185865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d63e40 0x559e93d6ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.756+0000 7f838b7fe640 1 -- 172.21.15.116:0/3675185865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d63e40 msgr2=0x559e93d6ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.756+0000 7f838b7fe640 1 --2- 172.21.15.116:0/3675185865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d63e40 0x559e93d6ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.756+0000 7f838b7fe640 1 -- 172.21.15.116:0/3675185865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 msgr2=0x559e93d6da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:21:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.756+0000 7f838b7fe640 1 --2- 172.21.15.116:0/3675185865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 0x559e93d6da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.756+0000 7f838b7fe640 1 -- 172.21.15.116:0/3675185865 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559e93d6f000 con 0x559e93d5e2e0 2024-03-19T01:21:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.757+0000 7f838b7fe640 1 --2- 172.21.15.116:0/3675185865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d6d510 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f837c04f430 tx=0x7f837c005e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e72cc21aba5dfb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:21:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.757+0000 7f836affd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:21:47.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.758+0000 7f836affd640 1 -- 172.21.15.116:0/3675185865 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837c03d300 con 0x559e93d5e2e0 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.758+0000 7f836affd640 1 -- 172.21.15.116:0/3675185865 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f837c019040 con 0x559e93d5e2e0 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.758+0000 7f8394db4740 1 -- 172.21.15.116:0/3675185865 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f838409ba60 con 0x559e93d5e2e0 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.758+0000 7f836affd640 1 -- 172.21.15.116:0/3675185865 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f837c072070 con 0x559e93d5e2e0 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.758+0000 7f8394db4740 10 client.25834 fetch_fsmap learned FSMap version 45 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.758+0000 7f8394db4740 1 -- 172.21.15.116:0/3675185865 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x559e93ddaa20 con 0x559e93d5e2e0 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.758+0000 7f836affd640 1 -- 172.21.15.116:0/3675185865 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f838409ba60 con 0x559e93d5e2e0 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.759+0000 7f836affd640 1 -- 172.21.15.116:0/3675185865 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f837c0458e0 con 0x559e93d5e2e0 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.759+0000 7f8394db4740 10 client.25834 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.759+0000 7f8394db4740 10 client.25834 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.759+0000 7f8394db4740 20 client.25834 populate_metadata read hostname 'smithi116' 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.759+0000 7f8394db4740 1 --2- 172.21.15.116:0/3675185865 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e93ddb3a0 0x559e93ddd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.759+0000 7f8394db4740 4 client.25834 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.759+0000 7f8394db4740 1 -- 172.21.15.116:0/3675185865 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559e93cca470 con 0x559e93ddb3a0 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.759+0000 7f838affd640 1 --2- 172.21.15.116:0/3675185865 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e93ddb3a0 0x559e93ddd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:21:47.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.759+0000 7f836affd640 10 client.25834 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:47.764 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.759+0000 7f838affd640 1 --2- 172.21.15.116:0/3675185865 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e93ddb3a0 0x559e93ddd7c0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cd4cb33b980c572 server_cookie=e63f3b7be9bb6428 in_seq=0 out_seq=0 2024-03-19T01:21:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.770+0000 7f836affd640 1 -- 172.21.15.116:0/3675185865 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559e93cca470 con 0x559e93ddb3a0 2024-03-19T01:21:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.770+0000 7f836affd640 10 client.25834 handle_command_reply: tid=0 2024-03-19T01:21:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.770+0000 7f8394db4740 1 client.25834 shutdown 2024-03-19T01:21:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.771+0000 7f8394db4740 1 -- 172.21.15.116:0/3675185865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 msgr2=0x559e93d6d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.771+0000 7f8394db4740 1 --2- 172.21.15.116:0/3675185865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d6d510 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f837c04f430 tx=0x7f837c005e60 comp rx=0 tx=0).stop 2024-03-19T01:21:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.771+0000 7f8394db4740 1 -- 172.21.15.116:0/3675185865 shutdown_connections 2024-03-19T01:21:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.771+0000 7f8394db4740 1 --2- 172.21.15.116:0/3675185865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559e93d63e40 0x559e93d6ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.773 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.771+0000 7f8394db4740 1 --2- 172.21.15.116:0/3675185865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559e93d5ecd0 0x559e93d6da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.771+0000 7f8394db4740 1 --2- 172.21.15.116:0/3675185865 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559e93ddb3a0 0x559e93ddd7c0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.771+0000 7f8394db4740 1 --2- 172.21.15.116:0/3675185865 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559e93d5e2e0 0x559e93d6d510 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.771+0000 7f8394db4740 1 -- 172.21.15.116:0/3675185865 >> 172.21.15.116:0/3675185865 conn(0x559e93d59850 msgr2=0x559e93d5f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.771+0000 7f836affd640 0 client.25834 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:21:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.771+0000 7f8394db4740 1 -- 172.21.15.116:0/3675185865 shutdown_connections 2024-03-19T01:21:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.771+0000 7f8394db4740 1 -- 172.21.15.116:0/3675185865 wait complete. 2024-03-19T01:21:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.772+0000 7f8394db4740 20 client.25834 trim_cache size 0 max 16384 2024-03-19T01:21:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:21:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:21:47.774 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:21:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:21:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.772+0000 7f8390e88640 1 -- 172.21.15.116:0/2498565325 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f83740a2a00 msgr2=0x7f83740a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.772+0000 7f8390e88640 1 --2- 172.21.15.116:0/2498565325 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f83740a2a00 0x7f83740a4ec0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f838c19f300 tx=0x7f83800092d0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.775 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.772+0000 7f8390e88640 1 -- 172.21.15.116:0/2498565325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838c109510 msgr2=0x7f838c1a53a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:21:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.772+0000 7f8390e88640 1 --2- 172.21.15.116:0/2498565325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838c109510 0x7f838c1a53a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f838400eb10 tx=0x7f838400c2d0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.772+0000 7f8390e88640 1 -- 172.21.15.116:0/2498565325 shutdown_connections 2024-03-19T01:21:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.772+0000 7f8390e88640 1 --2- 172.21.15.116:0/2498565325 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f83740a2a00 0x7f83740a4ec0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.772+0000 7f8390e88640 1 --2- 172.21.15.116:0/2498565325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f838c109510 0x7f838c1a53a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.772+0000 7f8390e88640 1 --2- 172.21.15.116:0/2498565325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f838c073430 0x7f838c19e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.772+0000 7f8390e88640 1 --2- 172.21.15.116:0/2498565325 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f838c072ad0 0x7f838c19ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:21:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.772+0000 7f8390e88640 1 -- 172.21.15.116:0/2498565325 >> 172.21.15.116:0/2498565325 conn(0x7f838c06e690 msgr2=0x7f838c10c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:21:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.773+0000 7f8390e88640 1 -- 172.21.15.116:0/2498565325 shutdown_connections 2024-03-19T01:21:47.776 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:21:47.773+0000 7f8390e88640 1 -- 172.21.15.116:0/2498565325 wait complete. 2024-03-19T01:21:48.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:48 smithi162 ceph-mon[67178]: pgmap v627: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.2 MiB/s wr, 584 op/s 2024-03-19T01:21:48.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:48 smithi116 ceph-mon[62701]: pgmap v627: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.2 MiB/s wr, 584 op/s 2024-03-19T01:21:48.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:48 smithi156 ceph-mon[66799]: pgmap v627: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.2 MiB/s wr, 584 op/s 2024-03-19T01:21:50.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:50 smithi162 ceph-mon[67178]: pgmap v628: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 MiB/s wr, 644 op/s 2024-03-19T01:21:50.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:50 smithi116 ceph-mon[62701]: pgmap v628: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 MiB/s wr, 644 op/s 2024-03-19T01:21:50.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:50 smithi156 ceph-mon[66799]: pgmap v628: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.9 MiB/s wr, 644 op/s 2024-03-19T01:21:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 545.0 s, 1257.6 tps, lat 80.442 ms stddev 98.895 2024-03-19T01:21:52.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:52 smithi162 ceph-mon[67178]: pgmap v629: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.2 MiB/s wr, 685 op/s 2024-03-19T01:21:52.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:52 smithi116 ceph-mon[62701]: pgmap v629: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.2 MiB/s wr, 685 op/s 2024-03-19T01:21:52.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:52 smithi156 ceph-mon[66799]: pgmap v629: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.2 MiB/s wr, 685 op/s 2024-03-19T01:21:54.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:54 smithi162 ceph-mon[67178]: pgmap v630: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.9 MiB/s wr, 557 op/s 2024-03-19T01:21:54.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:54 smithi116 ceph-mon[62701]: pgmap v630: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.9 MiB/s wr, 557 op/s 2024-03-19T01:21:54.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:54 smithi156 ceph-mon[66799]: pgmap v630: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 4.9 MiB/s wr, 557 op/s 2024-03-19T01:21:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 550.0 s, 1884.6 tps, lat 53.569 ms stddev 65.585 2024-03-19T01:21:56.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:56 smithi162 ceph-mon[67178]: pgmap v631: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.3 MiB/s wr, 614 op/s 2024-03-19T01:21:56.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:56 smithi116 ceph-mon[62701]: pgmap v631: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.3 MiB/s wr, 614 op/s 2024-03-19T01:21:56.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:56 smithi156 ceph-mon[66799]: pgmap v631: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 5.3 MiB/s wr, 614 op/s 2024-03-19T01:21:58.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:21:58 smithi162 ceph-mon[67178]: pgmap v632: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 5.0 MiB/s wr, 577 op/s 2024-03-19T01:21:58.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:21:58 smithi116 ceph-mon[62701]: pgmap v632: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 5.0 MiB/s wr, 577 op/s 2024-03-19T01:21:58.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:21:58 smithi156 ceph-mon[66799]: pgmap v632: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 5.0 MiB/s wr, 577 op/s 2024-03-19T01:22:00.650 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:00 smithi162 ceph-mon[67178]: pgmap v633: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 MiB/s wr, 597 op/s 2024-03-19T01:22:00.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:00 smithi116 ceph-mon[62701]: pgmap v633: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 MiB/s wr, 597 op/s 2024-03-19T01:22:00.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:00 smithi156 ceph-mon[66799]: pgmap v633: 97 pgs: 97 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 MiB/s wr, 597 op/s 2024-03-19T01:22:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.921+0000 7f9ba1c92640 1 -- 172.21.15.116:0/1269514279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9c10b190 msgr2=0x7f9b9c10d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:00.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.921+0000 7f9ba1c92640 1 --2- 172.21.15.116:0/1269514279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9c10b190 0x7f9b9c10d600 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9b9401acb0 tx=0x7f9b9403d150 comp rx=0 tx=0).stop 2024-03-19T01:22:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.923+0000 7f9ba1c92640 1 -- 172.21.15.116:0/1269514279 shutdown_connections 2024-03-19T01:22:00.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.923+0000 7f9ba1c92640 1 --2- 172.21.15.116:0/1269514279 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b9c10db40 0x7f9b9c10dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.924 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.923+0000 7f9ba1c92640 1 --2- 172.21.15.116:0/1269514279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9c10b190 0x7f9b9c10d600 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.924 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.923+0000 7f9ba1c92640 1 --2- 172.21.15.116:0/1269514279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b9c108820 0x7f9b9c10ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.924 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.923+0000 7f9ba1c92640 1 -- 172.21.15.116:0/1269514279 >> 172.21.15.116:0/1269514279 conn(0x7f9b9c06e860 msgr2=0x7f9b9c070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.928+0000 7f9ba1c92640 1 -- 172.21.15.116:0/1269514279 shutdown_connections 2024-03-19T01:22:00.928 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.928+0000 7f9ba1c92640 1 -- 172.21.15.116:0/1269514279 wait complete. 2024-03-19T01:22:00.929 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.929+0000 7f9ba1c92640 1 Processor -- start 2024-03-19T01:22:00.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.929+0000 7f9ba1c92640 1 -- start start 2024-03-19T01:22:00.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.929+0000 7f9ba1c92640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b9c108820 0x7f9b9c19e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.929+0000 7f9ba1c92640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9c10b190 0x7f9b9c19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.931 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.929+0000 7f9ba1c92640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b9c10db40 0x7f9b9c1a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.930+0000 7f9b9affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9c10b190 0x7f9b9c19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.930+0000 7f9b9affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9c10b190 0x7f9b9c19e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55900/0 (socket says 172.21.15.116:55900) 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.930+0000 7f9ba1c92640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c111af0 con 0x7f9b9c10b190 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.930+0000 7f9ba1c92640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c111970 con 0x7f9b9c108820 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.930+0000 7f9ba1c92640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c111c70 con 0x7f9b9c10db40 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.930+0000 7f9b9affd640 1 -- 172.21.15.116:0/2655397457 learned_addr learned my addr 172.21.15.116:0/2655397457 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.930+0000 7f9b9bfff640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b9c10db40 0x7f9b9c1a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.930+0000 7f9b9bfff640 1 -- 172.21.15.116:0/2655397457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b9c108820 msgr2=0x7f9b9c19e090 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.930+0000 7f9b9bfff640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b9c108820 0x7f9b9c19e090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.930+0000 7f9b9bfff640 1 -- 172.21.15.116:0/2655397457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9c10b190 msgr2=0x7f9b9c19e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.930+0000 7f9b9bfff640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9c10b190 0x7f9b9c19e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.930+0000 7f9b9bfff640 1 -- 172.21.15.116:0/2655397457 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9c1a5cb0 con 0x7f9b9c10db40 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.931+0000 7f9b9bfff640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b9c10db40 0x7f9b9c1a5640 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00e970 tx=0x7f9b8c00ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:00.932 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.931+0000 7f9b9affd640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9c10b190 0x7f9b9c19e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:22:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.932+0000 7f9b98ff9640 1 -- 172.21.15.116:0/2655397457 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c00ccc0 con 0x7f9b9c10db40 2024-03-19T01:22:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.932+0000 7f9b98ff9640 1 -- 172.21.15.116:0/2655397457 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b8c005020 con 0x7f9b9c10db40 2024-03-19T01:22:00.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.932+0000 7f9b98ff9640 1 -- 172.21.15.116:0/2655397457 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c004a00 con 0x7f9b9c10db40 2024-03-19T01:22:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.932+0000 7f9ba1c92640 1 -- 172.21.15.116:0/2655397457 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b9c1a5fa0 con 0x7f9b9c10db40 2024-03-19T01:22:00.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.932+0000 7f9ba1c92640 1 -- 172.21.15.116:0/2655397457 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b9c1a6410 con 0x7f9b9c10db40 2024-03-19T01:22:00.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.936+0000 7f9ba487d740 1 Processor -- start 2024-03-19T01:22:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.936+0000 7f9ba487d740 1 -- start start 2024-03-19T01:22:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.936+0000 7f9ba487d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc81120 0x55e89cc81540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.936+0000 7f9ba487d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc81b10 0x55e89cc86610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.936+0000 7f9ba487d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc86c80 0x55e89cc89110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.936+0000 7f9ba487d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e89cbe5be0 con 0x55e89cc81120 2024-03-19T01:22:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.936+0000 7f9ba487d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e89cb8e960 con 0x55e89cc81b10 2024-03-19T01:22:00.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.936+0000 7f9ba487d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e89cbe1c80 con 0x55e89cc86c80 2024-03-19T01:22:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.938+0000 7f9b9affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc86c80 0x55e89cc89110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.938+0000 7f9b9affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc86c80 0x55e89cc89110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55910/0 (socket says 172.21.15.116:55910) 2024-03-19T01:22:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.938+0000 7f9b9affd640 1 -- 172.21.15.116:0/2259537883 learned_addr learned my addr 172.21.15.116:0/2259537883 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.938+0000 7f9b9affd640 1 --2- 172.21.15.116:0/2259537883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc81120 0x55e89cc81540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.938+0000 7f9b9b7fe640 1 --2- 172.21.15.116:0/2259537883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc81b10 0x55e89cc86610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.939+0000 7f9b98ff9640 1 -- 172.21.15.116:0/2655397457 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f9b8c0051d0 con 0x7f9b9c10db40 2024-03-19T01:22:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.939+0000 7f9b9affd640 1 -- 172.21.15.116:0/2259537883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc81b10 msgr2=0x55e89cc86610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.939+0000 7f9b9affd640 1 --2- 172.21.15.116:0/2259537883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc81b10 0x55e89cc86610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.939+0000 7f9b9affd640 1 -- 172.21.15.116:0/2259537883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc81120 msgr2=0x55e89cc81540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.939+0000 7f9b9affd640 1 --2- 172.21.15.116:0/2259537883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc81120 0x55e89cc81540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.939+0000 7f9b9affd640 1 -- 172.21.15.116:0/2259537883 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e89cb85d20 con 0x55e89cc86c80 2024-03-19T01:22:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.940+0000 7f9b9affd640 1 --2- 172.21.15.116:0/2259537883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc86c80 0x55e89cc89110 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9b94044000 tx=0x7f9b94004a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2abcfd1810252013 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:00.940 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.940+0000 7f9b98ff9640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9b840a2ae0 0x7f9b840a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.940+0000 7f9b81ffb640 1 -- 172.21.15.116:0/2259537883 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9404a030 con 0x55e89cc86c80 2024-03-19T01:22:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.940+0000 7f9b9b7fe640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9b840a2ae0 0x7f9b840a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.941+0000 7f9b81ffb640 1 -- 172.21.15.116:0/2259537883 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b9404b040 con 0x55e89cc86c80 2024-03-19T01:22:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.942+0000 7f9ba487d740 1 -- 172.21.15.116:0/2259537883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc86c80 msgr2=0x55e89cc89110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:00.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.942+0000 7f9ba487d740 1 --2- 172.21.15.116:0/2259537883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc86c80 0x55e89cc89110 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9b94044000 tx=0x7f9b94004a80 comp rx=0 tx=0).stop 2024-03-19T01:22:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.943+0000 7f9b98ff9640 1 -- 172.21.15.116:0/2655397457 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f9b8c0ce9a0 con 0x7f9b9c10db40 2024-03-19T01:22:00.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.943+0000 7f9b9b7fe640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9b840a2ae0 0x7f9b840a4fa0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55e89cc81f10 tx=0x7f9b90005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.944+0000 7f9ba487d740 1 -- 172.21.15.116:0/2259537883 shutdown_connections 2024-03-19T01:22:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.944+0000 7f9ba487d740 1 --2- 172.21.15.116:0/2259537883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc86c80 0x55e89cc89110 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.944+0000 7f9ba487d740 1 --2- 172.21.15.116:0/2259537883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc81b10 0x55e89cc86610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.944+0000 7f9ba487d740 1 --2- 172.21.15.116:0/2259537883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc81120 0x55e89cc81540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.944+0000 7f9ba487d740 1 -- 172.21.15.116:0/2259537883 >> 172.21.15.116:0/2259537883 conn(0x55e89cc7c690 msgr2=0x55e89cc7ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:00.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.951+0000 7f9ba487d740 1 -- 172.21.15.116:0/2259537883 shutdown_connections 2024-03-19T01:22:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.951+0000 7f9ba487d740 1 -- 172.21.15.116:0/2259537883 wait complete. 2024-03-19T01:22:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.951+0000 7f9ba487d740 1 -- start start 2024-03-19T01:22:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.952+0000 7f9ba487d740 1 Processor -- start 2024-03-19T01:22:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.952+0000 7f9ba487d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc81120 0x55e89cc8b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.952+0000 7f9ba487d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc81b10 0x55e89cc8b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.952+0000 7f9ba487d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc86c80 0x55e89cc925a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.952+0000 7f9ba487d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55e89cc89bc0 con 0x55e89cc86c80 2024-03-19T01:22:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.952+0000 7f9ba487d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55e89cba1db0 con 0x55e89cc81120 2024-03-19T01:22:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.952+0000 7f9ba487d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55e89cc89d40 con 0x55e89cc81b10 2024-03-19T01:22:00.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.952+0000 7f9ba487d740 10 client.0 _ll_register_callbacks cb 0x55e89cba6e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:22:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.953+0000 7f9b9b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc81120 0x55e89cc8b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.953+0000 7f9b9b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc81120 0x55e89cc8b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:48618/0 (socket says 172.21.15.116:48618) 2024-03-19T01:22:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.953+0000 7f9b9b7fe640 1 -- 172.21.15.116:0/2781656863 learned_addr learned my addr 172.21.15.116:0/2781656863 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.953+0000 7f9b9affd640 1 --2- 172.21.15.116:0/2781656863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc81b10 0x55e89cc8b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.953+0000 7f9b9b7fe640 1 -- 172.21.15.116:0/2781656863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc81b10 msgr2=0x55e89cc8b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.953+0000 7f9b9b7fe640 1 --2- 172.21.15.116:0/2781656863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc81b10 0x55e89cc8b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.953+0000 7f9b9b7fe640 1 -- 172.21.15.116:0/2781656863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc86c80 msgr2=0x55e89cc925a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.953+0000 7f9b9b7fe640 1 --2- 172.21.15.116:0/2781656863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc86c80 0x55e89cc925a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.953+0000 7f9b9b7fe640 1 -- 172.21.15.116:0/2781656863 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e89cba0c60 con 0x55e89cc81120 2024-03-19T01:22:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.953+0000 7f9b9b7fe640 1 --2- 172.21.15.116:0/2781656863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc81120 0x55e89cc8b020 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9b90003870 tx=0x7f9b90004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fed26100d07764cf server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.953+0000 7f9b7affd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:22:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9b7affd640 1 -- 172.21.15.116:0/2781656863 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9000ecc0 con 0x55e89cc81120 2024-03-19T01:22:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9b7affd640 1 -- 172.21.15.116:0/2781656863 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f9b9001e420 con 0x55e89cc81120 2024-03-19T01:22:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9ba487d740 1 -- 172.21.15.116:0/2781656863 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b9404b040 con 0x55e89cc81120 2024-03-19T01:22:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9b7affd640 1 -- 172.21.15.116:0/2781656863 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f9b9002c070 con 0x55e89cc81120 2024-03-19T01:22:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9ba487d740 10 client.25859 fetch_fsmap learned FSMap version 45 2024-03-19T01:22:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9ba487d740 1 -- 172.21.15.116:0/2781656863 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55e89c8fa210 con 0x55e89cc81120 2024-03-19T01:22:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9b7affd640 1 -- 172.21.15.116:0/2781656863 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b9404b040 con 0x55e89cc81120 2024-03-19T01:22:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9b7affd640 1 -- 172.21.15.116:0/2781656863 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f9b90026590 con 0x55e89cc81120 2024-03-19T01:22:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9ba487d740 10 client.25859 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:22:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9ba487d740 10 client.25859 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:22:00.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9ba487d740 20 client.25859 populate_metadata read hostname 'smithi116' 2024-03-19T01:22:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9ba487d740 1 --2- 172.21.15.116:0/2781656863 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e89cc96f40 0x55e89cc99400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9ba487d740 4 client.25859 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:22:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.954+0000 7f9ba487d740 1 -- 172.21.15.116:0/2781656863 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e89cbf3120 con 0x55e89cc96f40 2024-03-19T01:22:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.955+0000 7f9b9affd640 1 --2- 172.21.15.116:0/2781656863 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e89cc96f40 0x55e89cc99400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.955+0000 7f9b7affd640 10 client.25859 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:00.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.955+0000 7f9b9affd640 1 --2- 172.21.15.116:0/2781656863 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e89cc96f40 0x55e89cc99400 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=797365c2286fb68c server_cookie=2f20c4b8e4243c1e in_seq=0 out_seq=0 2024-03-19T01:22:00.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.957+0000 7f9b7affd640 1 -- 172.21.15.116:0/2781656863 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e89cbf3120 con 0x55e89cc96f40 2024-03-19T01:22:00.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.957+0000 7f9b7affd640 10 client.25859 handle_command_reply: tid=0 2024-03-19T01:22:00.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.958+0000 7f9ba487d740 1 client.25859 shutdown 2024-03-19T01:22:00.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.958+0000 7f9ba487d740 1 -- 172.21.15.116:0/2781656863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc81120 msgr2=0x55e89cc8b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:00.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.958+0000 7f9ba487d740 1 --2- 172.21.15.116:0/2781656863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc81120 0x55e89cc8b020 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9b90003870 tx=0x7f9b90004730 comp rx=0 tx=0).stop 2024-03-19T01:22:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.958+0000 7f9ba487d740 1 -- 172.21.15.116:0/2781656863 shutdown_connections 2024-03-19T01:22:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.958+0000 7f9ba487d740 1 --2- 172.21.15.116:0/2781656863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc86c80 0x55e89cc925a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.958+0000 7f9ba487d740 1 --2- 172.21.15.116:0/2781656863 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e89cc96f40 0x55e89cc99400 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.958+0000 7f9ba487d740 1 --2- 172.21.15.116:0/2781656863 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc81b10 0x55e89cc8b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.958+0000 7f9ba487d740 1 --2- 172.21.15.116:0/2781656863 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc81120 0x55e89cc8b020 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9b90003870 tx=0x7f9b90004730 comp rx=0 tx=0).stop 2024-03-19T01:22:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.958+0000 7f9ba487d740 1 -- 172.21.15.116:0/2781656863 >> 172.21.15.116:0/2781656863 conn(0x55e89cc7c690 msgr2=0x55e89cc7d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.958+0000 7f9b7affd640 0 client.25859 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.958+0000 7f9ba487d740 1 -- 172.21.15.116:0/2781656863 shutdown_connections 2024-03-19T01:22:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.959+0000 7f9ba487d740 1 -- 172.21.15.116:0/2781656863 wait complete. 2024-03-19T01:22:00.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.959+0000 7f9ba487d740 20 client.25859 trim_cache size 0 max 16384 2024-03-19T01:22:00.983 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.981+0000 7f9ba487d740 1 Processor -- start 2024-03-19T01:22:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.981+0000 7f9ba487d740 1 -- start start 2024-03-19T01:22:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.982+0000 7f9ba487d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc81b10 0x55e89cc8ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.982+0000 7f9ba487d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc86c80 0x55e89cc92560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.982+0000 7f9ba487d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc8b170 0x55e89cc82200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.982+0000 7f9ba487d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c1af2f0 con 0x55e89cc8b170 2024-03-19T01:22:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.982+0000 7f9ba487d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c1b0e30 con 0x55e89cc86c80 2024-03-19T01:22:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.982+0000 7f9ba487d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c1af9c0 con 0x55e89cc81b10 2024-03-19T01:22:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.982+0000 7f9b9b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc81b10 0x55e89cc8ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.982+0000 7f9b9b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc81b10 0x55e89cc8ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55936/0 (socket says 172.21.15.116:55936) 2024-03-19T01:22:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.982+0000 7f9b9b7fe640 1 -- 172.21.15.116:0/3633391486 learned_addr learned my addr 172.21.15.116:0/3633391486 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.982+0000 7f9b9affd640 1 --2- 172.21.15.116:0/3633391486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc86c80 0x55e89cc92560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.982+0000 7f9b9bfff640 1 --2- 172.21.15.116:0/3633391486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc8b170 0x55e89cc82200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.984 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.983+0000 7f9b9affd640 1 -- 172.21.15.116:0/3633391486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc81b10 msgr2=0x55e89cc8ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.983+0000 7f9b9affd640 1 --2- 172.21.15.116:0/3633391486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc81b10 0x55e89cc8ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.983+0000 7f9b9affd640 1 -- 172.21.15.116:0/3633391486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc8b170 msgr2=0x55e89cc82200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.983+0000 7f9b9affd640 1 --2- 172.21.15.116:0/3633391486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc8b170 0x55e89cc82200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.983+0000 7f9b9affd640 1 -- 172.21.15.116:0/3633391486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e89c9c6620 con 0x55e89cc86c80 2024-03-19T01:22:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.983+0000 7f9b9b7fe640 1 --2- 172.21.15.116:0/3633391486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc81b10 0x55e89cc8ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:22:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.983+0000 7f9b9affd640 1 --2- 172.21.15.116:0/3633391486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc86c80 0x55e89cc92560 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9b9403d760 tx=0x7f9b9401a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98bec0c143c924a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.983+0000 7f9b817fa640 1 -- 172.21.15.116:0/3633391486 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9404a040 con 0x55e89cc86c80 2024-03-19T01:22:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.983+0000 7f9b817fa640 1 -- 172.21.15.116:0/3633391486 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b94067020 con 0x55e89cc86c80 2024-03-19T01:22:00.985 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.983+0000 7f9b817fa640 1 -- 172.21.15.116:0/3633391486 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b940437d0 con 0x55e89cc86c80 2024-03-19T01:22:00.988 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.984+0000 7f9ba487d740 1 -- 172.21.15.116:0/3633391486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc86c80 msgr2=0x55e89cc92560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:00.988 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.984+0000 7f9ba487d740 1 --2- 172.21.15.116:0/3633391486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc86c80 0x55e89cc92560 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9b9403d760 tx=0x7f9b9401a230 comp rx=0 tx=0).stop 2024-03-19T01:22:00.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.984+0000 7f9ba487d740 1 -- 172.21.15.116:0/3633391486 shutdown_connections 2024-03-19T01:22:00.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.984+0000 7f9ba487d740 1 --2- 172.21.15.116:0/3633391486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc8b170 0x55e89cc82200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.984+0000 7f9ba487d740 1 --2- 172.21.15.116:0/3633391486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc86c80 0x55e89cc92560 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.984+0000 7f9ba487d740 1 --2- 172.21.15.116:0/3633391486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc81b10 0x55e89cc8ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.984+0000 7f9ba487d740 1 -- 172.21.15.116:0/3633391486 >> 172.21.15.116:0/3633391486 conn(0x55e89cc7c690 msgr2=0x55e89cc7ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:00.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.984+0000 7f9ba487d740 1 -- 172.21.15.116:0/3633391486 shutdown_connections 2024-03-19T01:22:00.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.984+0000 7f9ba487d740 1 -- 172.21.15.116:0/3633391486 wait complete. 2024-03-19T01:22:00.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.984+0000 7f9ba487d740 1 -- start start 2024-03-19T01:22:00.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.984+0000 7f9ba487d740 1 Processor -- start 2024-03-19T01:22:00.989 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9ba487d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc81b10 0x55e89cc90350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9ba487d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc86c80 0x55e89cc90890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9b9b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc81b10 0x55e89cc90350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9ba487d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc8b170 0x55e89cc918a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9b9b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc81b10 0x55e89cc90350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55934/0 (socket says 172.21.15.116:55934) 2024-03-19T01:22:00.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9b9affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc86c80 0x55e89cc90890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9b9affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc86c80 0x55e89cc90890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:48636/0 (socket says 172.21.15.116:48636) 2024-03-19T01:22:00.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9ba487d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c1b0760 con 0x55e89cc81b10 2024-03-19T01:22:00.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9ba487d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c1b0090 con 0x55e89cc86c80 2024-03-19T01:22:00.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9ba487d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c1ad2d0 con 0x55e89cc8b170 2024-03-19T01:22:00.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9b9b7fe640 1 -- 172.21.15.116:0/4141443508 learned_addr learned my addr 172.21.15.116:0/4141443508 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:00.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9b9bfff640 1 --2- 172.21.15.116:0/4141443508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc8b170 0x55e89cc918a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.990 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9ba487d740 10 client.0 _ll_register_callbacks cb 0x7f9b90009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:22:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9b9affd640 1 -- 172.21.15.116:0/4141443508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc8b170 msgr2=0x55e89cc918a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9b9affd640 1 --2- 172.21.15.116:0/4141443508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc8b170 0x55e89cc918a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9b9affd640 1 -- 172.21.15.116:0/4141443508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc81b10 msgr2=0x55e89cc90350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9b9affd640 1 --2- 172.21.15.116:0/4141443508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc81b10 0x55e89cc90350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.985+0000 7f9b9affd640 1 -- 172.21.15.116:0/4141443508 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e89cc91e90 con 0x55e89cc86c80 2024-03-19T01:22:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.986+0000 7f9b9affd640 1 --2- 172.21.15.116:0/4141443508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc86c80 0x55e89cc90890 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9b9403d730 tx=0x7f9b9401a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47f4ab2a0051488c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.986+0000 7f9b7affd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:22:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.986+0000 7f9b7affd640 1 -- 172.21.15.116:0/4141443508 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b9404a070 con 0x55e89cc86c80 2024-03-19T01:22:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.986+0000 7f9b7affd640 1 -- 172.21.15.116:0/4141443508 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f9b94043750 con 0x55e89cc86c80 2024-03-19T01:22:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.986+0000 7f9ba487d740 1 -- 172.21.15.116:0/4141443508 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b94067020 con 0x55e89cc86c80 2024-03-19T01:22:00.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.987+0000 7f9b7affd640 1 -- 172.21.15.116:0/4141443508 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f9b94004550 con 0x55e89cc86c80 2024-03-19T01:22:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.987+0000 7f9ba487d740 10 client.25865 fetch_fsmap learned FSMap version 45 2024-03-19T01:22:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.987+0000 7f9ba487d740 1 -- 172.21.15.116:0/4141443508 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55e89ccfd940 con 0x55e89cc86c80 2024-03-19T01:22:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.988+0000 7f9b7affd640 1 -- 172.21.15.116:0/4141443508 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b940046f0 con 0x55e89cc86c80 2024-03-19T01:22:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.988+0000 7f9b7affd640 1 -- 172.21.15.116:0/4141443508 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f9b94063480 con 0x55e89cc86c80 2024-03-19T01:22:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.988+0000 7f9ba487d740 10 client.25865 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:22:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.988+0000 7f9ba487d740 10 client.25865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:22:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.988+0000 7f9ba487d740 20 client.25865 populate_metadata read hostname 'smithi116' 2024-03-19T01:22:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.988+0000 7f9ba487d740 1 --2- 172.21.15.116:0/4141443508 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e89ccfdd70 0x55e89cd00190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.988+0000 7f9ba487d740 4 client.25865 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:22:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.988+0000 7f9ba487d740 1 -- 172.21.15.116:0/4141443508 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f9b94067020 con 0x55e89ccfdd70 2024-03-19T01:22:00.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.988+0000 7f9b9b7fe640 1 --2- 172.21.15.116:0/4141443508 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e89ccfdd70 0x55e89cd00190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.988+0000 7f9b7affd640 10 client.25865 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:00.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.989+0000 7f9b9b7fe640 1 --2- 172.21.15.116:0/4141443508 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e89ccfdd70 0x55e89cd00190 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=862490db7cb48717 server_cookie=bb85edad3d594a32 in_seq=0 out_seq=0 2024-03-19T01:22:00.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.996+0000 7f9b7affd640 1 -- 172.21.15.116:0/4141443508 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f9b94067020 con 0x55e89ccfdd70 2024-03-19T01:22:00.996 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.996+0000 7f9b7affd640 10 client.25865 handle_command_reply: tid=0 2024-03-19T01:22:00.997 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.996+0000 7f9ba487d740 1 client.25865 shutdown 2024-03-19T01:22:00.997 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.996+0000 7f9ba487d740 1 -- 172.21.15.116:0/4141443508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc86c80 msgr2=0x55e89cc90890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:00.997 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.996+0000 7f9ba487d740 1 --2- 172.21.15.116:0/4141443508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc86c80 0x55e89cc90890 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9b9403d730 tx=0x7f9b9401a5d0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.997 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.996+0000 7f9ba487d740 1 -- 172.21.15.116:0/4141443508 shutdown_connections 2024-03-19T01:22:00.997 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.996+0000 7f9ba487d740 1 --2- 172.21.15.116:0/4141443508 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55e89ccfdd70 0x55e89cd00190 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.997 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.996+0000 7f9ba487d740 1 --2- 172.21.15.116:0/4141443508 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55e89cc8b170 0x55e89cc918a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.997 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.996+0000 7f9ba487d740 1 --2- 172.21.15.116:0/4141443508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55e89cc86c80 0x55e89cc90890 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.997 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.996+0000 7f9ba487d740 1 --2- 172.21.15.116:0/4141443508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55e89cc81b10 0x55e89cc90350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:00.997 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.996+0000 7f9ba487d740 1 -- 172.21.15.116:0/4141443508 >> 172.21.15.116:0/4141443508 conn(0x55e89cc7c690 msgr2=0x55e89cc7f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:00.997 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.996+0000 7f9b7affd640 0 client.25865 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:00.998 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.997+0000 7f9ba487d740 1 -- 172.21.15.116:0/4141443508 shutdown_connections 2024-03-19T01:22:00.998 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.997+0000 7f9ba487d740 1 -- 172.21.15.116:0/4141443508 wait complete. 2024-03-19T01:22:00.998 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.998+0000 7f9ba487d740 20 client.25865 trim_cache size 0 max 16384 2024-03-19T01:22:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:22:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:22:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:22:00.999 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:22:01.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.999+0000 7f9ba1c92640 1 -- 172.21.15.116:0/2655397457 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9b840a2ae0 msgr2=0x7f9b840a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:01.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.999+0000 7f9ba1c92640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9b840a2ae0 0x7f9b840a4fa0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55e89cc81f10 tx=0x7f9b90005f90 comp rx=0 tx=0).stop 2024-03-19T01:22:01.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.999+0000 7f9ba1c92640 1 -- 172.21.15.116:0/2655397457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b9c10db40 msgr2=0x7f9b9c1a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:01.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.999+0000 7f9ba1c92640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b9c10db40 0x7f9b9c1a5640 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00e970 tx=0x7f9b8c00ee30 comp rx=0 tx=0).stop 2024-03-19T01:22:01.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.999+0000 7f9ba1c92640 1 -- 172.21.15.116:0/2655397457 shutdown_connections 2024-03-19T01:22:01.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.999+0000 7f9ba1c92640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9b9c10db40 0x7f9b9c1a5640 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:01.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.999+0000 7f9ba1c92640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b9c10b190 0x7f9b9c19e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:01.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.999+0000 7f9ba1c92640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9b840a2ae0 0x7f9b840a4fa0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:01.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.999+0000 7f9ba1c92640 1 --2- 172.21.15.116:0/2655397457 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9b9c108820 0x7f9b9c19e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:01.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.999+0000 7f9ba1c92640 1 -- 172.21.15.116:0/2655397457 >> 172.21.15.116:0/2655397457 conn(0x7f9b9c06e860 msgr2=0x7f9b9c10b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:01.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.999+0000 7f9ba1c92640 1 -- 172.21.15.116:0/2655397457 shutdown_connections 2024-03-19T01:22:01.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:00.999+0000 7f9ba1c92640 1 -- 172.21.15.116:0/2655397457 wait complete. 2024-03-19T01:22:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 555.0 s, 2136.2 tps, lat 46.621 ms stddev 51.386 2024-03-19T01:22:02.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:02 smithi162 ceph-mon[67178]: pgmap v634: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.5 MiB/s wr, 721 op/s 2024-03-19T01:22:02.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:02 smithi116 ceph-mon[62701]: pgmap v634: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.5 MiB/s wr, 721 op/s 2024-03-19T01:22:02.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:02 smithi156 ceph-mon[66799]: pgmap v634: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.5 MiB/s wr, 721 op/s 2024-03-19T01:22:04.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:04 smithi162 ceph-mon[67178]: pgmap v635: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.8 MiB/s wr, 628 op/s 2024-03-19T01:22:04.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:04 smithi116 ceph-mon[62701]: pgmap v635: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.8 MiB/s wr, 628 op/s 2024-03-19T01:22:04.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:04 smithi156 ceph-mon[66799]: pgmap v635: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 6.8 MiB/s wr, 628 op/s 2024-03-19T01:22:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 560.0 s, 2524.8 tps, lat 39.820 ms stddev 46.001 2024-03-19T01:22:06.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:06 smithi162 ceph-mon[67178]: pgmap v636: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.4 MiB/s wr, 701 op/s 2024-03-19T01:22:06.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:06 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:22:06.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:06 smithi116 ceph-mon[62701]: pgmap v636: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.4 MiB/s wr, 701 op/s 2024-03-19T01:22:06.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:06 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:22:06.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:06 smithi156 ceph-mon[66799]: pgmap v636: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 7.4 MiB/s wr, 701 op/s 2024-03-19T01:22:06.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:06 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:22:07.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:07 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:22:07.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:07 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:22:07.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:07 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:22:07.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:07 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:22:07.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:07 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:22:07.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:07 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:22:07.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:07 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:22:07.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:07 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:22:07.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:07 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:22:09.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:08 smithi156 ceph-mon[66799]: pgmap v637: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.2 MiB/s wr, 668 op/s 2024-03-19T01:22:09.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:08 smithi162 ceph-mon[67178]: pgmap v637: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.2 MiB/s wr, 668 op/s 2024-03-19T01:22:09.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:08 smithi116 ceph-mon[62701]: pgmap v637: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.2 MiB/s wr, 668 op/s 2024-03-19T01:22:11.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:10 smithi156 ceph-mon[66799]: pgmap v638: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.3 MiB/s wr, 767 op/s 2024-03-19T01:22:11.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:10 smithi162 ceph-mon[67178]: pgmap v638: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.3 MiB/s wr, 767 op/s 2024-03-19T01:22:11.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:10 smithi116 ceph-mon[62701]: pgmap v638: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.3 MiB/s wr, 767 op/s 2024-03-19T01:22:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 565.0 s, 2560.4 tps, lat 38.643 ms stddev 45.184 2024-03-19T01:22:13.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:12 smithi156 ceph-mon[66799]: pgmap v639: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.4 MiB/s wr, 776 op/s 2024-03-19T01:22:13.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:12 smithi162 ceph-mon[67178]: pgmap v639: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.4 MiB/s wr, 776 op/s 2024-03-19T01:22:13.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:12 smithi116 ceph-mon[62701]: pgmap v639: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.4 MiB/s wr, 776 op/s 2024-03-19T01:22:14.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.132+0000 7f975e10d640 1 -- 172.21.15.116:0/1367456003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97500a5d60 msgr2=0x7f97500a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:14.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.132+0000 7f975e10d640 1 --2- 172.21.15.116:0/1367456003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97500a5d60 0x7f97500a61c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9758066f60 tx=0x7f975808efb0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.135+0000 7f975e10d640 1 -- 172.21.15.116:0/1367456003 shutdown_connections 2024-03-19T01:22:14.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.135+0000 7f975e10d640 1 --2- 172.21.15.116:0/1367456003 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97500a6700 0x7f97500aafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.135+0000 7f975e10d640 1 --2- 172.21.15.116:0/1367456003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97500a5d60 0x7f97500a61c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.135+0000 7f975e10d640 1 --2- 172.21.15.116:0/1367456003 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97500a4c30 0x7f97500a5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.135+0000 7f975e10d640 1 -- 172.21.15.116:0/1367456003 >> 172.21.15.116:0/1367456003 conn(0x7f97500a00c0 msgr2=0x7f97500a2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:14.142 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.140+0000 7f975e10d640 1 -- 172.21.15.116:0/1367456003 shutdown_connections 2024-03-19T01:22:14.143 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.143+0000 7f975e10d640 1 -- 172.21.15.116:0/1367456003 wait complete. 2024-03-19T01:22:14.144 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.143+0000 7f975e10d640 1 Processor -- start 2024-03-19T01:22:14.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.144+0000 7f975e10d640 1 -- start start 2024-03-19T01:22:14.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.144+0000 7f975e10d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97500a4c30 0x7f97500b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.144+0000 7f975e10d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97500a5d60 0x7f97500b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.144+0000 7f975e10d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97500a6700 0x7f9750153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.144+0000 7f975e10d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f97500b8ff0 con 0x7f97500a6700 2024-03-19T01:22:14.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.144+0000 7f975e10d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f97500b8e70 con 0x7f97500a5d60 2024-03-19T01:22:14.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.144+0000 7f975e10d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f97500b9170 con 0x7f97500a4c30 2024-03-19T01:22:14.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.144+0000 7f975c90a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97500a5d60 0x7f97500b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.144+0000 7f975d10b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97500a4c30 0x7f97500b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.144+0000 7f975c90a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97500a5d60 0x7f97500b1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:52420/0 (socket says 172.21.15.116:52420) 2024-03-19T01:22:14.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.144+0000 7f975c90a640 1 -- 172.21.15.116:0/3151499596 learned_addr learned my addr 172.21.15.116:0/3151499596 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:14.145 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.144+0000 7f975d10b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97500a4c30 0x7f97500b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41040/0 (socket says 172.21.15.116:41040) 2024-03-19T01:22:14.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.145+0000 7f975d10b640 1 -- 172.21.15.116:0/3151499596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97500a5d60 msgr2=0x7f97500b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:14.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.145+0000 7f975d10b640 1 --2- 172.21.15.116:0/3151499596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97500a5d60 0x7f97500b1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.145+0000 7f975d10b640 1 -- 172.21.15.116:0/3151499596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97500a6700 msgr2=0x7f9750153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:14.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.145+0000 7f975d10b640 1 --2- 172.21.15.116:0/3151499596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97500a6700 0x7f9750153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.145+0000 7f975d10b640 1 -- 172.21.15.116:0/3151499596 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9750153c40 con 0x7f97500a4c30 2024-03-19T01:22:14.146 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.146+0000 7f975d10b640 1 --2- 172.21.15.116:0/3151499596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97500a4c30 0x7f97500b1830 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9748009fd0 tx=0x7f974800b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:14.147 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.146+0000 7f974e7fc640 1 -- 172.21.15.116:0/3151499596 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9748015020 con 0x7f97500a4c30 2024-03-19T01:22:14.147 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.146+0000 7f975e10d640 1 -- 172.21.15.116:0/3151499596 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9750153e10 con 0x7f97500a4c30 2024-03-19T01:22:14.147 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.146+0000 7f975e10d640 1 -- 172.21.15.116:0/3151499596 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9750154290 con 0x7f97500a4c30 2024-03-19T01:22:14.147 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.146+0000 7f974e7fc640 1 -- 172.21.15.116:0/3151499596 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9748011d10 con 0x7f97500a4c30 2024-03-19T01:22:14.147 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.146+0000 7f974e7fc640 1 -- 172.21.15.116:0/3151499596 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974800f2e0 con 0x7f97500a4c30 2024-03-19T01:22:14.150 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.149+0000 7f974e7fc640 1 -- 172.21.15.116:0/3151499596 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f9748018020 con 0x7f97500a4c30 2024-03-19T01:22:14.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.151+0000 7f974e7fc640 1 --2- 172.21.15.116:0/3151499596 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f97440a29d0 0x7f97440a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.152 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.152+0000 7f974e7fc640 1 -- 172.21.15.116:0/3151499596 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f97480ce1f0 con 0x7f97500a4c30 2024-03-19T01:22:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.152+0000 7f9762030740 1 Processor -- start 2024-03-19T01:22:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f9762030740 1 -- start start 2024-03-19T01:22:14.153 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f9762030740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e8552e0 0x55b73e855700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f9762030740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e855cd0 0x55b73e85a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f9762030740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e85ae40 0x55b73e85d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f9762030740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b73e7b9be0 con 0x55b73e855cd0 2024-03-19T01:22:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f9762030740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b73e762960 con 0x55b73e85ae40 2024-03-19T01:22:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f9762030740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b73e7b5c80 con 0x55b73e8552e0 2024-03-19T01:22:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f975d10b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e8552e0 0x55b73e855700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f975d10b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e8552e0 0x55b73e855700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41056/0 (socket says 172.21.15.116:41056) 2024-03-19T01:22:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f975d10b640 1 -- 172.21.15.116:0/82518273 learned_addr learned my addr 172.21.15.116:0/82518273 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f975c90a640 1 --2- 172.21.15.116:0/3151499596 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f97440a29d0 0x7f97440a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f975d90c640 1 --2- 172.21.15.116:0/82518273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e85ae40 0x55b73e85d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.154 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f975c90a640 1 --2- 172.21.15.116:0/82518273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e855cd0 0x55b73e85a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f975d10b640 1 -- 172.21.15.116:0/82518273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e85ae40 msgr2=0x55b73e85d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:14.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f975d10b640 1 --2- 172.21.15.116:0/82518273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e85ae40 0x55b73e85d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f975d10b640 1 -- 172.21.15.116:0/82518273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e855cd0 msgr2=0x55b73e85a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:14.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f975d10b640 1 --2- 172.21.15.116:0/82518273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e855cd0 0x55b73e85a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.153+0000 7f975d10b640 1 -- 172.21.15.116:0/82518273 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b73e759d20 con 0x55b73e8552e0 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.154+0000 7f975d10b640 1 --2- 172.21.15.116:0/82518273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e8552e0 0x55b73e855700 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9748096f70 tx=0x7f974809a9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b1cc8bf510dcc62 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.154+0000 7f975c90a640 1 --2- 172.21.15.116:0/3151499596 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f97440a29d0 0x7f97440a4e90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f97580670d0 tx=0x7f975808f950 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.154+0000 7f97437fe640 1 -- 172.21.15.116:0/82518273 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97480b2030 con 0x55b73e8552e0 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.154+0000 7f97437fe640 1 -- 172.21.15.116:0/82518273 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f974809ab10 con 0x55b73e8552e0 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.154+0000 7f97437fe640 1 -- 172.21.15.116:0/82518273 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97480b3bd0 con 0x55b73e8552e0 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.155+0000 7f9762030740 1 -- 172.21.15.116:0/82518273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e8552e0 msgr2=0x55b73e855700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.155+0000 7f9762030740 1 --2- 172.21.15.116:0/82518273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e8552e0 0x55b73e855700 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9748096f70 tx=0x7f974809a9e0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.155+0000 7f9762030740 1 -- 172.21.15.116:0/82518273 shutdown_connections 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.155+0000 7f9762030740 1 --2- 172.21.15.116:0/82518273 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e85ae40 0x55b73e85d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.155+0000 7f9762030740 1 --2- 172.21.15.116:0/82518273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e855cd0 0x55b73e85a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.155+0000 7f9762030740 1 --2- 172.21.15.116:0/82518273 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e8552e0 0x55b73e855700 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.155+0000 7f9762030740 1 -- 172.21.15.116:0/82518273 >> 172.21.15.116:0/82518273 conn(0x55b73e850850 msgr2=0x55b73e850c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.155+0000 7f9762030740 1 -- 172.21.15.116:0/82518273 shutdown_connections 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.155+0000 7f9762030740 1 -- 172.21.15.116:0/82518273 wait complete. 2024-03-19T01:22:14.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.156+0000 7f9762030740 1 -- start start 2024-03-19T01:22:14.158 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.156+0000 7f9762030740 1 Processor -- start 2024-03-19T01:22:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.157+0000 7f9762030740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e8552e0 0x55b73e85f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.157+0000 7f9762030740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e855cd0 0x55b73e85f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.157+0000 7f9762030740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e85ae40 0x55b73e866800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.160 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.157+0000 7f975d10b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e8552e0 0x55b73e85f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.157+0000 7f975d10b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e8552e0 0x55b73e85f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:52438/0 (socket says 172.21.15.116:52438) 2024-03-19T01:22:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.157+0000 7f975d10b640 1 -- 172.21.15.116:0/980744645 learned_addr learned my addr 172.21.15.116:0/980744645 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.157+0000 7f9762030740 1 -- 172.21.15.116:0/980744645 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b73e85dd90 con 0x55b73e855cd0 2024-03-19T01:22:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.157+0000 7f975d90c640 1 --2- 172.21.15.116:0/980744645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e85ae40 0x55b73e866800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.157+0000 7f9762030740 1 -- 172.21.15.116:0/980744645 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b73e775db0 con 0x55b73e8552e0 2024-03-19T01:22:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.157+0000 7f9762030740 1 -- 172.21.15.116:0/980744645 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b73e85df10 con 0x55b73e85ae40 2024-03-19T01:22:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.158+0000 7f9762030740 10 client.0 _ll_register_callbacks cb 0x55b73e77ae70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:22:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.158+0000 7f975d90c640 1 -- 172.21.15.116:0/980744645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e8552e0 msgr2=0x55b73e85f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.158+0000 7f975d90c640 1 --2- 172.21.15.116:0/980744645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e8552e0 0x55b73e85f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.158+0000 7f975d90c640 1 -- 172.21.15.116:0/980744645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e855cd0 msgr2=0x55b73e85f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.158+0000 7f975d90c640 1 --2- 172.21.15.116:0/980744645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e855cd0 0x55b73e85f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.158+0000 7f975d90c640 1 -- 172.21.15.116:0/980744645 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b73e774c60 con 0x55b73e85ae40 2024-03-19T01:22:14.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.158+0000 7f975d10b640 1 --2- 172.21.15.116:0/980744645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e8552e0 0x55b73e85f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.158+0000 7f975d90c640 1 --2- 172.21.15.116:0/980744645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e85ae40 0x55b73e866800 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f97540116c0 tx=0x7f9754011b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bff515e9e733bf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.158+0000 7f9740ff9640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.158+0000 7f9740ff9640 1 -- 172.21.15.116:0/980744645 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975400f8c0 con 0x55b73e85ae40 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.158+0000 7f9740ff9640 1 -- 172.21.15.116:0/980744645 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f975400fa60 con 0x55b73e85ae40 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.159+0000 7f9762030740 1 -- 172.21.15.116:0/980744645 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f974809ab10 con 0x55b73e85ae40 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.159+0000 7f9740ff9640 1 -- 172.21.15.116:0/980744645 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f975400fc60 con 0x55b73e85ae40 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.159+0000 7f9762030740 10 client.25864 fetch_fsmap learned FSMap version 45 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.159+0000 7f9762030740 1 -- 172.21.15.116:0/980744645 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55b73e4ce210 con 0x55b73e85ae40 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.159+0000 7f9740ff9640 1 -- 172.21.15.116:0/980744645 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f975400fe00 con 0x55b73e85ae40 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.159+0000 7f9740ff9640 1 -- 172.21.15.116:0/980744645 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f9754018070 con 0x55b73e85ae40 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.159+0000 7f9762030740 10 client.25864 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.159+0000 7f9762030740 10 client.25864 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.159+0000 7f9762030740 20 client.25864 populate_metadata read hostname 'smithi116' 2024-03-19T01:22:14.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.159+0000 7f9762030740 1 --2- 172.21.15.116:0/980744645 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b73e86b260 0x55b73e86d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.163 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.159+0000 7f9762030740 4 client.25864 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:22:14.163 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.159+0000 7f9762030740 1 -- 172.21.15.116:0/980744645 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f974809ab10 con 0x55b73e86b260 2024-03-19T01:22:14.163 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.159+0000 7f975d10b640 1 --2- 172.21.15.116:0/980744645 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b73e86b260 0x55b73e86d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.163 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.160+0000 7f975d10b640 1 --2- 172.21.15.116:0/980744645 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b73e86b260 0x55b73e86d6e0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78f2ef3dc6a7a63b server_cookie=37cdc66dee066256 in_seq=0 out_seq=0 2024-03-19T01:22:14.163 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.160+0000 7f9740ff9640 10 client.25864 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.160+0000 7f9740ff9640 1 -- 172.21.15.116:0/980744645 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f974809ab10 con 0x55b73e86b260 2024-03-19T01:22:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.160+0000 7f9740ff9640 10 client.25864 handle_command_reply: tid=0 2024-03-19T01:22:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.160+0000 7f9762030740 1 client.25864 shutdown 2024-03-19T01:22:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.161+0000 7f9762030740 1 -- 172.21.15.116:0/980744645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e85ae40 msgr2=0x55b73e866800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.161+0000 7f9762030740 1 --2- 172.21.15.116:0/980744645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e85ae40 0x55b73e866800 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f97540116c0 tx=0x7f9754011b80 comp rx=0 tx=0).stop 2024-03-19T01:22:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.161+0000 7f9762030740 1 -- 172.21.15.116:0/980744645 shutdown_connections 2024-03-19T01:22:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.161+0000 7f9762030740 1 --2- 172.21.15.116:0/980744645 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b73e86b260 0x55b73e86d6e0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.161+0000 7f9762030740 1 --2- 172.21.15.116:0/980744645 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e85ae40 0x55b73e866800 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.161+0000 7f9740ff9640 0 client.25864 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:14.164 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.161+0000 7f9762030740 1 --2- 172.21.15.116:0/980744645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e855cd0 0x55b73e85f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.161+0000 7f9762030740 1 --2- 172.21.15.116:0/980744645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e8552e0 0x55b73e85f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.161+0000 7f9762030740 1 -- 172.21.15.116:0/980744645 >> 172.21.15.116:0/980744645 conn(0x55b73e850850 msgr2=0x55b73e84fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.161+0000 7f9762030740 1 -- 172.21.15.116:0/980744645 shutdown_connections 2024-03-19T01:22:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.161+0000 7f9762030740 1 -- 172.21.15.116:0/980744645 wait complete. 2024-03-19T01:22:14.165 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.161+0000 7f9762030740 20 client.25864 trim_cache size 0 max 16384 2024-03-19T01:22:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f9762030740 1 Processor -- start 2024-03-19T01:22:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f9762030740 1 -- start start 2024-03-19T01:22:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f9762030740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e8552e0 0x55b73e84f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f9762030740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 0x55b73e8667c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f9762030740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e85ae40 0x55b73e871fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f9762030740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f973c00e530 con 0x55b73e8552e0 2024-03-19T01:22:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f9762030740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f973c010070 con 0x55b73e855cd0 2024-03-19T01:22:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f9762030740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f973c00ec00 con 0x55b73e85ae40 2024-03-19T01:22:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f975d10b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e8552e0 0x55b73e84f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f975c90a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 0x55b73e8667c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f975c90a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 0x55b73e8667c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:52446/0 (socket says 172.21.15.116:52446) 2024-03-19T01:22:14.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f975c90a640 1 -- 172.21.15.116:0/4171801714 learned_addr learned my addr 172.21.15.116:0/4171801714 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:14.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f975d10b640 1 --2- 172.21.15.116:0/4171801714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e8552e0 0x55b73e84f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46796/0 (socket says 172.21.15.116:46796) 2024-03-19T01:22:14.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f975d90c640 1 --2- 172.21.15.116:0/4171801714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e85ae40 0x55b73e871fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f975c90a640 1 -- 172.21.15.116:0/4171801714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e85ae40 msgr2=0x55b73e871fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:14.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f975c90a640 1 --2- 172.21.15.116:0/4171801714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e85ae40 0x55b73e871fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f975c90a640 1 -- 172.21.15.116:0/4171801714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e8552e0 msgr2=0x55b73e84f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:14.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f975c90a640 1 --2- 172.21.15.116:0/4171801714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e8552e0 0x55b73e84f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f975c90a640 1 -- 172.21.15.116:0/4171801714 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b73e59a620 con 0x55b73e855cd0 2024-03-19T01:22:14.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f975d10b640 1 --2- 172.21.15.116:0/4171801714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e8552e0 0x55b73e84f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:22:14.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.180+0000 7f975c90a640 1 --2- 172.21.15.116:0/4171801714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 0x55b73e8667c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9758050d60 tx=0x7f97580ac000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee4f5194f9b09aa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:14.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.181+0000 7f9742ffd640 1 -- 172.21.15.116:0/4171801714 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97580a78b0 con 0x55b73e855cd0 2024-03-19T01:22:14.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.181+0000 7f9742ffd640 1 -- 172.21.15.116:0/4171801714 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9758093070 con 0x55b73e855cd0 2024-03-19T01:22:14.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.181+0000 7f9742ffd640 1 -- 172.21.15.116:0/4171801714 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97580a4c70 con 0x55b73e855cd0 2024-03-19T01:22:14.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.182+0000 7f9762030740 1 -- 172.21.15.116:0/4171801714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 msgr2=0x55b73e8667c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:14.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.182+0000 7f9762030740 1 --2- 172.21.15.116:0/4171801714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 0x55b73e8667c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9758050d60 tx=0x7f97580ac000 comp rx=0 tx=0).stop 2024-03-19T01:22:14.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.182+0000 7f9762030740 1 -- 172.21.15.116:0/4171801714 shutdown_connections 2024-03-19T01:22:14.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.182+0000 7f9762030740 1 --2- 172.21.15.116:0/4171801714 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e85ae40 0x55b73e871fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.182+0000 7f9762030740 1 --2- 172.21.15.116:0/4171801714 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 0x55b73e8667c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.182+0000 7f9762030740 1 --2- 172.21.15.116:0/4171801714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e8552e0 0x55b73e84f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.182+0000 7f9762030740 1 -- 172.21.15.116:0/4171801714 >> 172.21.15.116:0/4171801714 conn(0x55b73e850850 msgr2=0x55b73e84f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:14.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.182+0000 7f9762030740 1 -- 172.21.15.116:0/4171801714 shutdown_connections 2024-03-19T01:22:14.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.182+0000 7f9762030740 1 -- 172.21.15.116:0/4171801714 wait complete. 2024-03-19T01:22:14.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f9762030740 1 -- start start 2024-03-19T01:22:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f9762030740 1 Processor -- start 2024-03-19T01:22:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f9762030740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e8552e0 0x55b73e8645a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f9762030740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 0x55b73e864ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f9762030740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e85ae40 0x55b73e865aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f9762030740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f973c00f9a0 con 0x55b73e85ae40 2024-03-19T01:22:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f9762030740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f973c00f2d0 con 0x55b73e855cd0 2024-03-19T01:22:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f9762030740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f973c00c510 con 0x55b73e8552e0 2024-03-19T01:22:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f975d10b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e8552e0 0x55b73e8645a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f975c90a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 0x55b73e864ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f9762030740 10 client.0 _ll_register_callbacks cb 0x7f9754009710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:22:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f975d10b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e8552e0 0x55b73e8645a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:41076/0 (socket says 172.21.15.116:41076) 2024-03-19T01:22:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f975c90a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 0x55b73e864ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:52458/0 (socket says 172.21.15.116:52458) 2024-03-19T01:22:14.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f975d10b640 1 -- 172.21.15.116:0/1729479296 learned_addr learned my addr 172.21.15.116:0/1729479296 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.183+0000 7f975d90c640 1 --2- 172.21.15.116:0/1729479296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e85ae40 0x55b73e865aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.184+0000 7f975c90a640 1 -- 172.21.15.116:0/1729479296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e8552e0 msgr2=0x55b73e8645a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.184+0000 7f975c90a640 1 --2- 172.21.15.116:0/1729479296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e8552e0 0x55b73e8645a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.184+0000 7f975c90a640 1 -- 172.21.15.116:0/1729479296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e85ae40 msgr2=0x55b73e865aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.184+0000 7f975c90a640 1 --2- 172.21.15.116:0/1729479296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e85ae40 0x55b73e865aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.184+0000 7f975c90a640 1 -- 172.21.15.116:0/1729479296 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b73e866090 con 0x55b73e855cd0 2024-03-19T01:22:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.184+0000 7f975d90c640 1 --2- 172.21.15.116:0/1729479296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e85ae40 0x55b73e865aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:22:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.184+0000 7f975c90a640 1 --2- 172.21.15.116:0/1729479296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 0x55b73e864ae0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f97580507f0 tx=0x7f975808f630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd426e93b3cc821b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:14.188 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.184+0000 7f9740ff9640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.185+0000 7f9740ff9640 1 -- 172.21.15.116:0/1729479296 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97580a78b0 con 0x55b73e855cd0 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.185+0000 7f9740ff9640 1 -- 172.21.15.116:0/1729479296 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f97580a7a50 con 0x55b73e855cd0 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.185+0000 7f9762030740 1 -- 172.21.15.116:0/1729479296 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9758093070 con 0x55b73e855cd0 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.185+0000 7f9740ff9640 1 -- 172.21.15.116:0/1729479296 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f97580a7c90 con 0x55b73e855cd0 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.185+0000 7f9762030740 10 client.25880 fetch_fsmap learned FSMap version 45 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.185+0000 7f9762030740 1 -- 172.21.15.116:0/1729479296 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55b73e8d1c70 con 0x55b73e855cd0 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.186+0000 7f9740ff9640 1 -- 172.21.15.116:0/1729479296 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97580a7e30 con 0x55b73e855cd0 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.186+0000 7f9740ff9640 1 -- 172.21.15.116:0/1729479296 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f97580b05d0 con 0x55b73e855cd0 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.186+0000 7f9762030740 10 client.25880 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.186+0000 7f9762030740 10 client.25880 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.186+0000 7f9762030740 20 client.25880 populate_metadata read hostname 'smithi116' 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.186+0000 7f9762030740 1 --2- 172.21.15.116:0/1729479296 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b73e8d2530 0x55b73e8d4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.186+0000 7f9762030740 4 client.25880 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:22:14.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.186+0000 7f9762030740 1 -- 172.21.15.116:0/1729479296 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f9758093070 con 0x55b73e8d2530 2024-03-19T01:22:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.186+0000 7f975d10b640 1 --2- 172.21.15.116:0/1729479296 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b73e8d2530 0x55b73e8d4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.187+0000 7f9740ff9640 10 client.25880 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:14.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.187+0000 7f975d10b640 1 --2- 172.21.15.116:0/1729479296 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b73e8d2530 0x55b73e8d4950 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd9d006aa69baf9e server_cookie=44521b8dc3d6ac14 in_seq=0 out_seq=0 2024-03-19T01:22:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.194+0000 7f9740ff9640 1 -- 172.21.15.116:0/1729479296 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f9758093070 con 0x55b73e8d2530 2024-03-19T01:22:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.194+0000 7f9740ff9640 10 client.25880 handle_command_reply: tid=0 2024-03-19T01:22:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.194+0000 7f9762030740 1 client.25880 shutdown 2024-03-19T01:22:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.194+0000 7f9762030740 1 -- 172.21.15.116:0/1729479296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 msgr2=0x55b73e864ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.194+0000 7f9762030740 1 --2- 172.21.15.116:0/1729479296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 0x55b73e864ae0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f97580507f0 tx=0x7f975808f630 comp rx=0 tx=0).stop 2024-03-19T01:22:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.194+0000 7f9762030740 1 -- 172.21.15.116:0/1729479296 shutdown_connections 2024-03-19T01:22:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.194+0000 7f9762030740 1 --2- 172.21.15.116:0/1729479296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b73e85ae40 0x55b73e865aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.194+0000 7f9762030740 1 --2- 172.21.15.116:0/1729479296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b73e855cd0 0x55b73e864ae0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.194+0000 7f9762030740 1 --2- 172.21.15.116:0/1729479296 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b73e8d2530 0x55b73e8d4950 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.194+0000 7f9762030740 1 --2- 172.21.15.116:0/1729479296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b73e8552e0 0x55b73e8645a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.194+0000 7f9740ff9640 0 client.25880 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:14.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.194+0000 7f9762030740 1 -- 172.21.15.116:0/1729479296 >> 172.21.15.116:0/1729479296 conn(0x55b73e850850 msgr2=0x55b73e84f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:14.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.194+0000 7f9762030740 1 -- 172.21.15.116:0/1729479296 shutdown_connections 2024-03-19T01:22:14.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.195+0000 7f9762030740 1 -- 172.21.15.116:0/1729479296 wait complete. 2024-03-19T01:22:14.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.196+0000 7f9762030740 20 client.25880 trim_cache size 0 max 16384 2024-03-19T01:22:14.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:22:14.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:22:14.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:22:14.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:22:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.197+0000 7f975e10d640 1 -- 172.21.15.116:0/3151499596 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f97440a29d0 msgr2=0x7f97440a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.197+0000 7f975e10d640 1 --2- 172.21.15.116:0/3151499596 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f97440a29d0 0x7f97440a4e90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f97580670d0 tx=0x7f975808f950 comp rx=0 tx=0).stop 2024-03-19T01:22:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.197+0000 7f975e10d640 1 -- 172.21.15.116:0/3151499596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97500a4c30 msgr2=0x7f97500b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.197+0000 7f975e10d640 1 --2- 172.21.15.116:0/3151499596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97500a4c30 0x7f97500b1830 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9748009fd0 tx=0x7f974800b870 comp rx=0 tx=0).stop 2024-03-19T01:22:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.197+0000 7f975e10d640 1 -- 172.21.15.116:0/3151499596 shutdown_connections 2024-03-19T01:22:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.197+0000 7f975e10d640 1 --2- 172.21.15.116:0/3151499596 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f97500a6700 0x7f9750153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.197+0000 7f975e10d640 1 --2- 172.21.15.116:0/3151499596 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f97440a29d0 0x7f97440a4e90 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.197+0000 7f975e10d640 1 --2- 172.21.15.116:0/3151499596 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97500a5d60 0x7f97500b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.197+0000 7f975e10d640 1 --2- 172.21.15.116:0/3151499596 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f97500a4c30 0x7f97500b1830 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.197+0000 7f975e10d640 1 -- 172.21.15.116:0/3151499596 >> 172.21.15.116:0/3151499596 conn(0x7f97500a00c0 msgr2=0x7f97500a6fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.197+0000 7f975e10d640 1 -- 172.21.15.116:0/3151499596 shutdown_connections 2024-03-19T01:22:14.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:14.197+0000 7f975e10d640 1 -- 172.21.15.116:0/3151499596 wait complete. 2024-03-19T01:22:15.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:14 smithi156 ceph-mon[66799]: pgmap v640: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.1 MiB/s wr, 557 op/s 2024-03-19T01:22:15.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:14 smithi162 ceph-mon[67178]: pgmap v640: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.1 MiB/s wr, 557 op/s 2024-03-19T01:22:15.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:14 smithi116 ceph-mon[62701]: pgmap v640: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.1 MiB/s wr, 557 op/s 2024-03-19T01:22:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 570.0 s, 2578.0 tps, lat 39.014 ms stddev 47.889 2024-03-19T01:22:17.226 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:16 smithi156 ceph-mon[66799]: pgmap v641: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.5 MiB/s wr, 694 op/s 2024-03-19T01:22:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:16 smithi162 ceph-mon[67178]: pgmap v641: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.5 MiB/s wr, 694 op/s 2024-03-19T01:22:17.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:16 smithi116 ceph-mon[62701]: pgmap v641: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 6.5 MiB/s wr, 694 op/s 2024-03-19T01:22:19.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:18 smithi156 ceph-mon[66799]: pgmap v642: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.5 MiB/s wr, 672 op/s 2024-03-19T01:22:19.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:18 smithi162 ceph-mon[67178]: pgmap v642: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.5 MiB/s wr, 672 op/s 2024-03-19T01:22:19.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:18 smithi116 ceph-mon[62701]: pgmap v642: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.5 MiB/s wr, 672 op/s 2024-03-19T01:22:21.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:20 smithi162 ceph-mon[67178]: pgmap v643: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 706 op/s 2024-03-19T01:22:21.447 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:20 smithi116 ceph-mon[62701]: pgmap v643: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 706 op/s 2024-03-19T01:22:21.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:20 smithi156 ceph-mon[66799]: pgmap v643: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 MiB/s wr, 706 op/s 2024-03-19T01:22:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 575.0 s, 1637.2 tps, lat 60.925 ms stddev 69.361 2024-03-19T01:22:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:22 smithi162 ceph-mon[67178]: pgmap v644: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.8 MiB/s wr, 635 op/s 2024-03-19T01:22:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:22 smithi116 ceph-mon[62701]: pgmap v644: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.8 MiB/s wr, 635 op/s 2024-03-19T01:22:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:22 smithi156 ceph-mon[66799]: pgmap v644: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.8 MiB/s wr, 635 op/s 2024-03-19T01:22:25.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:24 smithi162 ceph-mon[67178]: pgmap v645: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.1 MiB/s wr, 633 op/s 2024-03-19T01:22:25.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:24 smithi116 ceph-mon[62701]: pgmap v645: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.1 MiB/s wr, 633 op/s 2024-03-19T01:22:25.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:24 smithi156 ceph-mon[66799]: pgmap v645: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.1 MiB/s wr, 633 op/s 2024-03-19T01:22:26.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 580.0 s, 1457.4 tps, lat 68.947 ms stddev 80.779 2024-03-19T01:22:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.318+0000 7f36f843f640 1 -- 172.21.15.116:0/564847938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f0102a40 msgr2=0x7f36f0102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.318+0000 7f36f843f640 1 --2- 172.21.15.116:0/564847938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f0102a40 0x7f36f0102ec0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009900 tx=0x7f36ec02bc30 comp rx=0 tx=0).stop 2024-03-19T01:22:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.318+0000 7f36f843f640 1 -- 172.21.15.116:0/564847938 shutdown_connections 2024-03-19T01:22:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.318+0000 7f36f843f640 1 --2- 172.21.15.116:0/564847938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36f0103400 0x7f36f0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.318+0000 7f36f843f640 1 --2- 172.21.15.116:0/564847938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f0102a40 0x7f36f0102ec0 secure :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009900 tx=0x7f36ec02bc30 comp rx=0 tx=0).stop 2024-03-19T01:22:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.318+0000 7f36f843f640 1 --2- 172.21.15.116:0/564847938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36f0101850 0x7f36f0101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.318+0000 7f36f843f640 1 -- 172.21.15.116:0/564847938 >> 172.21.15.116:0/564847938 conn(0x7f36f00fcde0 msgr2=0x7f36f00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:27.320 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.319+0000 7f36f843f640 1 -- 172.21.15.116:0/564847938 shutdown_connections 2024-03-19T01:22:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.319+0000 7f36f843f640 1 -- 172.21.15.116:0/564847938 wait complete. 2024-03-19T01:22:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f843f640 1 Processor -- start 2024-03-19T01:22:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f843f640 1 -- start start 2024-03-19T01:22:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f843f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36f0101850 0x7f36f0199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f843f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36f0103400 0x7f36f019a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f843f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f019f4b0 0x7f36f01a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f843f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36f010d4d0 con 0x7f36f019f4b0 2024-03-19T01:22:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f843f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f36f010d350 con 0x7f36f0103400 2024-03-19T01:22:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f843f640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36f010d650 con 0x7f36f0101850 2024-03-19T01:22:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f69dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f019f4b0 0x7f36f01a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f61db640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36f0101850 0x7f36f0199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f69dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f019f4b0 0x7f36f01a18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54592/0 (socket says 172.21.15.116:54592) 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f59da640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36f0103400 0x7f36f019a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f69dc640 1 -- 172.21.15.116:0/340478955 learned_addr learned my addr 172.21.15.116:0/340478955 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f69dc640 1 -- 172.21.15.116:0/340478955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36f0101850 msgr2=0x7f36f0199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f69dc640 1 --2- 172.21.15.116:0/340478955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36f0101850 0x7f36f0199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f69dc640 1 -- 172.21.15.116:0/340478955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36f0103400 msgr2=0x7f36f019a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f69dc640 1 --2- 172.21.15.116:0/340478955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36f0103400 0x7f36f019a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f69dc640 1 -- 172.21.15.116:0/340478955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f01a1f10 con 0x7f36f019f4b0 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.320+0000 7f36f59da640 1 --2- 172.21.15.116:0/340478955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36f0103400 0x7f36f019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.321+0000 7f36f69dc640 1 --2- 172.21.15.116:0/340478955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f019f4b0 0x7f36f01a18a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009fd0 tx=0x7f36e800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.321+0000 7f36e77fe640 1 -- 172.21.15.116:0/340478955 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e8017070 con 0x7f36f019f4b0 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.321+0000 7f36f843f640 1 -- 172.21.15.116:0/340478955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f01a2200 con 0x7f36f019f4b0 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.321+0000 7f36e77fe640 1 -- 172.21.15.116:0/340478955 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e8003e50 con 0x7f36f019f4b0 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.321+0000 7f36e77fe640 1 -- 172.21.15.116:0/340478955 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e80106c0 con 0x7f36f019f4b0 2024-03-19T01:22:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.321+0000 7f36f843f640 1 -- 172.21.15.116:0/340478955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f01a26d0 con 0x7f36f019f4b0 2024-03-19T01:22:27.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.323+0000 7f36e77fe640 1 -- 172.21.15.116:0/340478955 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f36e8003970 con 0x7f36f019f4b0 2024-03-19T01:22:27.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.324+0000 7f36e77fe640 1 --2- 172.21.15.116:0/340478955 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f36dc0a28a0 0x7f36dc0a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.324+0000 7f36f61db640 1 --2- 172.21.15.116:0/340478955 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f36dc0a28a0 0x7f36dc0a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.324+0000 7f36e77fe640 1 -- 172.21.15.116:0/340478955 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f36e80ce780 con 0x7f36f019f4b0 2024-03-19T01:22:27.326 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.325+0000 7f36f61db640 1 --2- 172.21.15.116:0/340478955 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f36dc0a28a0 0x7f36dc0a4d60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f36f01028a0 tx=0x7f36e0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.325+0000 7f36fb020740 1 Processor -- start 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.325+0000 7f36fb020740 1 -- start start 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.325+0000 7f36fb020740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081b100 0x562bd081b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.325+0000 7f36fb020740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd081baf0 0x562bd08205f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.325+0000 7f36fb020740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd0820c60 0x562bd08230f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.325+0000 7f36fb020740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562bd077fbe0 con 0x562bd081baf0 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.325+0000 7f36fb020740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562bd0728960 con 0x562bd0820c60 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.325+0000 7f36fb020740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562bd077bc80 con 0x562bd081b100 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.325+0000 7f36f59da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd081baf0 0x562bd08205f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.326+0000 7f36f59da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd081baf0 0x562bd08205f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54598/0 (socket says 172.21.15.116:54598) 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.326+0000 7f36f61db640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081b100 0x562bd081b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.326+0000 7f36f59da640 1 -- 172.21.15.116:0/1582525381 learned_addr learned my addr 172.21.15.116:0/1582525381 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.326+0000 7f36f69dc640 1 --2- 172.21.15.116:0/1582525381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd0820c60 0x562bd08230f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.327 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.326+0000 7f36f61db640 1 -- 172.21.15.116:0/1582525381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd0820c60 msgr2=0x562bd08230f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.326+0000 7f36f61db640 1 --2- 172.21.15.116:0/1582525381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd0820c60 0x562bd08230f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.326+0000 7f36f61db640 1 -- 172.21.15.116:0/1582525381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd081baf0 msgr2=0x562bd08205f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.326+0000 7f36f61db640 1 --2- 172.21.15.116:0/1582525381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd081baf0 0x562bd08205f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.326+0000 7f36f61db640 1 -- 172.21.15.116:0/1582525381 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bd071fd20 con 0x562bd081b100 2024-03-19T01:22:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.326+0000 7f36f61db640 1 --2- 172.21.15.116:0/1582525381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081b100 0x562bd081b520 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f36e00118d0 tx=0x7f36e0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0f3eeeed9cad552 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.326+0000 7f36e4bfe640 1 -- 172.21.15.116:0/1582525381 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e0026430 con 0x562bd081b100 2024-03-19T01:22:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.326+0000 7f36e4bfe640 1 -- 172.21.15.116:0/1582525381 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e001c8f0 con 0x562bd081b100 2024-03-19T01:22:27.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.326+0000 7f36e4bfe640 1 -- 172.21.15.116:0/1582525381 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e001d980 con 0x562bd081b100 2024-03-19T01:22:27.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.327+0000 7f36fb020740 1 -- 172.21.15.116:0/1582525381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081b100 msgr2=0x562bd081b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:27.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.327+0000 7f36fb020740 1 --2- 172.21.15.116:0/1582525381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081b100 0x562bd081b520 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f36e00118d0 tx=0x7f36e0014000 comp rx=0 tx=0).stop 2024-03-19T01:22:27.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.327+0000 7f36fb020740 1 -- 172.21.15.116:0/1582525381 shutdown_connections 2024-03-19T01:22:27.330 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.327+0000 7f36fb020740 1 --2- 172.21.15.116:0/1582525381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd0820c60 0x562bd08230f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.327+0000 7f36fb020740 1 --2- 172.21.15.116:0/1582525381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd081baf0 0x562bd08205f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.327+0000 7f36fb020740 1 --2- 172.21.15.116:0/1582525381 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081b100 0x562bd081b520 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.327+0000 7f36fb020740 1 -- 172.21.15.116:0/1582525381 >> 172.21.15.116:0/1582525381 conn(0x562bd0816670 msgr2=0x562bd0816a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.327+0000 7f36fb020740 1 -- 172.21.15.116:0/1582525381 shutdown_connections 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.327+0000 7f36fb020740 1 -- 172.21.15.116:0/1582525381 wait complete. 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.327+0000 7f36fb020740 1 -- start start 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.327+0000 7f36fb020740 1 Processor -- start 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36fb020740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd081b100 0x562bd0825070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36fb020740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081baf0 0x562bd08255b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36fb020740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd0820c60 0x562bd082c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36fb020740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562bd0823c10 con 0x562bd081b100 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36fb020740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562bd073bdb0 con 0x562bd0820c60 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36fb020740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562bd0823d90 con 0x562bd081baf0 2024-03-19T01:22:27.331 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36fb020740 10 client.0 _ll_register_callbacks cb 0x562bd0740e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:22:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36f61db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd081b100 0x562bd0825070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36f59da640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081baf0 0x562bd08255b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36f61db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd081b100 0x562bd0825070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54614/0 (socket says 172.21.15.116:54614) 2024-03-19T01:22:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36f61db640 1 -- 172.21.15.116:0/3991508065 learned_addr learned my addr 172.21.15.116:0/3991508065 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36f69dc640 1 --2- 172.21.15.116:0/3991508065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd0820c60 0x562bd082c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36f61db640 1 -- 172.21.15.116:0/3991508065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081baf0 msgr2=0x562bd08255b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36f61db640 1 --2- 172.21.15.116:0/3991508065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081baf0 0x562bd08255b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36f61db640 1 -- 172.21.15.116:0/3991508065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd0820c60 msgr2=0x562bd082c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36f61db640 1 --2- 172.21.15.116:0/3991508065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd0820c60 0x562bd082c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.328+0000 7f36f61db640 1 -- 172.21.15.116:0/3991508065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bd073ac60 con 0x562bd081b100 2024-03-19T01:22:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.329+0000 7f36f61db640 1 --2- 172.21.15.116:0/3991508065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd081b100 0x562bd0825070 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f36e000ee70 tx=0x7f36e000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b307dece474cd806 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.329+0000 7f36d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:22:27.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.330+0000 7f36d9ffb640 1 -- 172.21.15.116:0/3991508065 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e0034060 con 0x562bd081b100 2024-03-19T01:22:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.330+0000 7f36fb020740 1 -- 172.21.15.116:0/3991508065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36e001c8f0 con 0x562bd081b100 2024-03-19T01:22:27.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.330+0000 7f36d9ffb640 1 -- 172.21.15.116:0/3991508065 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f36e001cba0 con 0x562bd081b100 2024-03-19T01:22:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.330+0000 7f36d9ffb640 1 -- 172.21.15.116:0/3991508065 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f36e002b040 con 0x562bd081b100 2024-03-19T01:22:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.330+0000 7f36fb020740 10 client.16047 fetch_fsmap learned FSMap version 45 2024-03-19T01:22:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.330+0000 7f36fb020740 1 -- 172.21.15.116:0/3991508065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x562bd0494210 con 0x562bd081b100 2024-03-19T01:22:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.331+0000 7f36d9ffb640 1 -- 172.21.15.116:0/3991508065 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e001cda0 con 0x562bd081b100 2024-03-19T01:22:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.331+0000 7f36d9ffb640 1 -- 172.21.15.116:0/3991508065 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f36e002e030 con 0x562bd081b100 2024-03-19T01:22:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.331+0000 7f36fb020740 10 client.16047 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:22:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.331+0000 7f36fb020740 10 client.16047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:22:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.331+0000 7f36fb020740 20 client.16047 populate_metadata read hostname 'smithi116' 2024-03-19T01:22:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.331+0000 7f36fb020740 1 --2- 172.21.15.116:0/3991508065 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562bd0830be0 0x562bd0832fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.331+0000 7f36fb020740 4 client.16047 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.331+0000 7f36fb020740 1 -- 172.21.15.116:0/3991508065 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f36e001c8f0 con 0x562bd0830be0 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.331+0000 7f36f59da640 1 --2- 172.21.15.116:0/3991508065 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562bd0830be0 0x562bd0832fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.332+0000 7f36d9ffb640 10 client.16047 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.332+0000 7f36f59da640 1 --2- 172.21.15.116:0/3991508065 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562bd0830be0 0x562bd0832fd0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e628871ac4dd5b61 server_cookie=ec433b5db2d3c602 in_seq=0 out_seq=0 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36d9ffb640 1 -- 172.21.15.116:0/3991508065 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f36e001c8f0 con 0x562bd0830be0 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36d9ffb640 10 client.16047 handle_command_reply: tid=0 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36fb020740 1 client.16047 shutdown 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36fb020740 1 -- 172.21.15.116:0/3991508065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd081b100 msgr2=0x562bd0825070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36fb020740 1 --2- 172.21.15.116:0/3991508065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd081b100 0x562bd0825070 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f36e000ee70 tx=0x7f36e000eea0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36fb020740 1 -- 172.21.15.116:0/3991508065 shutdown_connections 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36fb020740 1 --2- 172.21.15.116:0/3991508065 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd0820c60 0x562bd082c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36fb020740 1 --2- 172.21.15.116:0/3991508065 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562bd0830be0 0x562bd0832fd0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36fb020740 1 --2- 172.21.15.116:0/3991508065 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081baf0 0x562bd08255b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36fb020740 1 --2- 172.21.15.116:0/3991508065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd081b100 0x562bd0825070 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.336 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36fb020740 1 -- 172.21.15.116:0/3991508065 >> 172.21.15.116:0/3991508065 conn(0x562bd0816670 msgr2=0x562bd08178c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:27.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36d9ffb640 0 client.16047 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:27.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36fb020740 1 -- 172.21.15.116:0/3991508065 shutdown_connections 2024-03-19T01:22:27.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36fb020740 1 -- 172.21.15.116:0/3991508065 wait complete. 2024-03-19T01:22:27.337 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.333+0000 7f36fb020740 20 client.16047 trim_cache size 0 max 16384 2024-03-19T01:22:27.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36fb020740 1 Processor -- start 2024-03-19T01:22:27.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36fb020740 1 -- start start 2024-03-19T01:22:27.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36fb020740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd081b100 0x562bd0825390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36fb020740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081baf0 0x562bd082c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36fb020740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd0820c60 0x562bd0815220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36fb020740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36f01ab610 con 0x562bd0820c60 2024-03-19T01:22:27.352 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36fb020740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f36f01ad150 con 0x562bd081b100 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36fb020740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36f01abce0 con 0x562bd081baf0 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36f69dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd0820c60 0x562bd0815220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36f61db640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd081b100 0x562bd0825390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36f59da640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081baf0 0x562bd082c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36f59da640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081baf0 0x562bd082c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52040/0 (socket says 172.21.15.116:52040) 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36f61db640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd081b100 0x562bd0825390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:56252/0 (socket says 172.21.15.116:56252) 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36f59da640 1 -- 172.21.15.116:0/974004213 learned_addr learned my addr 172.21.15.116:0/974004213 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36f59da640 1 -- 172.21.15.116:0/974004213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd081b100 msgr2=0x562bd0825390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36f59da640 1 --2- 172.21.15.116:0/974004213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd081b100 0x562bd0825390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36f59da640 1 -- 172.21.15.116:0/974004213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd0820c60 msgr2=0x562bd0815220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36f59da640 1 --2- 172.21.15.116:0/974004213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd0820c60 0x562bd0815220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36f59da640 1 -- 172.21.15.116:0/974004213 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bd0560620 con 0x562bd081baf0 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.351+0000 7f36f69dc640 1 --2- 172.21.15.116:0/974004213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd0820c60 0x562bd0815220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:22:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.352+0000 7f36f59da640 1 --2- 172.21.15.116:0/974004213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081baf0 0x562bd082c5b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f36ec036040 tx=0x7f36ec02e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efe001d2f7b3482e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:27.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.352+0000 7f36dbfff640 1 -- 172.21.15.116:0/974004213 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec04c070 con 0x562bd081baf0 2024-03-19T01:22:27.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.352+0000 7f36dbfff640 1 -- 172.21.15.116:0/974004213 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36ec048810 con 0x562bd081baf0 2024-03-19T01:22:27.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.352+0000 7f36dbfff640 1 -- 172.21.15.116:0/974004213 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec000e60 con 0x562bd081baf0 2024-03-19T01:22:27.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.353+0000 7f36fb020740 1 -- 172.21.15.116:0/974004213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081baf0 msgr2=0x562bd082c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:27.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.353+0000 7f36fb020740 1 --2- 172.21.15.116:0/974004213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081baf0 0x562bd082c5b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f36ec036040 tx=0x7f36ec02e040 comp rx=0 tx=0).stop 2024-03-19T01:22:27.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.353+0000 7f36fb020740 1 -- 172.21.15.116:0/974004213 shutdown_connections 2024-03-19T01:22:27.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.353+0000 7f36fb020740 1 --2- 172.21.15.116:0/974004213 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd0820c60 0x562bd0815220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.353+0000 7f36fb020740 1 --2- 172.21.15.116:0/974004213 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd081baf0 0x562bd082c5b0 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f36ec036040 tx=0x7f36ec02e040 comp rx=0 tx=0).stop 2024-03-19T01:22:27.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.353+0000 7f36fb020740 1 --2- 172.21.15.116:0/974004213 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd081b100 0x562bd0825390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.353+0000 7f36fb020740 1 -- 172.21.15.116:0/974004213 >> 172.21.15.116:0/974004213 conn(0x562bd0816670 msgr2=0x562bd0824c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:27.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.353+0000 7f36fb020740 1 -- 172.21.15.116:0/974004213 shutdown_connections 2024-03-19T01:22:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.354+0000 7f36fb020740 1 -- 172.21.15.116:0/974004213 wait complete. 2024-03-19T01:22:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.354+0000 7f36fb020740 1 -- start start 2024-03-19T01:22:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.354+0000 7f36fb020740 1 Processor -- start 2024-03-19T01:22:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.354+0000 7f36fb020740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd081b100 0x562bd0829680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.354+0000 7f36fb020740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd0820c60 0x562bd0829bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.354+0000 7f36fb020740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd082ab80 0x562bd082b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.354+0000 7f36fb020740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36f01aca80 con 0x562bd082ab80 2024-03-19T01:22:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.354+0000 7f36fb020740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f36f01ac3b0 con 0x562bd081b100 2024-03-19T01:22:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.354+0000 7f36fb020740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f36f01a95f0 con 0x562bd0820c60 2024-03-19T01:22:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.354+0000 7f36f59da640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd0820c60 0x562bd0829bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.354+0000 7f36fb020740 10 client.0 _ll_register_callbacks cb 0x7f36ec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:22:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.354+0000 7f36f61db640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd081b100 0x562bd0829680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36f69dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd082ab80 0x562bd082b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36f61db640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd081b100 0x562bd0829680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:56266/0 (socket says 172.21.15.116:56266) 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36f59da640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd0820c60 0x562bd0829bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52042/0 (socket says 172.21.15.116:52042) 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36f61db640 1 -- 172.21.15.116:0/2969404826 learned_addr learned my addr 172.21.15.116:0/2969404826 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36f59da640 1 -- 172.21.15.116:0/2969404826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd081b100 msgr2=0x562bd0829680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36f59da640 1 --2- 172.21.15.116:0/2969404826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd081b100 0x562bd0829680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36f59da640 1 -- 172.21.15.116:0/2969404826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd082ab80 msgr2=0x562bd082b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36f59da640 1 --2- 172.21.15.116:0/2969404826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd082ab80 0x562bd082b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36f59da640 1 -- 172.21.15.116:0/2969404826 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bd082b540 con 0x562bd0820c60 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36f61db640 1 --2- 172.21.15.116:0/2969404826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd081b100 0x562bd0829680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36f59da640 1 --2- 172.21.15.116:0/2969404826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd0820c60 0x562bd0829bc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x562bd08084a0 tx=0x7f36ec000e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=562ca7094263843d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36d9ffb640 1 -- 172.21.15.116:0/2969404826 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec055030 con 0x562bd0820c60 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36d9ffb640 1 -- 172.21.15.116:0/2969404826 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f36ec048330 con 0x562bd0820c60 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36fb020740 1 -- 172.21.15.116:0/2969404826 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36ec048810 con 0x562bd0820c60 2024-03-19T01:22:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36d9ffb640 1 -- 172.21.15.116:0/2969404826 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f36ec060070 con 0x562bd0820c60 2024-03-19T01:22:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.355+0000 7f36fb020740 10 client.25888 fetch_fsmap learned FSMap version 45 2024-03-19T01:22:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.356+0000 7f36fb020740 1 -- 172.21.15.116:0/2969404826 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x562bd082c850 con 0x562bd0820c60 2024-03-19T01:22:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.356+0000 7f36d9ffb640 1 -- 172.21.15.116:0/2969404826 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36ec048810 con 0x562bd0820c60 2024-03-19T01:22:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.356+0000 7f36d9ffb640 1 -- 172.21.15.116:0/2969404826 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f36ec0051d0 con 0x562bd0820c60 2024-03-19T01:22:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.356+0000 7f36fb020740 10 client.25888 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:22:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.356+0000 7f36fb020740 10 client.25888 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:22:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.356+0000 7f36fb020740 20 client.25888 populate_metadata read hostname 'smithi116' 2024-03-19T01:22:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.356+0000 7f36fb020740 1 --2- 172.21.15.116:0/2969404826 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562bd08973c0 0x562bd08997e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.356+0000 7f36fb020740 4 client.25888 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:22:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.356+0000 7f36fb020740 1 -- 172.21.15.116:0/2969404826 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562bd078d120 con 0x562bd08973c0 2024-03-19T01:22:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.356+0000 7f36f61db640 1 --2- 172.21.15.116:0/2969404826 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562bd08973c0 0x562bd08997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.357+0000 7f36d9ffb640 10 client.25888 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:27.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.357+0000 7f36f61db640 1 --2- 172.21.15.116:0/2969404826 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562bd08973c0 0x562bd08997e0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e62de56c05bdf81 server_cookie=b72e4e9aa59ed2e5 in_seq=0 out_seq=0 2024-03-19T01:22:27.373 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.372+0000 7f36d9ffb640 1 -- 172.21.15.116:0/2969404826 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562bd078d120 con 0x562bd08973c0 2024-03-19T01:22:27.373 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.372+0000 7f36d9ffb640 10 client.25888 handle_command_reply: tid=0 2024-03-19T01:22:27.373 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.372+0000 7f36fb020740 1 client.25888 shutdown 2024-03-19T01:22:27.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.373+0000 7f36fb020740 1 -- 172.21.15.116:0/2969404826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd0820c60 msgr2=0x562bd0829bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:27.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.373+0000 7f36fb020740 1 --2- 172.21.15.116:0/2969404826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd0820c60 0x562bd0829bc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x562bd08084a0 tx=0x7f36ec000e60 comp rx=0 tx=0).stop 2024-03-19T01:22:27.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.373+0000 7f36fb020740 1 -- 172.21.15.116:0/2969404826 shutdown_connections 2024-03-19T01:22:27.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.373+0000 7f36fb020740 1 --2- 172.21.15.116:0/2969404826 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562bd082ab80 0x562bd082b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.373+0000 7f36fb020740 1 --2- 172.21.15.116:0/2969404826 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562bd08973c0 0x562bd08997e0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.373+0000 7f36d9ffb640 0 client.25888 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:27.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.373+0000 7f36fb020740 1 --2- 172.21.15.116:0/2969404826 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562bd0820c60 0x562bd0829bc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.373+0000 7f36fb020740 1 --2- 172.21.15.116:0/2969404826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562bd081b100 0x562bd0829680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.373+0000 7f36fb020740 1 -- 172.21.15.116:0/2969404826 >> 172.21.15.116:0/2969404826 conn(0x562bd0816670 msgr2=0x562bd0825370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.373+0000 7f36fb020740 1 -- 172.21.15.116:0/2969404826 shutdown_connections 2024-03-19T01:22:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.373+0000 7f36fb020740 1 -- 172.21.15.116:0/2969404826 wait complete. 2024-03-19T01:22:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.374+0000 7f36fb020740 20 client.25888 trim_cache size 0 max 16384 2024-03-19T01:22:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:22:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:22:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:22:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:22:27.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.374+0000 7f36f843f640 1 -- 172.21.15.116:0/340478955 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f36dc0a28a0 msgr2=0x7f36dc0a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:27.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.374+0000 7f36f843f640 1 --2- 172.21.15.116:0/340478955 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f36dc0a28a0 0x7f36dc0a4d60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f36f01028a0 tx=0x7f36e0005f90 comp rx=0 tx=0).stop 2024-03-19T01:22:27.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.374+0000 7f36f843f640 1 -- 172.21.15.116:0/340478955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f019f4b0 msgr2=0x7f36f01a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:27.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.374+0000 7f36f843f640 1 --2- 172.21.15.116:0/340478955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f019f4b0 0x7f36f01a18a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009fd0 tx=0x7f36e800b260 comp rx=0 tx=0).stop 2024-03-19T01:22:27.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.374+0000 7f36f843f640 1 -- 172.21.15.116:0/340478955 shutdown_connections 2024-03-19T01:22:27.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.374+0000 7f36f843f640 1 --2- 172.21.15.116:0/340478955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36f019f4b0 0x7f36f01a18a0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.374+0000 7f36f843f640 1 --2- 172.21.15.116:0/340478955 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f36dc0a28a0 0x7f36dc0a4d60 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.374+0000 7f36f843f640 1 --2- 172.21.15.116:0/340478955 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36f0103400 0x7f36f019a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.374+0000 7f36f843f640 1 --2- 172.21.15.116:0/340478955 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f36f0101850 0x7f36f0199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:27.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.374+0000 7f36f843f640 1 -- 172.21.15.116:0/340478955 >> 172.21.15.116:0/340478955 conn(0x7f36f00fcde0 msgr2=0x7f36f0073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:27.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.375+0000 7f36f843f640 1 -- 172.21.15.116:0/340478955 shutdown_connections 2024-03-19T01:22:27.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:27.375+0000 7f36f843f640 1 -- 172.21.15.116:0/340478955 wait complete. 2024-03-19T01:22:27.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:26 smithi162 ceph-mon[67178]: pgmap v646: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.6 MiB/s wr, 705 op/s 2024-03-19T01:22:27.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:26 smithi116 ceph-mon[62701]: pgmap v646: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.6 MiB/s wr, 705 op/s 2024-03-19T01:22:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:26 smithi156 ceph-mon[66799]: pgmap v646: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.6 MiB/s wr, 705 op/s 2024-03-19T01:22:29.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:28 smithi162 ceph-mon[67178]: pgmap v647: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.6 MiB/s wr, 600 op/s 2024-03-19T01:22:29.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:28 smithi116 ceph-mon[62701]: pgmap v647: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.6 MiB/s wr, 600 op/s 2024-03-19T01:22:29.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:28 smithi156 ceph-mon[66799]: pgmap v647: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.6 MiB/s wr, 600 op/s 2024-03-19T01:22:31.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:30 smithi162 ceph-mon[67178]: pgmap v648: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 745 op/s 2024-03-19T01:22:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:30 smithi116 ceph-mon[62701]: pgmap v648: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 745 op/s 2024-03-19T01:22:31.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:30 smithi156 ceph-mon[66799]: pgmap v648: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 MiB/s wr, 745 op/s 2024-03-19T01:22:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 585.0 s, 1784.8 tps, lat 55.567 ms stddev 72.659 2024-03-19T01:22:33.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:32 smithi162 ceph-mon[67178]: pgmap v649: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 14 MiB/s wr, 795 op/s 2024-03-19T01:22:33.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:32 smithi116 ceph-mon[62701]: pgmap v649: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 14 MiB/s wr, 795 op/s 2024-03-19T01:22:33.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:32 smithi156 ceph-mon[66799]: pgmap v649: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 14 MiB/s wr, 795 op/s 2024-03-19T01:22:35.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:34 smithi162 ceph-mon[67178]: pgmap v650: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 14 MiB/s wr, 758 op/s 2024-03-19T01:22:35.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:34 smithi116 ceph-mon[62701]: pgmap v650: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 14 MiB/s wr, 758 op/s 2024-03-19T01:22:35.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:34 smithi156 ceph-mon[66799]: pgmap v650: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 14 MiB/s wr, 758 op/s 2024-03-19T01:22:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 590.0 s, 2001.8 tps, lat 50.016 ms stddev 62.413 2024-03-19T01:22:37.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:37 smithi162 ceph-mon[67178]: pgmap v651: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 MiB/s wr, 798 op/s 2024-03-19T01:22:37.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:37 smithi116 ceph-mon[62701]: pgmap v651: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 MiB/s wr, 798 op/s 2024-03-19T01:22:37.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:37 smithi156 ceph-mon[66799]: pgmap v651: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 17 MiB/s wr, 798 op/s 2024-03-19T01:22:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:39 smithi162 ceph-mon[67178]: pgmap v652: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 19 MiB/s wr, 787 op/s 2024-03-19T01:22:39.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:39 smithi116 ceph-mon[62701]: pgmap v652: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 19 MiB/s wr, 787 op/s 2024-03-19T01:22:39.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:39 smithi156 ceph-mon[66799]: pgmap v652: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 19 MiB/s wr, 787 op/s 2024-03-19T01:22:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.496+0000 7f55897b2640 1 -- 172.21.15.116:0/182851894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55841013a0 msgr2=0x7f5584108c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.496+0000 7f55897b2640 1 --2- 172.21.15.116:0/182851894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55841013a0 0x7f5584108c40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f557c009c10 tx=0x7f557c02bef0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.496+0000 7f55897b2640 1 -- 172.21.15.116:0/182851894 shutdown_connections 2024-03-19T01:22:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.496+0000 7f55897b2640 1 --2- 172.21.15.116:0/182851894 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55841013a0 0x7f5584108c40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.496+0000 7f55897b2640 1 --2- 172.21.15.116:0/182851894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f55841009e0 0x7f5584100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.496+0000 7f55897b2640 1 --2- 172.21.15.116:0/182851894 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5584101d40 0x7f5584102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.496+0000 7f55897b2640 1 -- 172.21.15.116:0/182851894 >> 172.21.15.116:0/182851894 conn(0x7f55840fb470 msgr2=0x7f55840fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.496+0000 7f55897b2640 1 -- 172.21.15.116:0/182851894 shutdown_connections 2024-03-19T01:22:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.497+0000 7f55897b2640 1 -- 172.21.15.116:0/182851894 wait complete. 2024-03-19T01:22:40.498 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.497+0000 7f55897b2640 1 Processor -- start 2024-03-19T01:22:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.497+0000 7f55897b2640 1 -- start start 2024-03-19T01:22:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f55897b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55841009e0 0x7f55841004a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f55897b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55841013a0 0x7f558419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f55897b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584101d40 0x7f558419abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f55897b2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f558410c3d0 con 0x7f5584101d40 2024-03-19T01:22:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f55897b2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f558410c250 con 0x7f55841013a0 2024-03-19T01:22:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f55897b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f558410c550 con 0x7f55841009e0 2024-03-19T01:22:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f557bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55841013a0 0x7f558419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f557bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55841013a0 0x7f558419a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:47322/0 (socket says 172.21.15.116:47322) 2024-03-19T01:22:40.501 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f557bfff640 1 -- 172.21.15.116:0/27973331 learned_addr learned my addr 172.21.15.116:0/27973331 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f557bfff640 1 -- 172.21.15.116:0/27973331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55841009e0 msgr2=0x7f55841004a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:22:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f557bfff640 1 --2- 172.21.15.116:0/27973331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55841009e0 0x7f55841004a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f557bfff640 1 -- 172.21.15.116:0/27973331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584101d40 msgr2=0x7f558419abf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:22:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f557bfff640 1 --2- 172.21.15.116:0/27973331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584101d40 0x7f558419abf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.498+0000 7f557bfff640 1 -- 172.21.15.116:0/27973331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558419b4b0 con 0x7f55841013a0 2024-03-19T01:22:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.499+0000 7f557bfff640 1 --2- 172.21.15.116:0/27973331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55841013a0 0x7f558419a590 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f556c00e9c0 tx=0x7f556c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.501+0000 7f5580ff9640 1 -- 172.21.15.116:0/27973331 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c00ccc0 con 0x7f55841013a0 2024-03-19T01:22:40.502 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.501+0000 7f5580ff9640 1 -- 172.21.15.116:0/27973331 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f556c005500 con 0x7f55841013a0 2024-03-19T01:22:40.505 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.502+0000 7f5580ff9640 1 -- 172.21.15.116:0/27973331 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c010430 con 0x7f55841013a0 2024-03-19T01:22:40.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.509+0000 7f55897b2640 1 -- 172.21.15.116:0/27973331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f558419b7a0 con 0x7f55841013a0 2024-03-19T01:22:40.511 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.509+0000 7f55897b2640 1 -- 172.21.15.116:0/27973331 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f558419bc10 con 0x7f55841013a0 2024-03-19T01:22:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.513+0000 7f558c3a0740 1 Processor -- start 2024-03-19T01:22:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.515+0000 7f558c3a0740 1 -- start start 2024-03-19T01:22:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.515+0000 7f558c3a0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343100 0x55aab5343520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.515+0000 7f558c3a0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 0x55aab53485f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.515+0000 7f558c3a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5348c60 0x55aab534b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.515+0000 7f558c3a0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aab52a7be0 con 0x55aab5343af0 2024-03-19T01:22:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.515+0000 7f558c3a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55aab5250960 con 0x55aab5348c60 2024-03-19T01:22:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.515+0000 7f558c3a0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55aab52a3c80 con 0x55aab5343100 2024-03-19T01:22:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.515+0000 7f5580ff9640 1 -- 172.21.15.116:0/27973331 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f556c004680 con 0x7f55841013a0 2024-03-19T01:22:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.516+0000 7f5580ff9640 1 --2- 172.21.15.116:0/27973331 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f55700a28e0 0x7f55700a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.517 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.516+0000 7f5580ff9640 1 -- 172.21.15.116:0/27973331 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f556c0cee90 con 0x7f55841013a0 2024-03-19T01:22:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.519+0000 7f557bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5348c60 0x55aab534b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.519+0000 7f55837fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343100 0x55aab5343520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.519+0000 7f55837fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343100 0x55aab5343520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42420/0 (socket says 172.21.15.116:42420) 2024-03-19T01:22:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.519+0000 7f55837fe640 1 -- 172.21.15.116:0/2497196947 learned_addr learned my addr 172.21.15.116:0/2497196947 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.521+0000 7f5582ffd640 1 --2- 172.21.15.116:0/2497196947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 0x55aab53485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.521 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.521+0000 7f55837fe640 1 --2- 172.21.15.116:0/27973331 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f55700a28e0 0x7f55700a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.522+0000 7f557bfff640 1 -- 172.21.15.116:0/2497196947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343100 msgr2=0x55aab5343520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.522+0000 7f557bfff640 1 --2- 172.21.15.116:0/2497196947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343100 0x55aab5343520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.522+0000 7f557bfff640 1 -- 172.21.15.116:0/2497196947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 msgr2=0x55aab53485f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.522+0000 7f557bfff640 1 --2- 172.21.15.116:0/2497196947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 0x55aab53485f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.523 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.522+0000 7f557bfff640 1 -- 172.21.15.116:0/2497196947 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aab5247d20 con 0x55aab5348c60 2024-03-19T01:22:40.526 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.526+0000 7f557bfff640 1 --2- 172.21.15.116:0/2497196947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5348c60 0x55aab534b0f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f556c00c6a0 tx=0x7f556c09b2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c9b2bc36c40727a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:40.530 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.528+0000 7f55837fe640 1 --2- 172.21.15.116:0/27973331 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f55700a28e0 0x7f55700a4da0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f557c0099a0 tx=0x7f557c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:40.530 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.530+0000 7f55797fa640 1 -- 172.21.15.116:0/2497196947 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c0b3020 con 0x55aab5348c60 2024-03-19T01:22:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.534+0000 7f55797fa640 1 -- 172.21.15.116:0/2497196947 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f556c09beb0 con 0x55aab5348c60 2024-03-19T01:22:40.534 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.534+0000 7f55797fa640 1 -- 172.21.15.116:0/2497196947 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c0afb90 con 0x55aab5348c60 2024-03-19T01:22:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.535+0000 7f558c3a0740 1 -- 172.21.15.116:0/2497196947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5348c60 msgr2=0x55aab534b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:40.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.535+0000 7f558c3a0740 1 --2- 172.21.15.116:0/2497196947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5348c60 0x55aab534b0f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f556c00c6a0 tx=0x7f556c09b2f0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.536+0000 7f558c3a0740 1 -- 172.21.15.116:0/2497196947 shutdown_connections 2024-03-19T01:22:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.536+0000 7f558c3a0740 1 --2- 172.21.15.116:0/2497196947 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5348c60 0x55aab534b0f0 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f556c00c6a0 tx=0x7f556c09b2f0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.536+0000 7f558c3a0740 1 --2- 172.21.15.116:0/2497196947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 0x55aab53485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.536+0000 7f558c3a0740 1 --2- 172.21.15.116:0/2497196947 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343100 0x55aab5343520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.536+0000 7f558c3a0740 1 -- 172.21.15.116:0/2497196947 >> 172.21.15.116:0/2497196947 conn(0x55aab533e670 msgr2=0x55aab533ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:40.543 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.543+0000 7f558c3a0740 1 -- 172.21.15.116:0/2497196947 shutdown_connections 2024-03-19T01:22:40.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.544+0000 7f558c3a0740 1 -- 172.21.15.116:0/2497196947 wait complete. 2024-03-19T01:22:40.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.545+0000 7f558c3a0740 1 -- start start 2024-03-19T01:22:40.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.545+0000 7f558c3a0740 1 Processor -- start 2024-03-19T01:22:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.546+0000 7f558c3a0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343100 0x55aab534d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.546+0000 7f558c3a0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343af0 0x55aab534d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.546+0000 7f558c3a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5352800 0x55aab5354bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.546+0000 7f558c3a0740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55aab534bc10 con 0x55aab5343100 2024-03-19T01:22:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.546+0000 7f558c3a0740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55aab5263db0 con 0x55aab5352800 2024-03-19T01:22:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.546+0000 7f558c3a0740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55aab534bd90 con 0x55aab5343af0 2024-03-19T01:22:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.546+0000 7f558c3a0740 10 client.0 _ll_register_callbacks cb 0x55aab5268e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:22:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.546+0000 7f5582ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343100 0x55aab534d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.546+0000 7f5582ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343100 0x55aab534d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53370/0 (socket says 172.21.15.116:53370) 2024-03-19T01:22:40.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.546+0000 7f5582ffd640 1 -- 172.21.15.116:0/1503167742 learned_addr learned my addr 172.21.15.116:0/1503167742 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:40.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.547+0000 7f5582ffd640 1 -- 172.21.15.116:0/1503167742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343af0 msgr2=0x55aab534d7d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:22:40.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.547+0000 7f5582ffd640 1 --2- 172.21.15.116:0/1503167742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343af0 0x55aab534d7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.547+0000 7f5582ffd640 1 -- 172.21.15.116:0/1503167742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5352800 msgr2=0x55aab5354bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:40.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.547+0000 7f5582ffd640 1 --2- 172.21.15.116:0/1503167742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5352800 0x55aab5354bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.547+0000 7f5582ffd640 1 -- 172.21.15.116:0/1503167742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aab5262c60 con 0x55aab5343100 2024-03-19T01:22:40.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.547+0000 7f5582ffd640 1 --2- 172.21.15.116:0/1503167742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343100 0x55aab534d290 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5574011410 tx=0x7f55740118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac3918e30071d85 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:40.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.547+0000 7f556a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:22:40.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.547+0000 7f556a7fc640 1 -- 172.21.15.116:0/1503167742 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574005220 con 0x55aab5343100 2024-03-19T01:22:40.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.548+0000 7f556a7fc640 1 -- 172.21.15.116:0/1503167742 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f55740053c0 con 0x55aab5343100 2024-03-19T01:22:40.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.548+0000 7f558c3a0740 1 -- 172.21.15.116:0/1503167742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f556c09beb0 con 0x55aab5343100 2024-03-19T01:22:40.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.548+0000 7f556a7fc640 1 -- 172.21.15.116:0/1503167742 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f55740055c0 con 0x55aab5343100 2024-03-19T01:22:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.550+0000 7f558c3a0740 10 client.16062 fetch_fsmap learned FSMap version 45 2024-03-19T01:22:40.550 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.550+0000 7f558c3a0740 1 -- 172.21.15.116:0/1503167742 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55aab4fbc210 con 0x55aab5343100 2024-03-19T01:22:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.551+0000 7f556a7fc640 1 -- 172.21.15.116:0/1503167742 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f556c09beb0 con 0x55aab5343100 2024-03-19T01:22:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.551+0000 7f556a7fc640 1 -- 172.21.15.116:0/1503167742 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f5574024470 con 0x55aab5343100 2024-03-19T01:22:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.551+0000 7f558c3a0740 10 client.16062 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:22:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.551+0000 7f558c3a0740 10 client.16062 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:22:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.551+0000 7f558c3a0740 20 client.16062 populate_metadata read hostname 'smithi116' 2024-03-19T01:22:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.551+0000 7f558c3a0740 1 --2- 172.21.15.116:0/1503167742 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55aab5359630 0x55aab535bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.551+0000 7f558c3a0740 4 client.16062 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:22:40.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.551+0000 7f558c3a0740 1 -- 172.21.15.116:0/1503167742 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aab52b5120 con 0x55aab5359630 2024-03-19T01:22:40.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.551+0000 7f557bfff640 1 --2- 172.21.15.116:0/1503167742 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55aab5359630 0x55aab535bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.553+0000 7f557bfff640 1 --2- 172.21.15.116:0/1503167742 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55aab5359630 0x55aab535bad0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8832f80f73adfa3b server_cookie=c1260f4a4ed6a000 in_seq=0 out_seq=0 2024-03-19T01:22:40.553 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.553+0000 7f556a7fc640 10 client.16062 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.556+0000 7f556a7fc640 1 -- 172.21.15.116:0/1503167742 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55aab52b5120 con 0x55aab5359630 2024-03-19T01:22:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.556+0000 7f556a7fc640 10 client.16062 handle_command_reply: tid=0 2024-03-19T01:22:40.556 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.556+0000 7f558c3a0740 1 client.16062 shutdown 2024-03-19T01:22:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.557+0000 7f558c3a0740 1 -- 172.21.15.116:0/1503167742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343100 msgr2=0x55aab534d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:40.557 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.557+0000 7f558c3a0740 1 --2- 172.21.15.116:0/1503167742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343100 0x55aab534d290 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5574011410 tx=0x7f55740118d0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.559+0000 7f558c3a0740 1 -- 172.21.15.116:0/1503167742 shutdown_connections 2024-03-19T01:22:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.559+0000 7f558c3a0740 1 --2- 172.21.15.116:0/1503167742 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5352800 0x55aab5354bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.559+0000 7f558c3a0740 1 --2- 172.21.15.116:0/1503167742 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55aab5359630 0x55aab535bad0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.559+0000 7f558c3a0740 1 --2- 172.21.15.116:0/1503167742 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343af0 0x55aab534d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.559+0000 7f558c3a0740 1 --2- 172.21.15.116:0/1503167742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343100 0x55aab534d290 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.560+0000 7f558c3a0740 1 -- 172.21.15.116:0/1503167742 >> 172.21.15.116:0/1503167742 conn(0x55aab533e670 msgr2=0x55aab533f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:40.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.560+0000 7f556a7fc640 0 client.16062 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:40.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.560+0000 7f558c3a0740 1 -- 172.21.15.116:0/1503167742 shutdown_connections 2024-03-19T01:22:40.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.561+0000 7f558c3a0740 1 -- 172.21.15.116:0/1503167742 wait complete. 2024-03-19T01:22:40.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.563+0000 7f558c3a0740 20 client.16062 trim_cache size 0 max 16384 2024-03-19T01:22:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.594+0000 7f558c3a0740 1 Processor -- start 2024-03-19T01:22:40.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.594+0000 7f558c3a0740 1 -- start start 2024-03-19T01:22:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.594+0000 7f558c3a0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343100 0x55aab533dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.594+0000 7f558c3a0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 0x55aab5341fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.594+0000 7f558c3a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5352800 0x55aab534c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.594+0000 7f5582ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343100 0x55aab533dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.594+0000 7f5582ffd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343100 0x55aab533dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:42432/0 (socket says 172.21.15.116:42432) 2024-03-19T01:22:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.594+0000 7f5582ffd640 1 -- 172.21.15.116:0/3737211287 learned_addr learned my addr 172.21.15.116:0/3737211287 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.595+0000 7f558c3a0740 1 -- 172.21.15.116:0/3737211287 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f55841a4b20 con 0x55aab5343af0 2024-03-19T01:22:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.595+0000 7f558c3a0740 1 -- 172.21.15.116:0/3737211287 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f55841b9680 con 0x55aab5352800 2024-03-19T01:22:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.595+0000 7f558c3a0740 1 -- 172.21.15.116:0/3737211287 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f55841a51f0 con 0x55aab5343100 2024-03-19T01:22:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.595+0000 7f55837fe640 1 --2- 172.21.15.116:0/3737211287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5352800 0x55aab534c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.596 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.595+0000 7f557bfff640 1 --2- 172.21.15.116:0/3737211287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 0x55aab5341fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.595+0000 7f55837fe640 1 -- 172.21.15.116:0/3737211287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343100 msgr2=0x55aab533dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.595+0000 7f55837fe640 1 --2- 172.21.15.116:0/3737211287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343100 0x55aab533dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.595+0000 7f55837fe640 1 -- 172.21.15.116:0/3737211287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 msgr2=0x55aab5341fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.595+0000 7f55837fe640 1 --2- 172.21.15.116:0/3737211287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 0x55aab5341fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.595+0000 7f55837fe640 1 -- 172.21.15.116:0/3737211287 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aab5088620 con 0x55aab5352800 2024-03-19T01:22:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.595+0000 7f55837fe640 1 --2- 172.21.15.116:0/3737211287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5352800 0x55aab534c560 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55aab5344350 tx=0x7f557c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22d8a396ee5635c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.596+0000 7f5578ff9640 1 -- 172.21.15.116:0/3737211287 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c052070 con 0x55aab5352800 2024-03-19T01:22:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.596+0000 7f5578ff9640 1 -- 172.21.15.116:0/3737211287 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f557c003d40 con 0x55aab5352800 2024-03-19T01:22:40.597 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.596+0000 7f5578ff9640 1 -- 172.21.15.116:0/3737211287 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c043650 con 0x55aab5352800 2024-03-19T01:22:40.598 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.597+0000 7f558c3a0740 1 -- 172.21.15.116:0/3737211287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5352800 msgr2=0x55aab534c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.597+0000 7f558c3a0740 1 --2- 172.21.15.116:0/3737211287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5352800 0x55aab534c560 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55aab5344350 tx=0x7f557c005fd0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.597+0000 7f558c3a0740 1 -- 172.21.15.116:0/3737211287 shutdown_connections 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.597+0000 7f558c3a0740 1 --2- 172.21.15.116:0/3737211287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5352800 0x55aab534c560 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.597+0000 7f558c3a0740 1 --2- 172.21.15.116:0/3737211287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 0x55aab5341fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.597+0000 7f558c3a0740 1 --2- 172.21.15.116:0/3737211287 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5343100 0x55aab533dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.597+0000 7f558c3a0740 1 -- 172.21.15.116:0/3737211287 >> 172.21.15.116:0/3737211287 conn(0x55aab533e670 msgr2=0x55aab53418d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.597+0000 7f558c3a0740 1 -- 172.21.15.116:0/3737211287 shutdown_connections 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.597+0000 7f558c3a0740 1 -- 172.21.15.116:0/3737211287 wait complete. 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.597+0000 7f558c3a0740 1 -- start start 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.598+0000 7f558c3a0740 1 Processor -- start 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.598+0000 7f558c3a0740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5343100 0x55aab536a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.598+0000 7f558c3a0740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 0x55aab536ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.598+0000 7f558c3a0740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5352800 0x55aab536bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.598+0000 7f5582ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5343100 0x55aab536a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.598+0000 7f5582ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5343100 0x55aab536a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:47380/0 (socket says 172.21.15.116:47380) 2024-03-19T01:22:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.598+0000 7f5582ffd640 1 -- 172.21.15.116:0/98945926 learned_addr learned my addr 172.21.15.116:0/98945926 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.598+0000 7f558c3a0740 1 -- 172.21.15.116:0/98945926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f55841b8fb0 con 0x55aab5343af0 2024-03-19T01:22:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.598+0000 7f558c3a0740 1 -- 172.21.15.116:0/98945926 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f55841b88e0 con 0x55aab5343100 2024-03-19T01:22:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.598+0000 7f558c3a0740 1 -- 172.21.15.116:0/98945926 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f55841a2b00 con 0x55aab5352800 2024-03-19T01:22:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.598+0000 7f55837fe640 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5352800 0x55aab536bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.598+0000 7f558c3a0740 10 client.0 _ll_register_callbacks cb 0x7f5574009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:22:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.599+0000 7f5582ffd640 1 -- 172.21.15.116:0/98945926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5352800 msgr2=0x55aab536bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.599+0000 7f5582ffd640 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5352800 0x55aab536bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.599+0000 7f5582ffd640 1 -- 172.21.15.116:0/98945926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 msgr2=0x55aab536ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.599+0000 7f557bfff640 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 0x55aab536ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.599+0000 7f5582ffd640 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 0x55aab536ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.599+0000 7f5582ffd640 1 -- 172.21.15.116:0/98945926 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aab536c320 con 0x55aab5343100 2024-03-19T01:22:40.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.599+0000 7f557bfff640 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 0x55aab536ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:22:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.600+0000 7f5582ffd640 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5343100 0x55aab536a7e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5574024890 tx=0x7f5574024b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d32eb66672da963d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.600+0000 7f556a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:22:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.600+0000 7f556a7fc640 1 -- 172.21.15.116:0/98945926 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574034060 con 0x55aab5343100 2024-03-19T01:22:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.600+0000 7f556a7fc640 1 -- 172.21.15.116:0/98945926 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f5574012b30 con 0x55aab5343100 2024-03-19T01:22:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.600+0000 7f558c3a0740 1 -- 172.21.15.116:0/98945926 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f557c003d40 con 0x55aab5343100 2024-03-19T01:22:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.601+0000 7f556a7fc640 1 -- 172.21.15.116:0/98945926 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f5574024cf0 con 0x55aab5343100 2024-03-19T01:22:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.601+0000 7f558c3a0740 10 client.25913 fetch_fsmap learned FSMap version 45 2024-03-19T01:22:40.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.601+0000 7f558c3a0740 1 -- 172.21.15.116:0/98945926 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55aab53bf8f0 con 0x55aab5343100 2024-03-19T01:22:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.601+0000 7f556a7fc640 1 -- 172.21.15.116:0/98945926 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f557c003d40 con 0x55aab5343100 2024-03-19T01:22:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.601+0000 7f556a7fc640 1 -- 172.21.15.116:0/98945926 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f557401a400 con 0x55aab5343100 2024-03-19T01:22:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.602+0000 7f558c3a0740 10 client.25913 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:22:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.602+0000 7f558c3a0740 10 client.25913 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:22:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.602+0000 7f558c3a0740 20 client.25913 populate_metadata read hostname 'smithi116' 2024-03-19T01:22:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.602+0000 7f558c3a0740 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55aab53c02c0 0x55aab53c26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:40.603 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.602+0000 7f558c3a0740 4 client.25913 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:22:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.602+0000 7f558c3a0740 1 -- 172.21.15.116:0/98945926 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55aab52af470 con 0x55aab53c02c0 2024-03-19T01:22:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.603+0000 7f557bfff640 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55aab53c02c0 0x55aab53c26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.603+0000 7f557bfff640 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55aab53c02c0 0x55aab53c26e0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e5fb6efa249cd50 server_cookie=2ef6c2ae58f4456a in_seq=0 out_seq=0 2024-03-19T01:22:40.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.603+0000 7f556a7fc640 10 client.25913 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f556a7fc640 1 -- 172.21.15.116:0/98945926 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55aab52af470 con 0x55aab53c02c0 2024-03-19T01:22:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f556a7fc640 10 client.25913 handle_command_reply: tid=0 2024-03-19T01:22:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f558c3a0740 1 client.25913 shutdown 2024-03-19T01:22:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f558c3a0740 1 -- 172.21.15.116:0/98945926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5343100 msgr2=0x55aab536a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f558c3a0740 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5343100 0x55aab536a7e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5574024890 tx=0x7f5574024b00 comp rx=0 tx=0).stop 2024-03-19T01:22:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f558c3a0740 1 -- 172.21.15.116:0/98945926 shutdown_connections 2024-03-19T01:22:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f558c3a0740 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55aab53c02c0 0x55aab53c26e0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f558c3a0740 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55aab5352800 0x55aab536bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f556a7fc640 0 client.25913 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f558c3a0740 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55aab5343af0 0x55aab536ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f558c3a0740 1 --2- 172.21.15.116:0/98945926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55aab5343100 0x55aab536a7e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f558c3a0740 1 -- 172.21.15.116:0/98945926 >> 172.21.15.116:0/98945926 conn(0x55aab533e670 msgr2=0x55aab5342160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:40.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f558c3a0740 1 -- 172.21.15.116:0/98945926 shutdown_connections 2024-03-19T01:22:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.612+0000 7f558c3a0740 1 -- 172.21.15.116:0/98945926 wait complete. 2024-03-19T01:22:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.613+0000 7f558c3a0740 20 client.25913 trim_cache size 0 max 16384 2024-03-19T01:22:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:22:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:22:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:22:40.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:22:40.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.613+0000 7f55897b2640 1 -- 172.21.15.116:0/27973331 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f55700a28e0 msgr2=0x7f55700a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:40.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.613+0000 7f55897b2640 1 --2- 172.21.15.116:0/27973331 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f55700a28e0 0x7f55700a4da0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f557c0099a0 tx=0x7f557c009500 comp rx=0 tx=0).stop 2024-03-19T01:22:40.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.613+0000 7f55897b2640 1 -- 172.21.15.116:0/27973331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55841013a0 msgr2=0x7f558419a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:40.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.613+0000 7f55897b2640 1 --2- 172.21.15.116:0/27973331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55841013a0 0x7f558419a590 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f556c00e9c0 tx=0x7f556c00ee80 comp rx=0 tx=0).stop 2024-03-19T01:22:40.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.613+0000 7f55897b2640 1 -- 172.21.15.116:0/27973331 shutdown_connections 2024-03-19T01:22:40.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.613+0000 7f55897b2640 1 --2- 172.21.15.116:0/27973331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584101d40 0x7f558419abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.613+0000 7f55897b2640 1 --2- 172.21.15.116:0/27973331 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f55700a28e0 0x7f55700a4da0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.613+0000 7f55897b2640 1 --2- 172.21.15.116:0/27973331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f55841013a0 0x7f558419a590 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.613+0000 7f55897b2640 1 --2- 172.21.15.116:0/27973331 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f55841009e0 0x7f55841004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:40.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.613+0000 7f55897b2640 1 -- 172.21.15.116:0/27973331 >> 172.21.15.116:0/27973331 conn(0x7f55840fb470 msgr2=0x7f5584104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:40.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.613+0000 7f55897b2640 1 -- 172.21.15.116:0/27973331 shutdown_connections 2024-03-19T01:22:40.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:40.613+0000 7f55897b2640 1 -- 172.21.15.116:0/27973331 wait complete. 2024-03-19T01:22:41.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:41 smithi162 ceph-mon[67178]: pgmap v653: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 20 MiB/s wr, 838 op/s 2024-03-19T01:22:41.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:41 smithi116 ceph-mon[62701]: pgmap v653: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 20 MiB/s wr, 838 op/s 2024-03-19T01:22:41.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:41 smithi156 ceph-mon[66799]: pgmap v653: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 20 MiB/s wr, 838 op/s 2024-03-19T01:22:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 595.0 s, 1694.0 tps, lat 59.019 ms stddev 67.501 2024-03-19T01:22:43.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:43 smithi162 ceph-mon[67178]: pgmap v654: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 17 MiB/s wr, 694 op/s 2024-03-19T01:22:43.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:43 smithi116 ceph-mon[62701]: pgmap v654: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 17 MiB/s wr, 694 op/s 2024-03-19T01:22:43.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:43 smithi156 ceph-mon[66799]: pgmap v654: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 17 MiB/s wr, 694 op/s 2024-03-19T01:22:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:45 smithi162 ceph-mon[67178]: pgmap v655: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 15 MiB/s wr, 620 op/s 2024-03-19T01:22:45.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:45 smithi116 ceph-mon[62701]: pgmap v655: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 15 MiB/s wr, 620 op/s 2024-03-19T01:22:45.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:45 smithi156 ceph-mon[66799]: pgmap v655: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 15 MiB/s wr, 620 op/s 2024-03-19T01:22:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 600.0 s, 1625.0 tps, lat 61.985 ms stddev 82.944 2024-03-19T01:22:47.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:47 smithi162 ceph-mon[67178]: pgmap v656: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 MiB/s wr, 658 op/s 2024-03-19T01:22:47.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:47 smithi116 ceph-mon[62701]: pgmap v656: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 MiB/s wr, 658 op/s 2024-03-19T01:22:47.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:47 smithi156 ceph-mon[66799]: pgmap v656: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 15 MiB/s wr, 658 op/s 2024-03-19T01:22:49.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:49 smithi162 ceph-mon[67178]: pgmap v657: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 13 MiB/s wr, 593 op/s 2024-03-19T01:22:49.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:49 smithi116 ceph-mon[62701]: pgmap v657: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 13 MiB/s wr, 593 op/s 2024-03-19T01:22:49.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:49 smithi156 ceph-mon[66799]: pgmap v657: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 13 MiB/s wr, 593 op/s 2024-03-19T01:22:51.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:51 smithi162 ceph-mon[67178]: pgmap v658: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 666 op/s 2024-03-19T01:22:51.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:51 smithi116 ceph-mon[62701]: pgmap v658: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 666 op/s 2024-03-19T01:22:51.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:51 smithi156 ceph-mon[66799]: pgmap v658: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 666 op/s 2024-03-19T01:22:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 605.0 s, 1591.8 tps, lat 61.466 ms stddev 74.137 2024-03-19T01:22:53.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:53 smithi162 ceph-mon[67178]: pgmap v659: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 605 op/s 2024-03-19T01:22:53.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:53 smithi116 ceph-mon[62701]: pgmap v659: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 605 op/s 2024-03-19T01:22:53.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:53 smithi156 ceph-mon[66799]: pgmap v659: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 605 op/s 2024-03-19T01:22:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.736+0000 7f08d2e3d640 1 -- 172.21.15.116:0/249273263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08cc0ffb00 msgr2=0x7f08cc0fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.736+0000 7f08d2e3d640 1 --2- 172.21.15.116:0/249273263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08cc0ffb00 0x7f08cc0fff20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009930 tx=0x7f08c002bc30 comp rx=0 tx=0).stop 2024-03-19T01:22:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.737+0000 7f08d2e3d640 1 -- 172.21.15.116:0/249273263 shutdown_connections 2024-03-19T01:22:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.737+0000 7f08d2e3d640 1 --2- 172.21.15.116:0/249273263 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08cc103490 0x7f08cc109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.737+0000 7f08d2e3d640 1 --2- 172.21.15.116:0/249273263 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08cc102ad0 0x7f08cc102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.737+0000 7f08d2e3d640 1 --2- 172.21.15.116:0/249273263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08cc0ffb00 0x7f08cc0fff20 secure :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009930 tx=0x7f08c002bc30 comp rx=0 tx=0).stop 2024-03-19T01:22:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.737+0000 7f08d2e3d640 1 -- 172.21.15.116:0/249273263 >> 172.21.15.116:0/249273263 conn(0x7f08cc0fb470 msgr2=0x7f08cc0fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.737+0000 7f08d2e3d640 1 -- 172.21.15.116:0/249273263 shutdown_connections 2024-03-19T01:22:53.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.737+0000 7f08d2e3d640 1 -- 172.21.15.116:0/249273263 wait complete. 2024-03-19T01:22:53.741 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.738+0000 7f08d2e3d640 1 Processor -- start 2024-03-19T01:22:53.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.738+0000 7f08d2e3d640 1 -- start start 2024-03-19T01:22:53.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.738+0000 7f08d2e3d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08cc102ad0 0x7f08cc19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.738+0000 7f08d2e3d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08cc103490 0x7f08cc19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.738+0000 7f08d2e3d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08cc1a38f0 0x7f08cc1a5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.738+0000 7f08d2e3d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc101310 con 0x7f08cc1a38f0 2024-03-19T01:22:53.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.738+0000 7f08d2e3d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc101190 con 0x7f08cc102ad0 2024-03-19T01:22:53.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.738+0000 7f08d2e3d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc101490 con 0x7f08cc103490 2024-03-19T01:22:53.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.738+0000 7f08d13da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08cc1a38f0 0x7f08cc1a5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.738+0000 7f08d0bd9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08cc102ad0 0x7f08cc19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.738+0000 7f08d0bd9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08cc102ad0 0x7f08cc19e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:35740/0 (socket says 172.21.15.116:35740) 2024-03-19T01:22:53.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.738+0000 7f08d0bd9640 1 -- 172.21.15.116:0/3490192960 learned_addr learned my addr 172.21.15.116:0/3490192960 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:53.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.738+0000 7f08cbfff640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08cc103490 0x7f08cc19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.739+0000 7f08d13da640 1 -- 172.21.15.116:0/3490192960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08cc103490 msgr2=0x7f08cc19eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.739+0000 7f08d13da640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08cc103490 0x7f08cc19eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.739+0000 7f08d13da640 1 -- 172.21.15.116:0/3490192960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08cc102ad0 msgr2=0x7f08cc19e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.739+0000 7f08d13da640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08cc102ad0 0x7f08cc19e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.739+0000 7f08d13da640 1 -- 172.21.15.116:0/3490192960 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08cc1a6380 con 0x7f08cc1a38f0 2024-03-19T01:22:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.739+0000 7f08d0bd9640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08cc102ad0 0x7f08cc19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:22:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.739+0000 7f08cbfff640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08cc103490 0x7f08cc19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:22:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.739+0000 7f08d13da640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08cc1a38f0 0x7f08cc1a5ce0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f08c400da10 tx=0x7f08c400ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.739+0000 7f08c9ffb640 1 -- 172.21.15.116:0/3490192960 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c400bd70 con 0x7f08cc1a38f0 2024-03-19T01:22:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.739+0000 7f08c9ffb640 1 -- 172.21.15.116:0/3490192960 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08c4003e50 con 0x7f08cc1a38f0 2024-03-19T01:22:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.739+0000 7f08c9ffb640 1 -- 172.21.15.116:0/3490192960 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c40106a0 con 0x7f08cc1a38f0 2024-03-19T01:22:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.739+0000 7f08d2e3d640 1 -- 172.21.15.116:0/3490192960 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08cc1a6670 con 0x7f08cc1a38f0 2024-03-19T01:22:53.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.740+0000 7f08d2e3d640 1 -- 172.21.15.116:0/3490192960 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08cc1a6b70 con 0x7f08cc1a38f0 2024-03-19T01:22:53.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.743+0000 7f08c9ffb640 1 -- 172.21.15.116:0/3490192960 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f08c4003970 con 0x7f08cc1a38f0 2024-03-19T01:22:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08d5a22740 1 Processor -- start 2024-03-19T01:22:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08d5a22740 1 -- start start 2024-03-19T01:22:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08d5a22740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03100 0x556870a03520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08d5a22740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a085f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08d5a22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a08c60 0x556870a0b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08d5a22740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556870967be0 con 0x556870a08c60 2024-03-19T01:22:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08d5a22740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556870910960 con 0x556870a03100 2024-03-19T01:22:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08d5a22740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556870963c80 con 0x556870a03af0 2024-03-19T01:22:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08cbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03100 0x556870a03520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08d0bd9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a085f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08cbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03100 0x556870a03520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:35748/0 (socket says 172.21.15.116:35748) 2024-03-19T01:22:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08d0bd9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a085f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46374/0 (socket says 172.21.15.116:46374) 2024-03-19T01:22:53.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08cbfff640 1 -- 172.21.15.116:0/338021728 learned_addr learned my addr 172.21.15.116:0/338021728 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08cbfff640 1 --2- 172.21.15.116:0/338021728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a08c60 0x556870a0b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08c9ffb640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f08bc0a2920 0x7f08bc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08cbfff640 1 -- 172.21.15.116:0/338021728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 msgr2=0x556870a085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08cbfff640 1 --2- 172.21.15.116:0/338021728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a085f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08cbfff640 1 -- 172.21.15.116:0/338021728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a08c60 msgr2=0x556870a0b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.744+0000 7f08cbfff640 1 --2- 172.21.15.116:0/338021728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a08c60 0x556870a0b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.745+0000 7f08cbfff640 1 -- 172.21.15.116:0/338021728 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556870907d20 con 0x556870a03100 2024-03-19T01:22:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.745+0000 7f08d13da640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f08bc0a2920 0x7f08bc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.745+0000 7f08c9ffb640 1 -- 172.21.15.116:0/3490192960 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f08c40cede0 con 0x7f08cc1a38f0 2024-03-19T01:22:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.745+0000 7f08cbfff640 1 --2- 172.21.15.116:0/338021728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03100 0x556870a03520 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f08b8011bb0 tx=0x7f08b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e7d7c66e6f1087a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.745+0000 7f08b6ffd640 1 -- 172.21.15.116:0/338021728 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b800fc50 con 0x556870a03100 2024-03-19T01:22:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.745+0000 7f08b6ffd640 1 -- 172.21.15.116:0/338021728 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08b800fdf0 con 0x556870a03100 2024-03-19T01:22:53.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.745+0000 7f08b6ffd640 1 -- 172.21.15.116:0/338021728 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b8003b50 con 0x556870a03100 2024-03-19T01:22:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.746+0000 7f08d5a22740 1 -- 172.21.15.116:0/338021728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03100 msgr2=0x556870a03520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.746+0000 7f08d5a22740 1 --2- 172.21.15.116:0/338021728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03100 0x556870a03520 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f08b8011bb0 tx=0x7f08b80046c0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.746+0000 7f08d13da640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f08bc0a2920 0x7f08bc0a4de0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f08c40a39d0 tx=0x7f08c4017b90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.746+0000 7f08d5a22740 1 -- 172.21.15.116:0/338021728 shutdown_connections 2024-03-19T01:22:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.746+0000 7f08d5a22740 1 --2- 172.21.15.116:0/338021728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a08c60 0x556870a0b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.746+0000 7f08d5a22740 1 --2- 172.21.15.116:0/338021728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.746+0000 7f08d5a22740 1 --2- 172.21.15.116:0/338021728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03100 0x556870a03520 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f08b8011bb0 tx=0x7f08b80046c0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.746+0000 7f08d5a22740 1 -- 172.21.15.116:0/338021728 >> 172.21.15.116:0/338021728 conn(0x5568709fe670 msgr2=0x5568709fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.746+0000 7f08d5a22740 1 -- 172.21.15.116:0/338021728 shutdown_connections 2024-03-19T01:22:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.746+0000 7f08d5a22740 1 -- 172.21.15.116:0/338021728 wait complete. 2024-03-19T01:22:53.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.746+0000 7f08d5a22740 1 -- start start 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.746+0000 7f08d5a22740 1 Processor -- start 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.747+0000 7f08d5a22740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03af0 0x556870a0d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.747+0000 7f08d5a22740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a08c60 0x556870a0d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.747+0000 7f08d5a22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a127c0 0x556870a14bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.747+0000 7f08d5a22740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x556870a0bc10 con 0x556870a127c0 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.747+0000 7f08d5a22740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x556870923db0 con 0x556870a03af0 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.747+0000 7f08d5a22740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x556870a0bd90 con 0x556870a08c60 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.747+0000 7f08d5a22740 10 client.0 _ll_register_callbacks cb 0x556870928e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.747+0000 7f08cbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03af0 0x556870a0d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.747+0000 7f08cbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03af0 0x556870a0d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:35762/0 (socket says 172.21.15.116:35762) 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.747+0000 7f08cbfff640 1 -- 172.21.15.116:0/1967815026 learned_addr learned my addr 172.21.15.116:0/1967815026 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.747+0000 7f08d0bd9640 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a08c60 0x556870a0d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.747+0000 7f08cbfff640 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a127c0 0x556870a14bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.748+0000 7f08d0bd9640 1 -- 172.21.15.116:0/1967815026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03af0 msgr2=0x556870a0d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.748+0000 7f08d0bd9640 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03af0 0x556870a0d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.748+0000 7f08d0bd9640 1 -- 172.21.15.116:0/1967815026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a127c0 msgr2=0x556870a14bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.748+0000 7f08d0bd9640 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a127c0 0x556870a14bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.748+0000 7f08d0bd9640 1 -- 172.21.15.116:0/1967815026 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556870922c60 con 0x556870a08c60 2024-03-19T01:22:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.748+0000 7f08cbfff640 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03af0 0x556870a0d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:22:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.748+0000 7f08d0bd9640 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a08c60 0x556870a0d790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009230 tx=0x7f08c00048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a79f2c5be7b89fb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.748+0000 7f08abfff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:22:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.748+0000 7f08abfff640 1 -- 172.21.15.116:0/1967815026 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0005d50 con 0x556870a08c60 2024-03-19T01:22:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.748+0000 7f08abfff640 1 -- 172.21.15.116:0/1967815026 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f08c0004ef0 con 0x556870a08c60 2024-03-19T01:22:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.749+0000 7f08d5a22740 1 -- 172.21.15.116:0/1967815026 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08b800fdf0 con 0x556870a08c60 2024-03-19T01:22:53.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.749+0000 7f08abfff640 1 -- 172.21.15.116:0/1967815026 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f08c002cd30 con 0x556870a08c60 2024-03-19T01:22:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.749+0000 7f08d5a22740 10 client.25912 fetch_fsmap learned FSMap version 45 2024-03-19T01:22:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.749+0000 7f08d5a22740 1 -- 172.21.15.116:0/1967815026 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55687067c210 con 0x556870a08c60 2024-03-19T01:22:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.749+0000 7f08abfff640 1 -- 172.21.15.116:0/1967815026 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08b800fdf0 con 0x556870a08c60 2024-03-19T01:22:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.750+0000 7f08abfff640 1 -- 172.21.15.116:0/1967815026 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f08c00039b0 con 0x556870a08c60 2024-03-19T01:22:53.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.750+0000 7f08d5a22740 10 client.25912 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:22:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.750+0000 7f08d5a22740 10 client.25912 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:22:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.750+0000 7f08d5a22740 20 client.25912 populate_metadata read hostname 'smithi116' 2024-03-19T01:22:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.750+0000 7f08d5a22740 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556870a19490 0x556870a1b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.750+0000 7f08d5a22740 4 client.25912 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:22:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.750+0000 7f08d5a22740 1 -- 172.21.15.116:0/1967815026 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556870975120 con 0x556870a19490 2024-03-19T01:22:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.750+0000 7f08cbfff640 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556870a19490 0x556870a1b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.751+0000 7f08abfff640 10 client.25912 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:53.756 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.751+0000 7f08cbfff640 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556870a19490 0x556870a1b8a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26ad4c4bdc235b4b server_cookie=b8e9e6f724d034e9 in_seq=0 out_seq=0 2024-03-19T01:22:53.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08abfff640 1 -- 172.21.15.116:0/1967815026 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556870975120 con 0x556870a19490 2024-03-19T01:22:53.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08abfff640 10 client.25912 handle_command_reply: tid=0 2024-03-19T01:22:53.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08d5a22740 1 client.25912 shutdown 2024-03-19T01:22:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08d5a22740 1 -- 172.21.15.116:0/1967815026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a08c60 msgr2=0x556870a0d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08d5a22740 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a08c60 0x556870a0d790 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009230 tx=0x7f08c00048f0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08d5a22740 1 -- 172.21.15.116:0/1967815026 shutdown_connections 2024-03-19T01:22:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08d5a22740 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a127c0 0x556870a14bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08d5a22740 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556870a19490 0x556870a1b8a0 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08d5a22740 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a08c60 0x556870a0d790 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f08c0009230 tx=0x7f08c00048f0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08d5a22740 1 --2- 172.21.15.116:0/1967815026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a03af0 0x556870a0d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08d5a22740 1 -- 172.21.15.116:0/1967815026 >> 172.21.15.116:0/1967815026 conn(0x5568709fe670 msgr2=0x5568709ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08abfff640 0 client.25912 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08d5a22740 1 -- 172.21.15.116:0/1967815026 shutdown_connections 2024-03-19T01:22:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08d5a22740 1 -- 172.21.15.116:0/1967815026 wait complete. 2024-03-19T01:22:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.752+0000 7f08d5a22740 20 client.25912 trim_cache size 0 max 16384 2024-03-19T01:22:53.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.779+0000 7f08d5a22740 1 Processor -- start 2024-03-19T01:22:53.781 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.780+0000 7f08d5a22740 1 -- start start 2024-03-19T01:22:53.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.780+0000 7f08d5a22740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a1b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.780+0000 7f08d5a22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a127c0 0x5568709fd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.780+0000 7f08d5a22740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a04560 0x556870a049f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.780+0000 7f08d5a22740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc1af880 con 0x556870a127c0 2024-03-19T01:22:53.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.780+0000 7f08d5a22740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc1b13c0 con 0x556870a04560 2024-03-19T01:22:53.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.780+0000 7f08d5a22740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc1aff50 con 0x556870a03af0 2024-03-19T01:22:53.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.783+0000 7f08d0bd9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a127c0 0x5568709fd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.783+0000 7f08d0bd9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a127c0 0x5568709fd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35246/0 (socket says 172.21.15.116:35246) 2024-03-19T01:22:53.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.783+0000 7f08d0bd9640 1 -- 172.21.15.116:0/921912903 learned_addr learned my addr 172.21.15.116:0/921912903 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:53.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.783+0000 7f08cbfff640 1 --2- 172.21.15.116:0/921912903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a1b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.783+0000 7f08cbfff640 1 --2- 172.21.15.116:0/921912903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a04560 0x556870a049f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.785+0000 7f08d0bd9640 1 -- 172.21.15.116:0/921912903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 msgr2=0x556870a1b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:53.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.785+0000 7f08d0bd9640 1 --2- 172.21.15.116:0/921912903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a1b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.785+0000 7f08d0bd9640 1 -- 172.21.15.116:0/921912903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a04560 msgr2=0x556870a049f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:53.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.785+0000 7f08d0bd9640 1 --2- 172.21.15.116:0/921912903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a04560 0x556870a049f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.785+0000 7f08d0bd9640 1 -- 172.21.15.116:0/921912903 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556870748620 con 0x556870a127c0 2024-03-19T01:22:53.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.786+0000 7f08d0bd9640 1 --2- 172.21.15.116:0/921912903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a127c0 0x5568709fd930 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f08c0003ef0 tx=0x7f08c0005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c2848e820f6265f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.790+0000 7f08b67fc640 1 -- 172.21.15.116:0/921912903 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c002cd30 con 0x556870a127c0 2024-03-19T01:22:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.790+0000 7f08b67fc640 1 -- 172.21.15.116:0/921912903 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08c0004cf0 con 0x556870a127c0 2024-03-19T01:22:53.791 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.790+0000 7f08b67fc640 1 -- 172.21.15.116:0/921912903 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c004bc30 con 0x556870a127c0 2024-03-19T01:22:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.791+0000 7f08d5a22740 1 -- 172.21.15.116:0/921912903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a127c0 msgr2=0x5568709fd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:53.793 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.792+0000 7f08d5a22740 1 --2- 172.21.15.116:0/921912903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a127c0 0x5568709fd930 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f08c0003ef0 tx=0x7f08c0005c00 comp rx=0 tx=0).stop 2024-03-19T01:22:53.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.793+0000 7f08d5a22740 1 -- 172.21.15.116:0/921912903 shutdown_connections 2024-03-19T01:22:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.793+0000 7f08d5a22740 1 --2- 172.21.15.116:0/921912903 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a04560 0x556870a049f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.793+0000 7f08d5a22740 1 --2- 172.21.15.116:0/921912903 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a127c0 0x5568709fd930 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.793+0000 7f08d5a22740 1 --2- 172.21.15.116:0/921912903 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a1b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.793+0000 7f08d5a22740 1 -- 172.21.15.116:0/921912903 >> 172.21.15.116:0/921912903 conn(0x5568709fe670 msgr2=0x5568709fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.799+0000 7f08d5a22740 1 -- 172.21.15.116:0/921912903 shutdown_connections 2024-03-19T01:22:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.800+0000 7f08d5a22740 1 -- 172.21.15.116:0/921912903 wait complete. 2024-03-19T01:22:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.800+0000 7f08d5a22740 1 -- start start 2024-03-19T01:22:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.800+0000 7f08d5a22740 1 Processor -- start 2024-03-19T01:22:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.801+0000 7f08d5a22740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a2c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.801+0000 7f08d5a22740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a04560 0x556870a2c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.801+0000 7f08d5a22740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a127c0 0x556870a2a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.801+0000 7f08d0bd9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a04560 0x556870a2c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.801+0000 7f08d0bd9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a04560 0x556870a2c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35258/0 (socket says 172.21.15.116:35258) 2024-03-19T01:22:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.801+0000 7f08cbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a2c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.801+0000 7f08cbfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a2c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:46404/0 (socket says 172.21.15.116:46404) 2024-03-19T01:22:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.801+0000 7f08d0bd9640 1 -- 172.21.15.116:0/3282641152 learned_addr learned my addr 172.21.15.116:0/3282641152 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:22:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.801+0000 7f08d5a22740 1 -- 172.21.15.116:0/3282641152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc1b0cf0 con 0x556870a04560 2024-03-19T01:22:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.801+0000 7f08d5a22740 1 -- 172.21.15.116:0/3282641152 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc1b0620 con 0x556870a127c0 2024-03-19T01:22:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.802+0000 7f08d5a22740 1 -- 172.21.15.116:0/3282641152 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc1ad860 con 0x556870a03af0 2024-03-19T01:22:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.802+0000 7f08d5a22740 10 client.0 _ll_register_callbacks cb 0x7f08c0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:22:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.802+0000 7f08cbfff640 1 --2- 172.21.15.116:0/3282641152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a127c0 0x556870a2a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.802+0000 7f08cbfff640 1 -- 172.21.15.116:0/3282641152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a127c0 msgr2=0x556870a2a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.802+0000 7f08cbfff640 1 --2- 172.21.15.116:0/3282641152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a127c0 0x556870a2a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.802+0000 7f08cbfff640 1 -- 172.21.15.116:0/3282641152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a04560 msgr2=0x556870a2c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:22:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.802+0000 7f08cbfff640 1 --2- 172.21.15.116:0/3282641152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a04560 0x556870a2c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.802+0000 7f08cbfff640 1 -- 172.21.15.116:0/3282641152 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556870a2b150 con 0x556870a03af0 2024-03-19T01:22:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.802+0000 7f08d0bd9640 1 --2- 172.21.15.116:0/3282641152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a04560 0x556870a2c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:22:53.805 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.802+0000 7f08cbfff640 1 --2- 172.21.15.116:0/3282641152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a2c310 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f08b8006010 tx=0x7f08c0004510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a9fbbae2f374b60 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:22:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.802+0000 7f08abfff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:22:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.803+0000 7f08abfff640 1 -- 172.21.15.116:0/3282641152 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b8011120 con 0x556870a03af0 2024-03-19T01:22:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.803+0000 7f08d5a22740 1 -- 172.21.15.116:0/3282641152 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08c0004cf0 con 0x556870a03af0 2024-03-19T01:22:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.803+0000 7f08abfff640 1 -- 172.21.15.116:0/3282641152 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f08b80112c0 con 0x556870a03af0 2024-03-19T01:22:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.803+0000 7f08abfff640 1 -- 172.21.15.116:0/3282641152 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f08b802e3c0 con 0x556870a03af0 2024-03-19T01:22:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.803+0000 7f08d5a22740 10 client.25918 fetch_fsmap learned FSMap version 45 2024-03-19T01:22:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.804+0000 7f08d5a22740 1 -- 172.21.15.116:0/3282641152 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x556870a7fce0 con 0x556870a03af0 2024-03-19T01:22:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.804+0000 7f08abfff640 1 -- 172.21.15.116:0/3282641152 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08b802e560 con 0x556870a03af0 2024-03-19T01:22:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.804+0000 7f08abfff640 1 -- 172.21.15.116:0/3282641152 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f08b803a470 con 0x556870a03af0 2024-03-19T01:22:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.804+0000 7f08d5a22740 10 client.25918 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:22:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.805+0000 7f08d5a22740 10 client.25918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:22:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.805+0000 7f08d5a22740 20 client.25918 populate_metadata read hostname 'smithi116' 2024-03-19T01:22:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.805+0000 7f08d5a22740 1 --2- 172.21.15.116:0/3282641152 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556870a805f0 0x556870a82a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:22:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.805+0000 7f08d5a22740 4 client.25918 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:22:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.805+0000 7f08d5a22740 1 -- 172.21.15.116:0/3282641152 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f08c0004cf0 con 0x556870a805f0 2024-03-19T01:22:53.808 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.805+0000 7f08d0bd9640 1 --2- 172.21.15.116:0/3282641152 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556870a805f0 0x556870a82a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:22:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.805+0000 7f08d0bd9640 1 --2- 172.21.15.116:0/3282641152 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556870a805f0 0x556870a82a10 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23c6d4619627f0ae server_cookie=c18ca98bed6ebbd8 in_seq=0 out_seq=0 2024-03-19T01:22:53.809 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.806+0000 7f08abfff640 10 client.25918 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.819+0000 7f08abfff640 1 -- 172.21.15.116:0/3282641152 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f08c0004cf0 con 0x556870a805f0 2024-03-19T01:22:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.819+0000 7f08abfff640 10 client.25918 handle_command_reply: tid=0 2024-03-19T01:22:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.820+0000 7f08d5a22740 1 client.25918 shutdown 2024-03-19T01:22:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.820+0000 7f08d5a22740 1 -- 172.21.15.116:0/3282641152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 msgr2=0x556870a2c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.820+0000 7f08d5a22740 1 --2- 172.21.15.116:0/3282641152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a2c310 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f08b8006010 tx=0x7f08c0004510 comp rx=0 tx=0).stop 2024-03-19T01:22:53.823 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.821+0000 7f08d5a22740 1 -- 172.21.15.116:0/3282641152 shutdown_connections 2024-03-19T01:22:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.821+0000 7f08d5a22740 1 --2- 172.21.15.116:0/3282641152 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x556870a127c0 0x556870a2a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.821+0000 7f08d5a22740 1 --2- 172.21.15.116:0/3282641152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x556870a04560 0x556870a2c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.821+0000 7f08d5a22740 1 --2- 172.21.15.116:0/3282641152 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x556870a805f0 0x556870a82a10 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.821+0000 7f08d5a22740 1 --2- 172.21.15.116:0/3282641152 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x556870a03af0 0x556870a2c310 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.821+0000 7f08abfff640 0 client.25918 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:22:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.821+0000 7f08d5a22740 1 -- 172.21.15.116:0/3282641152 >> 172.21.15.116:0/3282641152 conn(0x5568709fe670 msgr2=0x5568709fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.821+0000 7f08d5a22740 1 -- 172.21.15.116:0/3282641152 shutdown_connections 2024-03-19T01:22:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.821+0000 7f08d5a22740 1 -- 172.21.15.116:0/3282641152 wait complete. 2024-03-19T01:22:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.821+0000 7f08d5a22740 20 client.25918 trim_cache size 0 max 16384 2024-03-19T01:22:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:22:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:22:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:22:53.824 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:22:53.825 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.822+0000 7f08d2e3d640 1 -- 172.21.15.116:0/3490192960 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f08bc0a2920 msgr2=0x7f08bc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:53.825 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.822+0000 7f08d2e3d640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f08bc0a2920 0x7f08bc0a4de0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f08c40a39d0 tx=0x7f08c4017b90 comp rx=0 tx=0).stop 2024-03-19T01:22:53.825 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.822+0000 7f08d2e3d640 1 -- 172.21.15.116:0/3490192960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08cc1a38f0 msgr2=0x7f08cc1a5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:22:53.825 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.822+0000 7f08d2e3d640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08cc1a38f0 0x7f08cc1a5ce0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f08c400da10 tx=0x7f08c400ded0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.825 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.822+0000 7f08d2e3d640 1 -- 172.21.15.116:0/3490192960 shutdown_connections 2024-03-19T01:22:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.822+0000 7f08d2e3d640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f08cc1a38f0 0x7f08cc1a5ce0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.822+0000 7f08d2e3d640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f08cc103490 0x7f08cc19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.822+0000 7f08d2e3d640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f08bc0a2920 0x7f08bc0a4de0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.822+0000 7f08d2e3d640 1 --2- 172.21.15.116:0/3490192960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08cc102ad0 0x7f08cc19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:22:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.822+0000 7f08d2e3d640 1 -- 172.21.15.116:0/3490192960 >> 172.21.15.116:0/3490192960 conn(0x7f08cc0fb470 msgr2=0x7f08cc0fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:22:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.822+0000 7f08d2e3d640 1 -- 172.21.15.116:0/3490192960 shutdown_connections 2024-03-19T01:22:53.826 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:22:53.822+0000 7f08d2e3d640 1 -- 172.21.15.116:0/3490192960 wait complete. 2024-03-19T01:22:55.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:55 smithi162 ceph-mon[67178]: pgmap v660: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 601 op/s 2024-03-19T01:22:55.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:55 smithi116 ceph-mon[62701]: pgmap v660: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 601 op/s 2024-03-19T01:22:55.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:55 smithi156 ceph-mon[66799]: pgmap v660: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 MiB/s wr, 601 op/s 2024-03-19T01:22:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 610.0 s, 1716.8 tps, lat 59.231 ms stddev 70.120 2024-03-19T01:22:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:57 smithi162 ceph-mon[67178]: pgmap v661: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 MiB/s wr, 652 op/s 2024-03-19T01:22:57.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:57 smithi116 ceph-mon[62701]: pgmap v661: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 MiB/s wr, 652 op/s 2024-03-19T01:22:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:57 smithi156 ceph-mon[66799]: pgmap v661: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 MiB/s wr, 652 op/s 2024-03-19T01:22:59.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:22:59 smithi162 ceph-mon[67178]: pgmap v662: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 12 MiB/s wr, 608 op/s 2024-03-19T01:22:59.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:22:59 smithi116 ceph-mon[62701]: pgmap v662: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 12 MiB/s wr, 608 op/s 2024-03-19T01:22:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:22:59 smithi156 ceph-mon[66799]: pgmap v662: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 12 MiB/s wr, 608 op/s 2024-03-19T01:23:01.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:01 smithi162 ceph-mon[67178]: pgmap v663: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 711 op/s 2024-03-19T01:23:01.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:01 smithi116 ceph-mon[62701]: pgmap v663: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 711 op/s 2024-03-19T01:23:01.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:01 smithi156 ceph-mon[66799]: pgmap v663: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 711 op/s 2024-03-19T01:23:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 615.0 s, 1833.1 tps, lat 49.099 ms stddev 59.877 2024-03-19T01:23:03.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:03 smithi162 ceph-mon[67178]: pgmap v664: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 655 op/s 2024-03-19T01:23:03.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:03 smithi116 ceph-mon[62701]: pgmap v664: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 655 op/s 2024-03-19T01:23:03.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:03 smithi156 ceph-mon[66799]: pgmap v664: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 MiB/s wr, 655 op/s 2024-03-19T01:23:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:05 smithi162 ceph-mon[67178]: pgmap v665: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 19 MiB/s wr, 851 op/s 2024-03-19T01:23:05.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:05 smithi116 ceph-mon[62701]: pgmap v665: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 19 MiB/s wr, 851 op/s 2024-03-19T01:23:05.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:05 smithi156 ceph-mon[66799]: pgmap v665: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 19 MiB/s wr, 851 op/s 2024-03-19T01:23:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 620.0 s, 1475.8 tps, lat 74.925 ms stddev 126.842 2024-03-19T01:23:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.956+0000 7fc530d1b640 1 -- 172.21.15.116:0/1669526122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc52c100740 msgr2=0x7fc52c100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.956+0000 7fc530d1b640 1 --2- 172.21.15.116:0/1669526122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc52c100740 0x7fc52c100b60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc51c009900 tx=0x7fc51c02bc30 comp rx=0 tx=0).stop 2024-03-19T01:23:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.956+0000 7fc530d1b640 1 -- 172.21.15.116:0/1669526122 shutdown_connections 2024-03-19T01:23:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.956+0000 7fc530d1b640 1 --2- 172.21.15.116:0/1669526122 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc52c1022f0 0x7fc52c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.956+0000 7fc530d1b640 1 --2- 172.21.15.116:0/1669526122 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc52c101930 0x7fc52c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.957 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.956+0000 7fc530d1b640 1 --2- 172.21.15.116:0/1669526122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc52c100740 0x7fc52c100b60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.956+0000 7fc530d1b640 1 -- 172.21.15.116:0/1669526122 >> 172.21.15.116:0/1669526122 conn(0x7fc52c0fbcb0 msgr2=0x7fc52c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.956+0000 7fc530d1b640 1 -- 172.21.15.116:0/1669526122 shutdown_connections 2024-03-19T01:23:06.958 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.956+0000 7fc530d1b640 1 -- 172.21.15.116:0/1669526122 wait complete. 2024-03-19T01:23:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.957+0000 7fc530d1b640 1 Processor -- start 2024-03-19T01:23:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.957+0000 7fc530d1b640 1 -- start start 2024-03-19T01:23:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc530d1b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc52c100740 0x7fc52c1958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc530d1b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc52c101930 0x7fc52c195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc530d1b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc52c1022f0 0x7fc52c19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc52a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc52c100740 0x7fc52c1958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc530d1b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc52c10b310 con 0x7fc52c1022f0 2024-03-19T01:23:06.959 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc530d1b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc52c10b190 con 0x7fc52c101930 2024-03-19T01:23:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc52a59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc52c100740 0x7fc52c1958b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47216/0 (socket says 172.21.15.116:47216) 2024-03-19T01:23:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc530d1b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc52c10b490 con 0x7fc52c100740 2024-03-19T01:23:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc52a59c640 1 -- 172.21.15.116:0/3850872880 learned_addr learned my addr 172.21.15.116:0/3850872880 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc529d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc52c101930 0x7fc52c195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc52ad9d640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc52c1022f0 0x7fc52c19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc52a59c640 1 -- 172.21.15.116:0/3850872880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc52c101930 msgr2=0x7fc52c195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc52a59c640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc52c101930 0x7fc52c195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc52a59c640 1 -- 172.21.15.116:0/3850872880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc52c1022f0 msgr2=0x7fc52c19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc52a59c640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc52c1022f0 0x7fc52c19ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc52a59c640 1 -- 172.21.15.116:0/3850872880 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc52c06b370 con 0x7fc52c100740 2024-03-19T01:23:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.958+0000 7fc52ad9d640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc52c1022f0 0x7fc52c19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:23:06.960 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.959+0000 7fc52a59c640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc52c100740 0x7fc52c1958b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc51c02c240 tx=0x7fc51c0048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:06.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.960+0000 7fc51b7fe640 1 -- 172.21.15.116:0/3850872880 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c042020 con 0x7fc52c100740 2024-03-19T01:23:06.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.960+0000 7fc530d1b640 1 -- 172.21.15.116:0/3850872880 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc52c06b600 con 0x7fc52c100740 2024-03-19T01:23:06.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.960+0000 7fc530d1b640 1 -- 172.21.15.116:0/3850872880 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc52c06bba0 con 0x7fc52c100740 2024-03-19T01:23:06.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.960+0000 7fc51b7fe640 1 -- 172.21.15.116:0/3850872880 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc51c02cd30 con 0x7fc52c100740 2024-03-19T01:23:06.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.960+0000 7fc51b7fe640 1 -- 172.21.15.116:0/3850872880 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c003c60 con 0x7fc52c100740 2024-03-19T01:23:06.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.962+0000 7fc51b7fe640 1 -- 172.21.15.116:0/3850872880 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fc51c045020 con 0x7fc52c100740 2024-03-19T01:23:06.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.963+0000 7fc51b7fe640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc5140a4b80 0x7fc5140a7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:06.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.963+0000 7fc51b7fe640 1 -- 172.21.15.116:0/3850872880 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc51c1126e0 con 0x7fc52c100740 2024-03-19T01:23:06.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.963+0000 7fc529d9b640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc5140a4b80 0x7fc5140a7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.963+0000 7fc5339e8740 1 Processor -- start 2024-03-19T01:23:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.963+0000 7fc5339e8740 1 -- start start 2024-03-19T01:23:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.964+0000 7fc5339e8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35a100 0x55fdce35a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.964+0000 7fc5339e8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35aaf0 0x55fdce35f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.964+0000 7fc5339e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35fc60 0x55fdce3620f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.964+0000 7fc5339e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fdce2bebe0 con 0x55fdce35fc60 2024-03-19T01:23:06.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.964+0000 7fc5339e8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fdce267960 con 0x55fdce35a100 2024-03-19T01:23:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.964+0000 7fc5339e8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fdce2bac80 con 0x55fdce35aaf0 2024-03-19T01:23:06.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.964+0000 7fc52ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35fc60 0x55fdce3620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.964+0000 7fc529d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35aaf0 0x55fdce35f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.964+0000 7fc52ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35fc60 0x55fdce3620f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33674/0 (socket says 172.21.15.116:33674) 2024-03-19T01:23:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.964+0000 7fc52ad9d640 1 -- 172.21.15.116:0/728488732 learned_addr learned my addr 172.21.15.116:0/728488732 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.965+0000 7fc529d9b640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc5140a4b80 0x7fc5140a7040 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc5200095f0 tx=0x7fc520005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.965+0000 7fc529d9b640 1 -- 172.21.15.116:0/728488732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35a100 msgr2=0x55fdce35a520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:23:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.965+0000 7fc529d9b640 1 --2- 172.21.15.116:0/728488732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35a100 0x55fdce35a520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.965+0000 7fc529d9b640 1 -- 172.21.15.116:0/728488732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35fc60 msgr2=0x55fdce3620f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:06.966 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.965+0000 7fc529d9b640 1 --2- 172.21.15.116:0/728488732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35fc60 0x55fdce3620f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.965+0000 7fc529d9b640 1 -- 172.21.15.116:0/728488732 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdce25ed20 con 0x55fdce35aaf0 2024-03-19T01:23:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.965+0000 7fc529d9b640 1 --2- 172.21.15.116:0/728488732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35aaf0 0x55fdce35f5f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc52001e920 tx=0x7fc520018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc96ee023f6fe98 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.965+0000 7fc518ff9640 1 -- 172.21.15.116:0/728488732 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc520004030 con 0x55fdce35aaf0 2024-03-19T01:23:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.965+0000 7fc518ff9640 1 -- 172.21.15.116:0/728488732 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc52000ed90 con 0x55fdce35aaf0 2024-03-19T01:23:06.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.965+0000 7fc518ff9640 1 -- 172.21.15.116:0/728488732 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc520003c60 con 0x55fdce35aaf0 2024-03-19T01:23:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.966+0000 7fc5339e8740 1 -- 172.21.15.116:0/728488732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35aaf0 msgr2=0x55fdce35f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.966+0000 7fc5339e8740 1 --2- 172.21.15.116:0/728488732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35aaf0 0x55fdce35f5f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc52001e920 tx=0x7fc520018000 comp rx=0 tx=0).stop 2024-03-19T01:23:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.966+0000 7fc5339e8740 1 -- 172.21.15.116:0/728488732 shutdown_connections 2024-03-19T01:23:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.966+0000 7fc5339e8740 1 --2- 172.21.15.116:0/728488732 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35fc60 0x55fdce3620f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.966+0000 7fc5339e8740 1 --2- 172.21.15.116:0/728488732 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35aaf0 0x55fdce35f5f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.966+0000 7fc5339e8740 1 --2- 172.21.15.116:0/728488732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35a100 0x55fdce35a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.966+0000 7fc5339e8740 1 -- 172.21.15.116:0/728488732 >> 172.21.15.116:0/728488732 conn(0x55fdce355670 msgr2=0x55fdce355a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.966+0000 7fc5339e8740 1 -- 172.21.15.116:0/728488732 shutdown_connections 2024-03-19T01:23:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.966+0000 7fc5339e8740 1 -- 172.21.15.116:0/728488732 wait complete. 2024-03-19T01:23:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.967+0000 7fc5339e8740 1 -- start start 2024-03-19T01:23:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.967+0000 7fc5339e8740 1 Processor -- start 2024-03-19T01:23:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.968+0000 7fc5339e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35a100 0x55fdce36ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.968+0000 7fc5339e8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35aaf0 0x55fdce36cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.968+0000 7fc5339e8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35fc60 0x55fdce373f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.968+0000 7fc52a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35a100 0x55fdce36ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.968+0000 7fc529d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35aaf0 0x55fdce36cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.968+0000 7fc52a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35a100 0x55fdce36ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33684/0 (socket says 172.21.15.116:33684) 2024-03-19T01:23:06.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.968+0000 7fc52a59c640 1 -- 172.21.15.116:0/1126740178 learned_addr learned my addr 172.21.15.116:0/1126740178 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.968+0000 7fc52ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35fc60 0x55fdce373f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.968+0000 7fc5339e8740 1 -- 172.21.15.116:0/1126740178 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fdce362c60 con 0x55fdce35a100 2024-03-19T01:23:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.968+0000 7fc5339e8740 1 -- 172.21.15.116:0/1126740178 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fdce27adb0 con 0x55fdce35aaf0 2024-03-19T01:23:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.968+0000 7fc5339e8740 1 -- 172.21.15.116:0/1126740178 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fdce362de0 con 0x55fdce35fc60 2024-03-19T01:23:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.969+0000 7fc529d9b640 1 -- 172.21.15.116:0/1126740178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35fc60 msgr2=0x55fdce373f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.969+0000 7fc5339e8740 10 client.0 _ll_register_callbacks cb 0x55fdce27fe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:23:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.969+0000 7fc529d9b640 1 --2- 172.21.15.116:0/1126740178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35fc60 0x55fdce373f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.969+0000 7fc529d9b640 1 -- 172.21.15.116:0/1126740178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35a100 msgr2=0x55fdce36ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.969+0000 7fc529d9b640 1 --2- 172.21.15.116:0/1126740178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35a100 0x55fdce36ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.969+0000 7fc529d9b640 1 -- 172.21.15.116:0/1126740178 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdce279c60 con 0x55fdce35aaf0 2024-03-19T01:23:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.969+0000 7fc52ad9d640 1 --2- 172.21.15.116:0/1126740178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35fc60 0x55fdce373f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.969+0000 7fc529d9b640 1 --2- 172.21.15.116:0/1126740178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35aaf0 0x55fdce36cf60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc52001e8f0 tx=0x7fc520005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57655b36a0faa17f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:06.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.969+0000 7fc511ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:23:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.969+0000 7fc52a59c640 1 --2- 172.21.15.116:0/1126740178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35a100 0x55fdce36ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.969+0000 7fc5339e8740 1 -- 172.21.15.116:0/1126740178 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc52000ed90 con 0x55fdce35aaf0 2024-03-19T01:23:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.969+0000 7fc511ffb640 1 -- 172.21.15.116:0/1126740178 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52001e170 con 0x55fdce35aaf0 2024-03-19T01:23:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.970+0000 7fc511ffb640 1 -- 172.21.15.116:0/1126740178 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc52001e310 con 0x55fdce35aaf0 2024-03-19T01:23:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.970+0000 7fc511ffb640 1 -- 172.21.15.116:0/1126740178 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc52003a050 con 0x55fdce35aaf0 2024-03-19T01:23:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.970+0000 7fc5339e8740 10 client.25940 fetch_fsmap learned FSMap version 45 2024-03-19T01:23:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.970+0000 7fc5339e8740 1 -- 172.21.15.116:0/1126740178 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({fsmap=45}) -- 0x55fdcdfd3210 con 0x55fdce35aaf0 2024-03-19T01:23:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.970+0000 7fc511ffb640 1 -- 172.21.15.116:0/1126740178 <== mon.2 v2:172.21.15.156:3300/0 4 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fc52002a430 con 0x55fdce35aaf0 2024-03-19T01:23:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.970+0000 7fc5339e8740 10 client.25940 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:23:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.970+0000 7fc5339e8740 10 client.25940 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:23:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.970+0000 7fc5339e8740 20 client.25940 populate_metadata read hostname 'smithi116' 2024-03-19T01:23:06.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.970+0000 7fc5339e8740 1 --2- 172.21.15.116:0/1126740178 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fdce367d20 0x55fdce36a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.970+0000 7fc5339e8740 4 client.25940 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:23:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.971+0000 7fc5339e8740 1 -- 172.21.15.116:0/1126740178 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fdce2cc120 con 0x55fdce367d20 2024-03-19T01:23:06.974 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.971+0000 7fc52a59c640 1 --2- 172.21.15.116:0/1126740178 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fdce367d20 0x55fdce36a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:06.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.971+0000 7fc52a59c640 1 --2- 172.21.15.116:0/1126740178 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fdce367d20 0x55fdce36a1a0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e84a1eaa9321dd31 server_cookie=546ace488e7017a5 in_seq=0 out_seq=0 2024-03-19T01:23:06.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.971+0000 7fc511ffb640 10 client.25940 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:06.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.972+0000 7fc511ffb640 1 -- 172.21.15.116:0/1126740178 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fdce2cc120 con 0x55fdce367d20 2024-03-19T01:23:06.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.972+0000 7fc511ffb640 10 client.25940 handle_command_reply: tid=0 2024-03-19T01:23:06.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.972+0000 7fc5339e8740 1 client.25940 shutdown 2024-03-19T01:23:06.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.973+0000 7fc5339e8740 1 -- 172.21.15.116:0/1126740178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35aaf0 msgr2=0x55fdce36cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:06.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.973+0000 7fc5339e8740 1 --2- 172.21.15.116:0/1126740178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35aaf0 0x55fdce36cf60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc52001e8f0 tx=0x7fc520005dc0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.973+0000 7fc5339e8740 1 -- 172.21.15.116:0/1126740178 shutdown_connections 2024-03-19T01:23:06.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.973+0000 7fc5339e8740 1 --2- 172.21.15.116:0/1126740178 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fdce367d20 0x55fdce36a1a0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.973+0000 7fc5339e8740 1 --2- 172.21.15.116:0/1126740178 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35fc60 0x55fdce373f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.973+0000 7fc5339e8740 1 --2- 172.21.15.116:0/1126740178 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35aaf0 0x55fdce36cf60 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.973+0000 7fc5339e8740 1 --2- 172.21.15.116:0/1126740178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35a100 0x55fdce36ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:06.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.973+0000 7fc5339e8740 1 -- 172.21.15.116:0/1126740178 >> 172.21.15.116:0/1126740178 conn(0x55fdce355670 msgr2=0x55fdce3568c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:06.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.973+0000 7fc511ffb640 0 client.25940 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:06.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.973+0000 7fc5339e8740 1 -- 172.21.15.116:0/1126740178 shutdown_connections 2024-03-19T01:23:06.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.973+0000 7fc5339e8740 1 -- 172.21.15.116:0/1126740178 wait complete. 2024-03-19T01:23:06.976 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.973+0000 7fc5339e8740 20 client.25940 trim_cache size 0 max 16384 2024-03-19T01:23:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.998+0000 7fc5339e8740 1 Processor -- start 2024-03-19T01:23:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.998+0000 7fc5339e8740 1 -- start start 2024-03-19T01:23:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc5339e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35a100 0x55fdce3643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc5339e8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35aaf0 0x55fdce354220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc5339e8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35fc60 0x55fdce354760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc5339e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c00e4c0 con 0x55fdce35a100 2024-03-19T01:23:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc5339e8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c010000 con 0x55fdce35fc60 2024-03-19T01:23:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc5339e8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c00eb90 con 0x55fdce35aaf0 2024-03-19T01:23:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc52a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35a100 0x55fdce3643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:07.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc529d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35aaf0 0x55fdce354220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc52ad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35fc60 0x55fdce354760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc52ad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35fc60 0x55fdce354760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:42766/0 (socket says 172.21.15.116:42766) 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc529d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35aaf0 0x55fdce354220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47234/0 (socket says 172.21.15.116:47234) 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc52ad9d640 1 -- 172.21.15.116:0/3638153321 learned_addr learned my addr 172.21.15.116:0/3638153321 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc52a59c640 1 -- 172.21.15.116:0/3638153321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35aaf0 msgr2=0x55fdce354220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc52a59c640 1 --2- 172.21.15.116:0/3638153321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35aaf0 0x55fdce354220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc52a59c640 1 -- 172.21.15.116:0/3638153321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35fc60 msgr2=0x55fdce354760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc52a59c640 1 --2- 172.21.15.116:0/3638153321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35fc60 0x55fdce354760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc52a59c640 1 -- 172.21.15.116:0/3638153321 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fdce09f620 con 0x55fdce35a100 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc52ad9d640 1 --2- 172.21.15.116:0/3638153321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35fc60 0x55fdce354760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc52a59c640 1 --2- 172.21.15.116:0/3638153321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35a100 0x55fdce3643b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc51c006010 tx=0x7fc51c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c015f7f4c6966d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:06.999+0000 7fc513fff640 1 -- 172.21.15.116:0/3638153321 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c0df880 con 0x55fdce35a100 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.000+0000 7fc513fff640 1 -- 172.21.15.116:0/3638153321 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc51c0df180 con 0x55fdce35a100 2024-03-19T01:23:07.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.000+0000 7fc513fff640 1 -- 172.21.15.116:0/3638153321 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c004ba0 con 0x55fdce35a100 2024-03-19T01:23:07.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.000+0000 7fc5339e8740 1 -- 172.21.15.116:0/3638153321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35a100 msgr2=0x55fdce3643b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.000+0000 7fc5339e8740 1 --2- 172.21.15.116:0/3638153321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35a100 0x55fdce3643b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc51c006010 tx=0x7fc51c000bc0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.001+0000 7fc5339e8740 1 -- 172.21.15.116:0/3638153321 shutdown_connections 2024-03-19T01:23:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.001+0000 7fc5339e8740 1 --2- 172.21.15.116:0/3638153321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35fc60 0x55fdce354760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.001+0000 7fc5339e8740 1 --2- 172.21.15.116:0/3638153321 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35aaf0 0x55fdce354220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.001+0000 7fc5339e8740 1 --2- 172.21.15.116:0/3638153321 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35a100 0x55fdce3643b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.001+0000 7fc5339e8740 1 -- 172.21.15.116:0/3638153321 >> 172.21.15.116:0/3638153321 conn(0x55fdce355670 msgr2=0x55fdce369d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.002+0000 7fc5339e8740 1 -- 172.21.15.116:0/3638153321 shutdown_connections 2024-03-19T01:23:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.002+0000 7fc5339e8740 1 -- 172.21.15.116:0/3638153321 wait complete. 2024-03-19T01:23:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.002+0000 7fc5339e8740 1 -- start start 2024-03-19T01:23:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.002+0000 7fc5339e8740 1 Processor -- start 2024-03-19T01:23:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.002+0000 7fc5339e8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35a100 0x55fdce3729c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc5339e8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35aaf0 0x55fdce372f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:07.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc5339e8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35fc60 0x55fdce3d40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc5339e8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c00f930 con 0x55fdce35aaf0 2024-03-19T01:23:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc5339e8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c00f260 con 0x55fdce35a100 2024-03-19T01:23:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc5339e8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fc50c00c4a0 con 0x55fdce35fc60 2024-03-19T01:23:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc529d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35aaf0 0x55fdce372f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc52a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35a100 0x55fdce3729c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc52ad9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35fc60 0x55fdce3d40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc52a59c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35a100 0x55fdce3729c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:42770/0 (socket says 172.21.15.116:42770) 2024-03-19T01:23:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc52a59c640 1 -- 172.21.15.116:0/3970570561 learned_addr learned my addr 172.21.15.116:0/3970570561 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc529d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35aaf0 0x55fdce372f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33710/0 (socket says 172.21.15.116:33710) 2024-03-19T01:23:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc5339e8740 10 client.0 _ll_register_callbacks cb 0x7fc524009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:23:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc52a59c640 1 -- 172.21.15.116:0/3970570561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35fc60 msgr2=0x55fdce3d40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc52a59c640 1 --2- 172.21.15.116:0/3970570561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35fc60 0x55fdce3d40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.007 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc52a59c640 1 -- 172.21.15.116:0/3970570561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35aaf0 msgr2=0x55fdce372f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc52a59c640 1 --2- 172.21.15.116:0/3970570561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35aaf0 0x55fdce372f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc52a59c640 1 -- 172.21.15.116:0/3970570561 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fdce3d46f0 con 0x55fdce35a100 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc5339e8740 1 -- 172.21.15.116:0/3970570561 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc51c0df180 con 0x55fdce35a100 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc52a59c640 1 --2- 172.21.15.116:0/3970570561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35a100 0x55fdce3729c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0e73f0 tx=0x7fc51c004450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33f785e72f12a260 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc511ffb640 10 client.25946 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.003+0000 7fc511ffb640 1 -- 172.21.15.116:0/3970570561 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c0df880 con 0x55fdce35a100 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.004+0000 7fc511ffb640 1 -- 172.21.15.116:0/3970570561 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fc51c0db7f0 con 0x55fdce35a100 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.004+0000 7fc511ffb640 1 -- 172.21.15.116:0/3970570561 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc51c0e7920 con 0x55fdce35a100 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.004+0000 7fc5339e8740 10 client.25946 fetch_fsmap learned FSMap version 45 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.004+0000 7fc5339e8740 1 -- 172.21.15.116:0/3970570561 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({fsmap=45}) -- 0x55fdce3d5940 con 0x55fdce35a100 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.004+0000 7fc511ffb640 1 -- 172.21.15.116:0/3970570561 <== mon.2 v2:172.21.15.156:3300/0 4 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fc51c06f020 con 0x55fdce35a100 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.004+0000 7fc5339e8740 10 client.25946 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.004+0000 7fc5339e8740 10 client.25946 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.004+0000 7fc5339e8740 20 client.25946 populate_metadata read hostname 'smithi116' 2024-03-19T01:23:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.004+0000 7fc5339e8740 1 --2- 172.21.15.116:0/3970570561 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fdce3d6160 0x55fdce3d8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:07.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.005+0000 7fc5339e8740 4 client.25946 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:23:07.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.005+0000 7fc5339e8740 1 -- 172.21.15.116:0/3970570561 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fdce2c6470 con 0x55fdce3d6160 2024-03-19T01:23:07.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.005+0000 7fc529d9b640 1 --2- 172.21.15.116:0/3970570561 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fdce3d6160 0x55fdce3d8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:07.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.005+0000 7fc529d9b640 1 --2- 172.21.15.116:0/3970570561 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fdce3d6160 0x55fdce3d8580 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b36f00efabb7c17 server_cookie=782da88d1a74a346 in_seq=0 out_seq=0 2024-03-19T01:23:07.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.005+0000 7fc511ffb640 10 client.25946 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:07.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.013+0000 7fc511ffb640 1 -- 172.21.15.116:0/3970570561 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fdce2c6470 con 0x55fdce3d6160 2024-03-19T01:23:07.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.013+0000 7fc511ffb640 10 client.25946 handle_command_reply: tid=0 2024-03-19T01:23:07.013 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.013+0000 7fc5339e8740 1 client.25946 shutdown 2024-03-19T01:23:07.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.014+0000 7fc5339e8740 1 -- 172.21.15.116:0/3970570561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35a100 msgr2=0x55fdce3729c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:07.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.014+0000 7fc5339e8740 1 --2- 172.21.15.116:0/3970570561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35a100 0x55fdce3729c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0e73f0 tx=0x7fc51c004450 comp rx=0 tx=0).stop 2024-03-19T01:23:07.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.014+0000 7fc5339e8740 1 -- 172.21.15.116:0/3970570561 shutdown_connections 2024-03-19T01:23:07.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.014+0000 7fc5339e8740 1 --2- 172.21.15.116:0/3970570561 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fdce3d6160 0x55fdce3d8580 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.014+0000 7fc5339e8740 1 --2- 172.21.15.116:0/3970570561 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fdce35fc60 0x55fdce3d40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.014+0000 7fc5339e8740 1 --2- 172.21.15.116:0/3970570561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fdce35aaf0 0x55fdce372f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.014 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.014+0000 7fc5339e8740 1 --2- 172.21.15.116:0/3970570561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fdce35a100 0x55fdce3729c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.014+0000 7fc5339e8740 1 -- 172.21.15.116:0/3970570561 >> 172.21.15.116:0/3970570561 conn(0x55fdce355670 msgr2=0x55fdce369d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:07.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.014+0000 7fc511ffb640 0 client.25946 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:07.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.014+0000 7fc5339e8740 1 -- 172.21.15.116:0/3970570561 shutdown_connections 2024-03-19T01:23:07.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.014+0000 7fc5339e8740 1 -- 172.21.15.116:0/3970570561 wait complete. 2024-03-19T01:23:07.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.014+0000 7fc5339e8740 20 client.25946 trim_cache size 0 max 16384 2024-03-19T01:23:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:23:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:23:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:23:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:23:07.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.015+0000 7fc530d1b640 1 -- 172.21.15.116:0/3850872880 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc5140a4b80 msgr2=0x7fc5140a7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.015+0000 7fc530d1b640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc5140a4b80 0x7fc5140a7040 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc5200095f0 tx=0x7fc520005f90 comp rx=0 tx=0).stop 2024-03-19T01:23:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.015+0000 7fc530d1b640 1 -- 172.21.15.116:0/3850872880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc52c100740 msgr2=0x7fc52c1958b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.015+0000 7fc530d1b640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc52c100740 0x7fc52c1958b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc51c02c240 tx=0x7fc51c0048f0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.015+0000 7fc530d1b640 1 -- 172.21.15.116:0/3850872880 shutdown_connections 2024-03-19T01:23:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.015+0000 7fc530d1b640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc52c1022f0 0x7fc52c19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.015+0000 7fc530d1b640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fc5140a4b80 0x7fc5140a7040 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.015+0000 7fc530d1b640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc52c101930 0x7fc52c195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.015+0000 7fc530d1b640 1 --2- 172.21.15.116:0/3850872880 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fc52c100740 0x7fc52c1958b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.015+0000 7fc530d1b640 1 -- 172.21.15.116:0/3850872880 >> 172.21.15.116:0/3850872880 conn(0x7fc52c0fbcb0 msgr2=0x7fc52c104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.016+0000 7fc530d1b640 1 -- 172.21.15.116:0/3850872880 shutdown_connections 2024-03-19T01:23:07.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:07.016+0000 7fc530d1b640 1 -- 172.21.15.116:0/3850872880 wait complete. 2024-03-19T01:23:07.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:07 smithi162 ceph-mon[67178]: pgmap v666: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 38 MiB/s wr, 1.50k op/s 2024-03-19T01:23:07.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:07 smithi116 ceph-mon[62701]: pgmap v666: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 38 MiB/s wr, 1.50k op/s 2024-03-19T01:23:07.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:07 smithi156 ceph-mon[66799]: pgmap v666: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 38 MiB/s wr, 1.50k op/s 2024-03-19T01:23:08.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:08 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:23:08.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:08 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:23:08.465 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:08 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:23:09.445 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:09 smithi156 ceph-mon[66799]: pgmap v667: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 MiB/s wr, 1.46k op/s 2024-03-19T01:23:09.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:09 smithi162 ceph-mon[67178]: pgmap v667: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 MiB/s wr, 1.46k op/s 2024-03-19T01:23:09.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:09 smithi116 ceph-mon[62701]: pgmap v667: 97 pgs: 97 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 MiB/s wr, 1.46k op/s 2024-03-19T01:23:11.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:10 smithi162 ceph-mon[67178]: pgmap v668: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 40 MiB/s wr, 1.55k op/s 2024-03-19T01:23:11.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:10 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:10 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:10 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:10 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:10 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:10 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:10 smithi116 ceph-mon[62701]: pgmap v668: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 40 MiB/s wr, 1.55k op/s 2024-03-19T01:23:11.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:10 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:10 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:10 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:10 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:10 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:10 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:10 smithi156 ceph-mon[66799]: pgmap v668: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 40 MiB/s wr, 1.55k op/s 2024-03-19T01:23:11.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:10 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:10 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:10 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:10 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:10 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:10 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 625.0 s, 1459.8 tps, lat 68.181 ms stddev 76.016 2024-03-19T01:23:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:23:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:23:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:11 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:12.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:23:12.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:23:12.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:11 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:12.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:23:12.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:23:12.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:11 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:23:13.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:12 smithi162 ceph-mon[67178]: pgmap v669: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 39 MiB/s wr, 1.46k op/s 2024-03-19T01:23:13.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:12 smithi116 ceph-mon[62701]: pgmap v669: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 39 MiB/s wr, 1.46k op/s 2024-03-19T01:23:13.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:12 smithi156 ceph-mon[66799]: pgmap v669: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 39 MiB/s wr, 1.46k op/s 2024-03-19T01:23:15.276 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:14 smithi156 ceph-mon[66799]: pgmap v670: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 37 MiB/s wr, 1.42k op/s 2024-03-19T01:23:15.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:14 smithi162 ceph-mon[67178]: pgmap v670: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 37 MiB/s wr, 1.42k op/s 2024-03-19T01:23:15.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:14 smithi116 ceph-mon[62701]: pgmap v670: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 37 MiB/s wr, 1.42k op/s 2024-03-19T01:23:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 630.0 s, 1530.6 tps, lat 64.769 ms stddev 75.293 2024-03-19T01:23:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:16 smithi162 ceph-mon[67178]: pgmap v671: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 31 MiB/s wr, 1.27k op/s 2024-03-19T01:23:17.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:16 smithi116 ceph-mon[62701]: pgmap v671: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 31 MiB/s wr, 1.27k op/s 2024-03-19T01:23:17.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:16 smithi156 ceph-mon[66799]: pgmap v671: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 31 MiB/s wr, 1.27k op/s 2024-03-19T01:23:19.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:18 smithi162 ceph-mon[67178]: pgmap v672: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 621 op/s 2024-03-19T01:23:19.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:18 smithi116 ceph-mon[62701]: pgmap v672: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 621 op/s 2024-03-19T01:23:19.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:18 smithi156 ceph-mon[66799]: pgmap v672: 97 pgs: 97 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 MiB/s wr, 621 op/s 2024-03-19T01:23:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.155+0000 7f2d74c54640 1 -- 172.21.15.116:0/126664479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d70108820 msgr2=0x7f2d7010ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.156 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.155+0000 7f2d74c54640 1 --2- 172.21.15.116:0/126664479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d70108820 0x7f2d7010ac50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2d60009be0 tx=0x7f2d6002bf10 comp rx=0 tx=0).stop 2024-03-19T01:23:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.161+0000 7f2d74c54640 1 -- 172.21.15.116:0/126664479 shutdown_connections 2024-03-19T01:23:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.161+0000 7f2d74c54640 1 --2- 172.21.15.116:0/126664479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d7010db40 0x7f2d7010dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.161+0000 7f2d74c54640 1 --2- 172.21.15.116:0/126664479 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d7010b190 0x7f2d7010d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.161+0000 7f2d74c54640 1 --2- 172.21.15.116:0/126664479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d70108820 0x7f2d7010ac50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.161+0000 7f2d74c54640 1 -- 172.21.15.116:0/126664479 >> 172.21.15.116:0/126664479 conn(0x7f2d7006e860 msgr2=0x7f2d70070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:20.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.165+0000 7f2d74c54640 1 -- 172.21.15.116:0/126664479 shutdown_connections 2024-03-19T01:23:20.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.168+0000 7f2d74c54640 1 -- 172.21.15.116:0/126664479 wait complete. 2024-03-19T01:23:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.169+0000 7f2d74c54640 1 Processor -- start 2024-03-19T01:23:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.169+0000 7f2d74c54640 1 -- start start 2024-03-19T01:23:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.170+0000 7f2d74c54640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d70108820 0x7f2d70119990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.170+0000 7f2d74c54640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d7010b190 0x7f2d70119ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.170+0000 7f2d74c54640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d7010db40 0x7f2d70117fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.170+0000 7f2d74c54640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2d70110960 con 0x7f2d70108820 2024-03-19T01:23:20.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.170+0000 7f2d74c54640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2d701107e0 con 0x7f2d7010db40 2024-03-19T01:23:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.170+0000 7f2d74c54640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2d70110ae0 con 0x7f2d7010b190 2024-03-19T01:23:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.170+0000 7f2d6dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d7010b190 0x7f2d70119ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.170+0000 7f2d6dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d7010b190 0x7f2d70119ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55344/0 (socket says 172.21.15.116:55344) 2024-03-19T01:23:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.170+0000 7f2d6dd9b640 1 -- 172.21.15.116:0/1904789039 learned_addr learned my addr 172.21.15.116:0/1904789039 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.171+0000 7f2d6ed9d640 1 --2- 172.21.15.116:0/1904789039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d7010db40 0x7f2d70117fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.171+0000 7f2d6dd9b640 1 -- 172.21.15.116:0/1904789039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d7010db40 msgr2=0x7f2d70117fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.171+0000 7f2d6dd9b640 1 --2- 172.21.15.116:0/1904789039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d7010db40 0x7f2d70117fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.171+0000 7f2d6dd9b640 1 -- 172.21.15.116:0/1904789039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d70108820 msgr2=0x7f2d70119990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-19T01:23:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.171+0000 7f2d6dd9b640 1 --2- 172.21.15.116:0/1904789039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d70108820 0x7f2d70119990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.171+0000 7f2d6dd9b640 1 -- 172.21.15.116:0/1904789039 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d70118670 con 0x7f2d7010b190 2024-03-19T01:23:20.175 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.171+0000 7f2d6dd9b640 1 --2- 172.21.15.116:0/1904789039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d7010b190 0x7f2d70119ed0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800bea0 tx=0x7f2d5800bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.173+0000 7f2d677fe640 1 -- 172.21.15.116:0/1904789039 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d58039070 con 0x7f2d7010b190 2024-03-19T01:23:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.173+0000 7f2d74c54640 1 -- 172.21.15.116:0/1904789039 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d70118900 con 0x7f2d7010b190 2024-03-19T01:23:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.173+0000 7f2d74c54640 1 -- 172.21.15.116:0/1904789039 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d70073a50 con 0x7f2d7010b190 2024-03-19T01:23:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.173+0000 7f2d677fe640 1 -- 172.21.15.116:0/1904789039 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d58009c30 con 0x7f2d7010b190 2024-03-19T01:23:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.174+0000 7f2d677fe640 1 -- 172.21.15.116:0/1904789039 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d580316c0 con 0x7f2d7010b190 2024-03-19T01:23:20.178 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.178+0000 7f2d7783d740 1 Processor -- start 2024-03-19T01:23:20.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.178+0000 7f2d7783d740 1 -- start start 2024-03-19T01:23:20.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.178+0000 7f2d7783d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f654100 0x56158f654520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.178+0000 7f2d7783d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 0x56158f6595f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.178+0000 7f2d7783d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f659c60 0x56158f65c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.178+0000 7f2d7783d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56158f5b8be0 con 0x56158f659c60 2024-03-19T01:23:20.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.178+0000 7f2d7783d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56158f561960 con 0x56158f654100 2024-03-19T01:23:20.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.178+0000 7f2d7783d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56158f5b4c80 con 0x56158f654af0 2024-03-19T01:23:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.179+0000 7f2d6ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f654100 0x56158f654520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.179+0000 7f2d6ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f654100 0x56158f654520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:60516/0 (socket says 172.21.15.116:60516) 2024-03-19T01:23:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.179+0000 7f2d6ed9d640 1 -- 172.21.15.116:0/2330015963 learned_addr learned my addr 172.21.15.116:0/2330015963 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.179+0000 7f2d6e59c640 1 --2- 172.21.15.116:0/2330015963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 0x56158f6595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.179+0000 7f2d6ed9d640 1 -- 172.21.15.116:0/2330015963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 msgr2=0x56158f6595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:20.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.179+0000 7f2d6ed9d640 1 --2- 172.21.15.116:0/2330015963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 0x56158f6595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.179+0000 7f2d6ed9d640 1 -- 172.21.15.116:0/2330015963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f659c60 msgr2=0x56158f65c0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:23:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.179+0000 7f2d6ed9d640 1 --2- 172.21.15.116:0/2330015963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f659c60 0x56158f65c0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.179+0000 7f2d6ed9d640 1 -- 172.21.15.116:0/2330015963 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56158f558d20 con 0x56158f654100 2024-03-19T01:23:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.179+0000 7f2d6ed9d640 1 --2- 172.21.15.116:0/2330015963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f654100 0x56158f654520 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009bb0 tx=0x7f2d680117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea3a51fc0cd3693d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.180+0000 7f2d677fe640 1 -- 172.21.15.116:0/1904789039 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f2d58031860 con 0x7f2d7010b190 2024-03-19T01:23:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.181+0000 7f2d64ff9640 1 -- 172.21.15.116:0/2330015963 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68012450 con 0x56158f654100 2024-03-19T01:23:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.181+0000 7f2d64ff9640 1 -- 172.21.15.116:0/2330015963 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d68012ad0 con 0x56158f654100 2024-03-19T01:23:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.181+0000 7f2d677fe640 1 --2- 172.21.15.116:0/1904789039 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2d5c0a2ae0 0x7f2d5c0a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.181+0000 7f2d6e59c640 1 --2- 172.21.15.116:0/1904789039 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2d5c0a2ae0 0x7f2d5c0a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.181+0000 7f2d677fe640 1 -- 172.21.15.116:0/1904789039 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2d580f05d0 con 0x7f2d7010b190 2024-03-19T01:23:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.182+0000 7f2d7783d740 1 -- 172.21.15.116:0/2330015963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f654100 msgr2=0x56158f654520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.182+0000 7f2d7783d740 1 --2- 172.21.15.116:0/2330015963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f654100 0x56158f654520 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009bb0 tx=0x7f2d680117b0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.184 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.182+0000 7f2d6e59c640 1 --2- 172.21.15.116:0/1904789039 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2d5c0a2ae0 0x7f2d5c0a4fa0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2d60036040 tx=0x7f2d60005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.183+0000 7f2d7783d740 1 -- 172.21.15.116:0/2330015963 shutdown_connections 2024-03-19T01:23:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.183+0000 7f2d7783d740 1 --2- 172.21.15.116:0/2330015963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f659c60 0x56158f65c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.183+0000 7f2d7783d740 1 --2- 172.21.15.116:0/2330015963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 0x56158f6595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.183+0000 7f2d7783d740 1 --2- 172.21.15.116:0/2330015963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f654100 0x56158f654520 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.183+0000 7f2d7783d740 1 -- 172.21.15.116:0/2330015963 >> 172.21.15.116:0/2330015963 conn(0x56158f64f670 msgr2=0x56158f64fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.188+0000 7f2d7783d740 1 -- 172.21.15.116:0/2330015963 shutdown_connections 2024-03-19T01:23:20.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.190+0000 7f2d7783d740 1 -- 172.21.15.116:0/2330015963 wait complete. 2024-03-19T01:23:20.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.191+0000 7f2d7783d740 1 -- start start 2024-03-19T01:23:20.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.191+0000 7f2d7783d740 1 Processor -- start 2024-03-19T01:23:20.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.197+0000 7f2d7783d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654100 0x56158f65e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.197+0000 7f2d7783d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654af0 0x56158f65e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.197+0000 7f2d7783d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f665900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.197+0000 7f2d7783d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56158f65cc60 con 0x56158f654af0 2024-03-19T01:23:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.197+0000 7f2d7783d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56158f574db0 con 0x56158f659c60 2024-03-19T01:23:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.197+0000 7f2d7783d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56158f65cde0 con 0x56158f654100 2024-03-19T01:23:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.197+0000 7f2d7783d740 10 client.0 _ll_register_callbacks cb 0x56158f579e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:23:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.197+0000 7f2d6ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f665900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.197+0000 7f2d6ed9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f665900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:60532/0 (socket says 172.21.15.116:60532) 2024-03-19T01:23:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.197+0000 7f2d6ed9d640 1 -- 172.21.15.116:0/2379639069 learned_addr learned my addr 172.21.15.116:0/2379639069 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.197+0000 7f2d6dd9b640 1 --2- 172.21.15.116:0/2379639069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654af0 0x56158f65e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.197+0000 7f2d6e59c640 1 --2- 172.21.15.116:0/2379639069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654100 0x56158f65e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.198+0000 7f2d6ed9d640 1 -- 172.21.15.116:0/2379639069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654100 msgr2=0x56158f65e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:20.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.198+0000 7f2d6ed9d640 1 --2- 172.21.15.116:0/2379639069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654100 0x56158f65e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.198+0000 7f2d6ed9d640 1 -- 172.21.15.116:0/2379639069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654af0 msgr2=0x56158f65e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:20.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.198+0000 7f2d6ed9d640 1 --2- 172.21.15.116:0/2379639069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654af0 0x56158f65e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.198+0000 7f2d6ed9d640 1 -- 172.21.15.116:0/2379639069 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56158f573c60 con 0x56158f659c60 2024-03-19T01:23:20.200 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.199+0000 7f2d6ed9d640 1 --2- 172.21.15.116:0/2379639069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f665900 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2d6801c040 tx=0x7f2d6800fde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee1938afbb4ff798 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.200+0000 7f2d55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:23:20.202 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.200+0000 7f2d55ffb640 1 -- 172.21.15.116:0/2379639069 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d68028020 con 0x56158f659c60 2024-03-19T01:23:20.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.200+0000 7f2d55ffb640 1 -- 172.21.15.116:0/2379639069 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2d68005070 con 0x56158f659c60 2024-03-19T01:23:20.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.202+0000 7f2d7783d740 1 -- 172.21.15.116:0/2379639069 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56158f5c6120 con 0x56158f659c60 2024-03-19T01:23:20.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.203+0000 7f2d55ffb640 1 -- 172.21.15.116:0/2379639069 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f2d68005290 con 0x56158f659c60 2024-03-19T01:23:20.203 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.203+0000 7f2d7783d740 10 client.25958 fetch_fsmap learned FSMap version 45 2024-03-19T01:23:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.203+0000 7f2d7783d740 1 -- 172.21.15.116:0/2379639069 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56158f2cd210 con 0x56158f659c60 2024-03-19T01:23:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.204+0000 7f2d55ffb640 1 -- 172.21.15.116:0/2379639069 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x56158f5c6120 con 0x56158f659c60 2024-03-19T01:23:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.204+0000 7f2d55ffb640 1 -- 172.21.15.116:0/2379639069 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f2d68024660 con 0x56158f659c60 2024-03-19T01:23:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.204+0000 7f2d7783d740 10 client.25958 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:23:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.204+0000 7f2d7783d740 10 client.25958 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:23:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.204+0000 7f2d7783d740 20 client.25958 populate_metadata read hostname 'smithi116' 2024-03-19T01:23:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.204+0000 7f2d7783d740 1 --2- 172.21.15.116:0/2379639069 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56158f66a530 0x56158f66c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.204+0000 7f2d7783d740 4 client.25958 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:23:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.204+0000 7f2d7783d740 1 -- 172.21.15.116:0/2379639069 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56158f5c0470 con 0x56158f66a530 2024-03-19T01:23:20.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.204+0000 7f2d6e59c640 1 --2- 172.21.15.116:0/2379639069 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56158f66a530 0x56158f66c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.206+0000 7f2d6e59c640 1 --2- 172.21.15.116:0/2379639069 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56158f66a530 0x56158f66c950 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de11484c09017fe4 server_cookie=f5577d9830932561 in_seq=0 out_seq=0 2024-03-19T01:23:20.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.206+0000 7f2d55ffb640 10 client.25958 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.207+0000 7f2d55ffb640 1 -- 172.21.15.116:0/2379639069 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56158f5c0470 con 0x56158f66a530 2024-03-19T01:23:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.207+0000 7f2d55ffb640 10 client.25958 handle_command_reply: tid=0 2024-03-19T01:23:20.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.207+0000 7f2d7783d740 1 client.25958 shutdown 2024-03-19T01:23:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.209+0000 7f2d7783d740 1 -- 172.21.15.116:0/2379639069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 msgr2=0x56158f665900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.209+0000 7f2d7783d740 1 --2- 172.21.15.116:0/2379639069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f665900 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2d6801c040 tx=0x7f2d6800fde0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.210+0000 7f2d7783d740 1 -- 172.21.15.116:0/2379639069 shutdown_connections 2024-03-19T01:23:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.210+0000 7f2d7783d740 1 --2- 172.21.15.116:0/2379639069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f665900 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.210+0000 7f2d7783d740 1 --2- 172.21.15.116:0/2379639069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654af0 0x56158f65e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.210+0000 7f2d7783d740 1 --2- 172.21.15.116:0/2379639069 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56158f66a530 0x56158f66c950 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.210+0000 7f2d7783d740 1 --2- 172.21.15.116:0/2379639069 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654100 0x56158f65e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.210+0000 7f2d7783d740 1 -- 172.21.15.116:0/2379639069 >> 172.21.15.116:0/2379639069 conn(0x56158f64f670 msgr2=0x56158f6508c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.210+0000 7f2d55ffb640 0 client.25958 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.210+0000 7f2d7783d740 1 -- 172.21.15.116:0/2379639069 shutdown_connections 2024-03-19T01:23:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.210+0000 7f2d7783d740 1 -- 172.21.15.116:0/2379639069 wait complete. 2024-03-19T01:23:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.211+0000 7f2d7783d740 20 client.25958 trim_cache size 0 max 16384 2024-03-19T01:23:20.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.233+0000 7f2d7783d740 1 Processor -- start 2024-03-19T01:23:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.233+0000 7f2d7783d740 1 -- start start 2024-03-19T01:23:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.233+0000 7f2d7783d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 0x56158f64e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.233+0000 7f2d7783d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 0x56158f6658c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d7783d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f64e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d7783d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5000e690 con 0x56158f654100 2024-03-19T01:23:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d6e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 0x56158f64e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d6e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 0x56158f64e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40650/0 (socket says 172.21.15.116:40650) 2024-03-19T01:23:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d6e59c640 1 -- 172.21.15.116:0/1798920801 learned_addr learned my addr 172.21.15.116:0/1798920801 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d6dd9b640 1 --2- 172.21.15.116:0/1798920801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 0x56158f6658c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d6ed9d640 1 --2- 172.21.15.116:0/1798920801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f64e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d7783d740 1 -- 172.21.15.116:0/1798920801 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2d500101d0 con 0x56158f659c60 2024-03-19T01:23:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d7783d740 1 -- 172.21.15.116:0/1798920801 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5000ed60 con 0x56158f654af0 2024-03-19T01:23:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d6e59c640 1 -- 172.21.15.116:0/1798920801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 msgr2=0x56158f6658c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d6e59c640 1 --2- 172.21.15.116:0/1798920801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 0x56158f6658c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d6e59c640 1 -- 172.21.15.116:0/1798920801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 msgr2=0x56158f64e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d6e59c640 1 --2- 172.21.15.116:0/1798920801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f64e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d6e59c640 1 -- 172.21.15.116:0/1798920801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56158f399620 con 0x56158f654100 2024-03-19T01:23:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d6dd9b640 1 --2- 172.21.15.116:0/1798920801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 0x56158f6658c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:20.239 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d6e59c640 1 --2- 172.21.15.116:0/1798920801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 0x56158f64e220 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2d600096f0 tx=0x7f2d6003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419e92db9cfe0fdb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.234+0000 7f2d57fff640 1 -- 172.21.15.116:0/1798920801 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d60047040 con 0x56158f654100 2024-03-19T01:23:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.235+0000 7f2d57fff640 1 -- 172.21.15.116:0/1798920801 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d60048040 con 0x56158f654100 2024-03-19T01:23:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.235+0000 7f2d57fff640 1 -- 172.21.15.116:0/1798920801 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d600537e0 con 0x56158f654100 2024-03-19T01:23:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.235+0000 7f2d7783d740 1 -- 172.21.15.116:0/1798920801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 msgr2=0x56158f64e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.235+0000 7f2d7783d740 1 --2- 172.21.15.116:0/1798920801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 0x56158f64e220 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2d600096f0 tx=0x7f2d6003e000 comp rx=0 tx=0).stop 2024-03-19T01:23:20.240 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.235+0000 7f2d7783d740 1 -- 172.21.15.116:0/1798920801 shutdown_connections 2024-03-19T01:23:20.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.235+0000 7f2d7783d740 1 --2- 172.21.15.116:0/1798920801 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f64e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.235+0000 7f2d7783d740 1 --2- 172.21.15.116:0/1798920801 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 0x56158f6658c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.235+0000 7f2d7783d740 1 --2- 172.21.15.116:0/1798920801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 0x56158f64e220 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.235+0000 7f2d7783d740 1 -- 172.21.15.116:0/1798920801 >> 172.21.15.116:0/1798920801 conn(0x56158f64f670 msgr2=0x56158f65d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:20.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.235+0000 7f2d7783d740 1 -- 172.21.15.116:0/1798920801 shutdown_connections 2024-03-19T01:23:20.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.235+0000 7f2d7783d740 1 -- 172.21.15.116:0/1798920801 wait complete. 2024-03-19T01:23:20.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d7783d740 1 -- start start 2024-03-19T01:23:20.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d7783d740 1 Processor -- start 2024-03-19T01:23:20.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d7783d740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 0x56158f665380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d7783d740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 0x56158f6658c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.241 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d7783d740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f661e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d7783d740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5000fb00 con 0x56158f654100 2024-03-19T01:23:20.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d7783d740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5000f430 con 0x56158f659c60 2024-03-19T01:23:20.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d7783d740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5000c670 con 0x56158f654af0 2024-03-19T01:23:20.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d7783d740 10 client.0 _ll_register_callbacks cb 0x7f2d68009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:23:20.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d6e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 0x56158f665380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d6dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 0x56158f6658c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d6e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 0x56158f665380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40654/0 (socket says 172.21.15.116:40654) 2024-03-19T01:23:20.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d6e59c640 1 -- 172.21.15.116:0/3527251250 learned_addr learned my addr 172.21.15.116:0/3527251250 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:20.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.236+0000 7f2d6ed9d640 1 --2- 172.21.15.116:0/3527251250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f661e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.242 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.237+0000 7f2d6e59c640 1 -- 172.21.15.116:0/3527251250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 msgr2=0x56158f6658c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:20.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.237+0000 7f2d6e59c640 1 --2- 172.21.15.116:0/3527251250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 0x56158f6658c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.237+0000 7f2d6e59c640 1 -- 172.21.15.116:0/3527251250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 msgr2=0x56158f661e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:20.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.237+0000 7f2d6e59c640 1 --2- 172.21.15.116:0/3527251250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f661e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.243 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.237+0000 7f2d6e59c640 1 -- 172.21.15.116:0/3527251250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56158f662650 con 0x56158f654100 2024-03-19T01:23:20.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.238+0000 7f2d6e59c640 1 --2- 172.21.15.116:0/3527251250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 0x56158f665380 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2d60056040 tx=0x7f2d60053300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6828fc3b14ee157 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.238+0000 7f2d55ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:23:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.238+0000 7f2d55ffb640 1 -- 172.21.15.116:0/3527251250 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d60047040 con 0x56158f654100 2024-03-19T01:23:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.238+0000 7f2d7783d740 1 -- 172.21.15.116:0/3527251250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2d60048040 con 0x56158f654100 2024-03-19T01:23:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.238+0000 7f2d55ffb640 1 -- 172.21.15.116:0/3527251250 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2d6002c8b0 con 0x56158f654100 2024-03-19T01:23:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.238+0000 7f2d55ffb640 1 -- 172.21.15.116:0/3527251250 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f2d6002cab0 con 0x56158f654100 2024-03-19T01:23:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.238+0000 7f2d7783d740 10 client.16113 fetch_fsmap learned FSMap version 45 2024-03-19T01:23:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.239+0000 7f2d7783d740 1 -- 172.21.15.116:0/3527251250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x56158f6cf6c0 con 0x56158f654100 2024-03-19T01:23:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.239+0000 7f2d55ffb640 1 -- 172.21.15.116:0/3527251250 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d60061070 con 0x56158f654100 2024-03-19T01:23:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.239+0000 7f2d55ffb640 1 -- 172.21.15.116:0/3527251250 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f2d60034430 con 0x56158f654100 2024-03-19T01:23:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.239+0000 7f2d7783d740 10 client.16113 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:23:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.239+0000 7f2d7783d740 10 client.16113 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:23:20.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.239+0000 7f2d7783d740 20 client.16113 populate_metadata read hostname 'smithi116' 2024-03-19T01:23:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.240+0000 7f2d7783d740 1 --2- 172.21.15.116:0/3527251250 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56158f6d0010 0x56158f6d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.240+0000 7f2d7783d740 4 client.16113 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:23:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.240+0000 7f2d7783d740 1 -- 172.21.15.116:0/3527251250 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f2d60048040 con 0x56158f6d0010 2024-03-19T01:23:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.240+0000 7f2d6dd9b640 1 --2- 172.21.15.116:0/3527251250 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56158f6d0010 0x56158f6d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.241+0000 7f2d6dd9b640 1 --2- 172.21.15.116:0/3527251250 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56158f6d0010 0x56158f6d2430 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e7e949f16a4f58a server_cookie=4e34f88bb681d9f in_seq=0 out_seq=0 2024-03-19T01:23:20.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.241+0000 7f2d55ffb640 10 client.16113 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:20.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d55ffb640 1 -- 172.21.15.116:0/3527251250 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f2d60048040 con 0x56158f6d0010 2024-03-19T01:23:20.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d55ffb640 10 client.16113 handle_command_reply: tid=0 2024-03-19T01:23:20.250 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d7783d740 1 client.16113 shutdown 2024-03-19T01:23:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d7783d740 1 -- 172.21.15.116:0/3527251250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 msgr2=0x56158f665380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d7783d740 1 --2- 172.21.15.116:0/3527251250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 0x56158f665380 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2d60056040 tx=0x7f2d60053300 comp rx=0 tx=0).stop 2024-03-19T01:23:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d7783d740 1 -- 172.21.15.116:0/3527251250 shutdown_connections 2024-03-19T01:23:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d7783d740 1 --2- 172.21.15.116:0/3527251250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56158f659c60 0x56158f661e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d7783d740 1 --2- 172.21.15.116:0/3527251250 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56158f6d0010 0x56158f6d2430 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d55ffb640 0 client.16113 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:20.251 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d7783d740 1 --2- 172.21.15.116:0/3527251250 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56158f654af0 0x56158f6658c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d7783d740 1 --2- 172.21.15.116:0/3527251250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56158f654100 0x56158f665380 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d7783d740 1 -- 172.21.15.116:0/3527251250 >> 172.21.15.116:0/3527251250 conn(0x56158f64f670 msgr2=0x56158f6528d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d7783d740 1 -- 172.21.15.116:0/3527251250 shutdown_connections 2024-03-19T01:23:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.250+0000 7f2d7783d740 1 -- 172.21.15.116:0/3527251250 wait complete. 2024-03-19T01:23:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.251+0000 7f2d7783d740 20 client.16113 trim_cache size 0 max 16384 2024-03-19T01:23:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:23:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:23:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:23:20.252 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:23:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.251+0000 7f2d74c54640 1 -- 172.21.15.116:0/1904789039 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2d5c0a2ae0 msgr2=0x7f2d5c0a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.251+0000 7f2d74c54640 1 --2- 172.21.15.116:0/1904789039 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2d5c0a2ae0 0x7f2d5c0a4fa0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2d60036040 tx=0x7f2d60005fd0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.251+0000 7f2d74c54640 1 -- 172.21.15.116:0/1904789039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d7010b190 msgr2=0x7f2d70119ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.251+0000 7f2d74c54640 1 --2- 172.21.15.116:0/1904789039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d7010b190 0x7f2d70119ed0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800bea0 tx=0x7f2d5800bf90 comp rx=0 tx=0).stop 2024-03-19T01:23:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.251+0000 7f2d74c54640 1 -- 172.21.15.116:0/1904789039 shutdown_connections 2024-03-19T01:23:20.253 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.251+0000 7f2d74c54640 1 --2- 172.21.15.116:0/1904789039 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2d5c0a2ae0 0x7f2d5c0a4fa0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.251+0000 7f2d74c54640 1 --2- 172.21.15.116:0/1904789039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2d7010db40 0x7f2d70117fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.251+0000 7f2d74c54640 1 --2- 172.21.15.116:0/1904789039 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2d7010b190 0x7f2d70119ed0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.251+0000 7f2d74c54640 1 --2- 172.21.15.116:0/1904789039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d70108820 0x7f2d70119990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.251+0000 7f2d74c54640 1 -- 172.21.15.116:0/1904789039 >> 172.21.15.116:0/1904789039 conn(0x7f2d7006e860 msgr2=0x7f2d70108e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.251+0000 7f2d74c54640 1 -- 172.21.15.116:0/1904789039 shutdown_connections 2024-03-19T01:23:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:20.251+0000 7f2d74c54640 1 -- 172.21.15.116:0/1904789039 wait complete. 2024-03-19T01:23:21.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:20 smithi162 ceph-mon[67178]: pgmap v673: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.0 MiB/s wr, 662 op/s 2024-03-19T01:23:21.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:20 smithi116 ceph-mon[62701]: pgmap v673: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.0 MiB/s wr, 662 op/s 2024-03-19T01:23:21.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:20 smithi156 ceph-mon[66799]: pgmap v673: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 9.0 MiB/s wr, 662 op/s 2024-03-19T01:23:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 635.0 s, 2216.2 tps, lat 45.629 ms stddev 54.100 2024-03-19T01:23:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:23 smithi162 ceph-mon[67178]: pgmap v674: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 573 op/s 2024-03-19T01:23:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:23 smithi116 ceph-mon[62701]: pgmap v674: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 573 op/s 2024-03-19T01:23:23.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:23 smithi156 ceph-mon[66799]: pgmap v674: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.4 MiB/s wr, 573 op/s 2024-03-19T01:23:25.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:25 smithi162 ceph-mon[67178]: pgmap v675: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.0 MiB/s wr, 559 op/s 2024-03-19T01:23:25.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:25 smithi116 ceph-mon[62701]: pgmap v675: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.0 MiB/s wr, 559 op/s 2024-03-19T01:23:25.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:25 smithi156 ceph-mon[66799]: pgmap v675: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.0 MiB/s wr, 559 op/s 2024-03-19T01:23:26.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 640.0 s, 1665.6 tps, lat 59.525 ms stddev 65.809 2024-03-19T01:23:27.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:27 smithi162 ceph-mon[67178]: pgmap v676: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.8 MiB/s wr, 661 op/s 2024-03-19T01:23:27.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:27 smithi116 ceph-mon[62701]: pgmap v676: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.8 MiB/s wr, 661 op/s 2024-03-19T01:23:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:27 smithi156 ceph-mon[66799]: pgmap v676: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 8.8 MiB/s wr, 661 op/s 2024-03-19T01:23:29.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:29 smithi162 ceph-mon[67178]: pgmap v677: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.1 MiB/s wr, 613 op/s 2024-03-19T01:23:29.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:29 smithi116 ceph-mon[62701]: pgmap v677: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.1 MiB/s wr, 613 op/s 2024-03-19T01:23:29.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:29 smithi156 ceph-mon[66799]: pgmap v677: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.1 MiB/s wr, 613 op/s 2024-03-19T01:23:31.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:31 smithi162 ceph-mon[67178]: pgmap v678: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 672 op/s 2024-03-19T01:23:31.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:31 smithi116 ceph-mon[62701]: pgmap v678: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 672 op/s 2024-03-19T01:23:31.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:31 smithi156 ceph-mon[66799]: pgmap v678: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.6 MiB/s wr, 672 op/s 2024-03-19T01:23:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 645.0 s, 1646.0 tps, lat 60.579 ms stddev 67.889 2024-03-19T01:23:33.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.377+0000 7fb16cc9a640 1 -- 172.21.15.116:0/1641906094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1681053d0 msgr2=0x7fb168105850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.377+0000 7fb16cc9a640 1 --2- 172.21.15.116:0/1641906094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1681053d0 0x7fb168105850 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb160009be0 tx=0x7fb16002bef0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.377+0000 7fb16cc9a640 1 -- 172.21.15.116:0/1641906094 shutdown_connections 2024-03-19T01:23:33.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.377+0000 7fb16cc9a640 1 --2- 172.21.15.116:0/1641906094 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1681053d0 0x7fb168105850 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.377+0000 7fb16cc9a640 1 --2- 172.21.15.116:0/1641906094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168102a20 0x7fb168104e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.377+0000 7fb16cc9a640 1 --2- 172.21.15.116:0/1641906094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1681000b0 0x7fb1681024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.377+0000 7fb16cc9a640 1 -- 172.21.15.116:0/1641906094 >> 172.21.15.116:0/1641906094 conn(0x7fb1680fbcb0 msgr2=0x7fb1680fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.377+0000 7fb16cc9a640 1 -- 172.21.15.116:0/1641906094 shutdown_connections 2024-03-19T01:23:33.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.377+0000 7fb16cc9a640 1 -- 172.21.15.116:0/1641906094 wait complete. 2024-03-19T01:23:33.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb16cc9a640 1 Processor -- start 2024-03-19T01:23:33.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb16cc9a640 1 -- start start 2024-03-19T01:23:33.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb16cc9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1681000b0 0x7fb16819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb16cc9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168102a20 0x7fb16819e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb16cc9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1681053d0 0x7fb1681a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb16cc9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb168113ab0 con 0x7fb168102a20 2024-03-19T01:23:33.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb16cc9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb168113930 con 0x7fb1681053d0 2024-03-19T01:23:33.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb16cc9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb168113c30 con 0x7fb1681000b0 2024-03-19T01:23:33.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb166d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1681053d0 0x7fb1681a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb165d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168102a20 0x7fb16819e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.381 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb16659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1681000b0 0x7fb16819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb16659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1681000b0 0x7fb16819e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36456/0 (socket says 172.21.15.116:36456) 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb16659c640 1 -- 172.21.15.116:0/3141581883 learned_addr learned my addr 172.21.15.116:0/3141581883 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb165d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168102a20 0x7fb16819e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40052/0 (socket says 172.21.15.116:40052) 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.378+0000 7fb166d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1681053d0 0x7fb1681a55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:33760/0 (socket says 172.21.15.116:33760) 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.379+0000 7fb16659c640 1 -- 172.21.15.116:0/3141581883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1681053d0 msgr2=0x7fb1681a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.379+0000 7fb16659c640 1 --2- 172.21.15.116:0/3141581883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1681053d0 0x7fb1681a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.379+0000 7fb16659c640 1 -- 172.21.15.116:0/3141581883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168102a20 msgr2=0x7fb16819e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.379+0000 7fb16659c640 1 --2- 172.21.15.116:0/3141581883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168102a20 0x7fb16819e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.379+0000 7fb16659c640 1 -- 172.21.15.116:0/3141581883 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1681a5c30 con 0x7fb1681000b0 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.379+0000 7fb16659c640 1 --2- 172.21.15.116:0/3141581883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1681000b0 0x7fb16819e010 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb15800e9f0 tx=0x7fb15800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.379+0000 7fb1577fe640 1 -- 172.21.15.116:0/3141581883 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15800cd80 con 0x7fb1681000b0 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.379+0000 7fb1577fe640 1 -- 172.21.15.116:0/3141581883 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb158005500 con 0x7fb1681000b0 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.379+0000 7fb1577fe640 1 -- 172.21.15.116:0/3141581883 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb158010690 con 0x7fb1681000b0 2024-03-19T01:23:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.380+0000 7fb16cc9a640 1 -- 172.21.15.116:0/3141581883 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1681a5f20 con 0x7fb1681000b0 2024-03-19T01:23:33.383 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.381+0000 7fb16cc9a640 1 -- 172.21.15.116:0/3141581883 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1681a6330 con 0x7fb1681000b0 2024-03-19T01:23:33.384 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.384+0000 7fb1577fe640 1 -- 172.21.15.116:0/3141581883 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fb1580046c0 con 0x7fb1681000b0 2024-03-19T01:23:33.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.384+0000 7fb16f967740 1 Processor -- start 2024-03-19T01:23:33.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.385+0000 7fb16f967740 1 -- start start 2024-03-19T01:23:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.396+0000 7fb16f967740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cac100 0x55b0c1cac520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.396+0000 7fb16f967740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cb15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.396+0000 7fb16f967740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cb40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.396+0000 7fb16f967740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b0c1c10be0 con 0x55b0c1cacaf0 2024-03-19T01:23:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.396+0000 7fb16f967740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b0c1bb9960 con 0x55b0c1cb1c60 2024-03-19T01:23:33.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.396+0000 7fb16f967740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b0c1c0cc80 con 0x55b0c1cac100 2024-03-19T01:23:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.396+0000 7fb1577fe640 1 --2- 172.21.15.116:0/3141581883 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb1500a29a0 0x7fb1500a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.397+0000 7fb1577fe640 1 -- 172.21.15.116:0/3141581883 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb1580cef10 con 0x7fb1681000b0 2024-03-19T01:23:33.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:33 smithi162 ceph-mon[67178]: pgmap v679: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 8.4 MiB/s wr, 645 op/s 2024-03-19T01:23:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.398+0000 7fb166d9d640 1 --2- 172.21.15.116:0/3141581883 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb1500a29a0 0x7fb1500a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.398+0000 7fb166d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cac100 0x55b0c1cac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.398+0000 7fb166d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cac100 0x55b0c1cac520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36462/0 (socket says 172.21.15.116:36462) 2024-03-19T01:23:33.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.398+0000 7fb166d9d640 1 -- 172.21.15.116:0/3647163200 learned_addr learned my addr 172.21.15.116:0/3647163200 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:33.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.399+0000 7fb16659c640 1 --2- 172.21.15.116:0/3647163200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cb15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.399+0000 7fb166d9d640 1 -- 172.21.15.116:0/3647163200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cb1c60 msgr2=0x55b0c1cb40f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:23:33.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.399+0000 7fb166d9d640 1 --2- 172.21.15.116:0/3647163200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cb40f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.399+0000 7fb166d9d640 1 -- 172.21.15.116:0/3647163200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cacaf0 msgr2=0x55b0c1cb15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.399+0000 7fb166d9d640 1 --2- 172.21.15.116:0/3647163200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cb15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.399+0000 7fb166d9d640 1 -- 172.21.15.116:0/3647163200 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0c1bb0d20 con 0x55b0c1cac100 2024-03-19T01:23:33.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.399+0000 7fb166d9d640 1 --2- 172.21.15.116:0/3141581883 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb1500a29a0 0x7fb1500a4e60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb160033ff0 tx=0x7fb1600098c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.399+0000 7fb166d9d640 1 --2- 172.21.15.116:0/3647163200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cac100 0x55b0c1cac520 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb160036040 tx=0x7fb160048000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c12849d67371307a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.399+0000 7fb154ff9640 1 -- 172.21.15.116:0/3647163200 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160003e00 con 0x55b0c1cac100 2024-03-19T01:23:33.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.400+0000 7fb154ff9640 1 -- 172.21.15.116:0/3647163200 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb160051070 con 0x55b0c1cac100 2024-03-19T01:23:33.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.400+0000 7fb154ff9640 1 -- 172.21.15.116:0/3647163200 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160004520 con 0x55b0c1cac100 2024-03-19T01:23:33.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.401+0000 7fb16f967740 1 -- 172.21.15.116:0/3647163200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cac100 msgr2=0x55b0c1cac520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.401+0000 7fb16f967740 1 --2- 172.21.15.116:0/3647163200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cac100 0x55b0c1cac520 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb160036040 tx=0x7fb160048000 comp rx=0 tx=0).stop 2024-03-19T01:23:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.403+0000 7fb16f967740 1 -- 172.21.15.116:0/3647163200 shutdown_connections 2024-03-19T01:23:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.403+0000 7fb16f967740 1 --2- 172.21.15.116:0/3647163200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cb40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.403+0000 7fb16f967740 1 --2- 172.21.15.116:0/3647163200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cb15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.403+0000 7fb16f967740 1 --2- 172.21.15.116:0/3647163200 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cac100 0x55b0c1cac520 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.403 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.403+0000 7fb16f967740 1 -- 172.21.15.116:0/3647163200 >> 172.21.15.116:0/3647163200 conn(0x55b0c1ca7670 msgr2=0x55b0c1ca7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:33 smithi116 ceph-mon[62701]: pgmap v679: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 8.4 MiB/s wr, 645 op/s 2024-03-19T01:23:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.406+0000 7fb16f967740 1 -- 172.21.15.116:0/3647163200 shutdown_connections 2024-03-19T01:23:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.406+0000 7fb16f967740 1 -- 172.21.15.116:0/3647163200 wait complete. 2024-03-19T01:23:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.406+0000 7fb16f967740 1 -- start start 2024-03-19T01:23:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.407+0000 7fb16f967740 1 Processor -- start 2024-03-19T01:23:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.407+0000 7fb16f967740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cac100 0x55b0c1cb6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.407+0000 7fb16f967740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cb65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.407+0000 7fb16f967740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cbd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.407+0000 7fb16f967740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b0c1cb4bb0 con 0x55b0c1cb1c60 2024-03-19T01:23:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.407+0000 7fb16f967740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55b0c1bccdb0 con 0x55b0c1cac100 2024-03-19T01:23:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.407+0000 7fb16f967740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55b0c1cb4d30 con 0x55b0c1cacaf0 2024-03-19T01:23:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.407+0000 7fb16f967740 10 client.0 _ll_register_callbacks cb 0x55b0c1bd1e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:23:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.407+0000 7fb165d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cb65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.408+0000 7fb166d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cbd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.408+0000 7fb165d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cb65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:36472/0 (socket says 172.21.15.116:36472) 2024-03-19T01:23:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.408+0000 7fb165d9b640 1 -- 172.21.15.116:0/3890053080 learned_addr learned my addr 172.21.15.116:0/3890053080 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.408+0000 7fb16659c640 1 --2- 172.21.15.116:0/3890053080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cac100 0x55b0c1cb6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.408+0000 7fb166d9d640 1 -- 172.21.15.116:0/3890053080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cacaf0 msgr2=0x55b0c1cb65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.408+0000 7fb166d9d640 1 --2- 172.21.15.116:0/3890053080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cb65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.408+0000 7fb166d9d640 1 -- 172.21.15.116:0/3890053080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cac100 msgr2=0x55b0c1cb6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.408+0000 7fb166d9d640 1 --2- 172.21.15.116:0/3890053080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cac100 0x55b0c1cb6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.408+0000 7fb166d9d640 1 -- 172.21.15.116:0/3890053080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0c1bcbc60 con 0x55b0c1cb1c60 2024-03-19T01:23:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.408+0000 7fb166d9d640 1 --2- 172.21.15.116:0/3890053080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cbd5f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb160034fd0 tx=0x7fb160033cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f699ad0aebfc7544 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.409+0000 7fb14dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:23:33.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.409+0000 7fb14dffb640 1 -- 172.21.15.116:0/3890053080 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160059030 con 0x55b0c1cb1c60 2024-03-19T01:23:33.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.409+0000 7fb16f967740 1 -- 172.21.15.116:0/3890053080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb160051070 con 0x55b0c1cb1c60 2024-03-19T01:23:33.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.409+0000 7fb14dffb640 1 -- 172.21.15.116:0/3890053080 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb160049040 con 0x55b0c1cb1c60 2024-03-19T01:23:33.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.410+0000 7fb14dffb640 1 -- 172.21.15.116:0/3890053080 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb160064070 con 0x55b0c1cb1c60 2024-03-19T01:23:33.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.410+0000 7fb16f967740 10 client.16125 fetch_fsmap learned FSMap version 45 2024-03-19T01:23:33.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.410+0000 7fb16f967740 1 -- 172.21.15.116:0/3890053080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55b0c1925210 con 0x55b0c1cb1c60 2024-03-19T01:23:33.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.410+0000 7fb14dffb640 1 -- 172.21.15.116:0/3890053080 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb160051070 con 0x55b0c1cb1c60 2024-03-19T01:23:33.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.411+0000 7fb14dffb640 1 -- 172.21.15.116:0/3890053080 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fb1600345d0 con 0x55b0c1cb1c60 2024-03-19T01:23:33.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.411+0000 7fb16f967740 10 client.16125 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:23:33.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.411+0000 7fb16f967740 10 client.16125 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:23:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.411+0000 7fb16f967740 20 client.16125 populate_metadata read hostname 'smithi116' 2024-03-19T01:23:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.411+0000 7fb16f967740 1 --2- 172.21.15.116:0/3890053080 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b0c1cd2fa0 0x55b0c1cd53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.411+0000 7fb16f967740 4 client.16125 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:23:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.411+0000 7fb16f967740 1 -- 172.21.15.116:0/3890053080 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0c1c1e120 con 0x55b0c1cd2fa0 2024-03-19T01:23:33.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.412+0000 7fb16659c640 1 --2- 172.21.15.116:0/3890053080 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b0c1cd2fa0 0x55b0c1cd53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.413+0000 7fb14dffb640 10 client.16125 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:33.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.413+0000 7fb16659c640 1 --2- 172.21.15.116:0/3890053080 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b0c1cd2fa0 0x55b0c1cd53c0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13119e75872e43ad server_cookie=dcbef6568744eb6c in_seq=0 out_seq=0 2024-03-19T01:23:33.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.414+0000 7fb14dffb640 1 -- 172.21.15.116:0/3890053080 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b0c1c1e120 con 0x55b0c1cd2fa0 2024-03-19T01:23:33.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.414+0000 7fb14dffb640 10 client.16125 handle_command_reply: tid=0 2024-03-19T01:23:33.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.414+0000 7fb16f967740 1 client.16125 shutdown 2024-03-19T01:23:33.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.414+0000 7fb16f967740 1 -- 172.21.15.116:0/3890053080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cb1c60 msgr2=0x55b0c1cbd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.414+0000 7fb16f967740 1 --2- 172.21.15.116:0/3890053080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cbd5f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb160034fd0 tx=0x7fb160033cb0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.414+0000 7fb16f967740 1 -- 172.21.15.116:0/3890053080 shutdown_connections 2024-03-19T01:23:33.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.414+0000 7fb16f967740 1 --2- 172.21.15.116:0/3890053080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cbd5f0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.414+0000 7fb16f967740 1 --2- 172.21.15.116:0/3890053080 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b0c1cd2fa0 0x55b0c1cd53c0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.414+0000 7fb16f967740 1 --2- 172.21.15.116:0/3890053080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cb65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.414+0000 7fb16f967740 1 --2- 172.21.15.116:0/3890053080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cac100 0x55b0c1cb6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.414+0000 7fb16f967740 1 -- 172.21.15.116:0/3890053080 >> 172.21.15.116:0/3890053080 conn(0x55b0c1ca7670 msgr2=0x55b0c1ca88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.414+0000 7fb14dffb640 0 client.16125 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:33.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.415+0000 7fb16f967740 1 -- 172.21.15.116:0/3890053080 shutdown_connections 2024-03-19T01:23:33.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.415+0000 7fb16f967740 1 -- 172.21.15.116:0/3890053080 wait complete. 2024-03-19T01:23:33.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.415+0000 7fb16f967740 20 client.16125 trim_cache size 0 max 16384 2024-03-19T01:23:33.440 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.439+0000 7fb16f967740 1 Processor -- start 2024-03-19T01:23:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.439+0000 7fb16f967740 1 -- start start 2024-03-19T01:23:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.440+0000 7fb16f967740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cac100 0x55b0c1cb6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.440+0000 7fb16f967740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cbd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.440+0000 7fb16f967740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cb65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.440+0000 7fb16f967740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb1681a6b70 con 0x55b0c1cb1c60 2024-03-19T01:23:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.440+0000 7fb16f967740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb1681a86b0 con 0x55b0c1cacaf0 2024-03-19T01:23:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.440+0000 7fb16f967740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb1681a7240 con 0x55b0c1cac100 2024-03-19T01:23:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.440+0000 7fb166d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cb65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.440+0000 7fb165d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cbd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.440+0000 7fb165d9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cbd5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:33792/0 (socket says 172.21.15.116:33792) 2024-03-19T01:23:33.441 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.440+0000 7fb165d9b640 1 -- 172.21.15.116:0/1722104719 learned_addr learned my addr 172.21.15.116:0/1722104719 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.441+0000 7fb16659c640 1 --2- 172.21.15.116:0/1722104719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cac100 0x55b0c1cb6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.441+0000 7fb165d9b640 1 -- 172.21.15.116:0/1722104719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cac100 msgr2=0x55b0c1cb6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.441+0000 7fb165d9b640 1 --2- 172.21.15.116:0/1722104719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cac100 0x55b0c1cb6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.441+0000 7fb165d9b640 1 -- 172.21.15.116:0/1722104719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cb1c60 msgr2=0x55b0c1cb65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.441+0000 7fb165d9b640 1 --2- 172.21.15.116:0/1722104719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cb65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.441+0000 7fb165d9b640 1 -- 172.21.15.116:0/1722104719 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0c19f1620 con 0x55b0c1cacaf0 2024-03-19T01:23:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.441+0000 7fb165d9b640 1 --2- 172.21.15.116:0/1722104719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cbd5b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009530 tx=0x7fb15c012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7339ffbd7ca3a3b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.441+0000 7fb14ffff640 1 -- 172.21.15.116:0/1722104719 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15c0120d0 con 0x55b0c1cacaf0 2024-03-19T01:23:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.441+0000 7fb14ffff640 1 -- 172.21.15.116:0/1722104719 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb15c012270 con 0x55b0c1cacaf0 2024-03-19T01:23:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.441+0000 7fb14ffff640 1 -- 172.21.15.116:0/1722104719 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15c01c6d0 con 0x55b0c1cacaf0 2024-03-19T01:23:33.446 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.442+0000 7fb16f967740 1 -- 172.21.15.116:0/1722104719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cacaf0 msgr2=0x55b0c1cbd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.442+0000 7fb16f967740 1 --2- 172.21.15.116:0/1722104719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cbd5b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009530 tx=0x7fb15c012820 comp rx=0 tx=0).stop 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.442+0000 7fb16f967740 1 -- 172.21.15.116:0/1722104719 shutdown_connections 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.442+0000 7fb16f967740 1 --2- 172.21.15.116:0/1722104719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cb65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.442+0000 7fb16f967740 1 --2- 172.21.15.116:0/1722104719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cbd5b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.442+0000 7fb16f967740 1 --2- 172.21.15.116:0/1722104719 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cac100 0x55b0c1cb6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.442+0000 7fb16f967740 1 -- 172.21.15.116:0/1722104719 >> 172.21.15.116:0/1722104719 conn(0x55b0c1ca7670 msgr2=0x55b0c1cb1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.442+0000 7fb16f967740 1 -- 172.21.15.116:0/1722104719 shutdown_connections 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.442+0000 7fb16f967740 1 -- 172.21.15.116:0/1722104719 wait complete. 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb16f967740 1 -- start start 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb16f967740 1 Processor -- start 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb16f967740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cac100 0x55b0c1cba760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb16f967740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cbaca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb16f967740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cbbc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb16f967740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb1681a7fe0 con 0x55b0c1cac100 2024-03-19T01:23:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb16f967740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb1681a7910 con 0x55b0c1cb1c60 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb16f967740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb168071fe0 con 0x55b0c1cacaf0 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb16f967740 10 client.0 _ll_register_callbacks cb 0x7fb160009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb166d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cbbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb16659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cac100 0x55b0c1cba760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb166d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cbbc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:33804/0 (socket says 172.21.15.116:33804) 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb165d9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cbaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.443+0000 7fb166d9d640 1 -- 172.21.15.116:0/1164733960 learned_addr learned my addr 172.21.15.116:0/1164733960 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.444+0000 7fb166d9d640 1 -- 172.21.15.116:0/1164733960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cacaf0 msgr2=0x55b0c1cbaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.444+0000 7fb166d9d640 1 --2- 172.21.15.116:0/1164733960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cbaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.444+0000 7fb166d9d640 1 -- 172.21.15.116:0/1164733960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cac100 msgr2=0x55b0c1cba760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.444+0000 7fb166d9d640 1 --2- 172.21.15.116:0/1164733960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cac100 0x55b0c1cba760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.444+0000 7fb166d9d640 1 -- 172.21.15.116:0/1164733960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0c1cbc250 con 0x55b0c1cb1c60 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.444+0000 7fb166d9d640 1 --2- 172.21.15.116:0/1164733960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cbbc60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb160009be0 tx=0x7fb160034e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a2b95e21e4e0089 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.444+0000 7fb14dffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.444+0000 7fb14dffb640 1 -- 172.21.15.116:0/1164733960 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160059070 con 0x55b0c1cb1c60 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.444+0000 7fb14dffb640 1 -- 172.21.15.116:0/1164733960 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb160034430 con 0x55b0c1cb1c60 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.444+0000 7fb16f967740 1 -- 172.21.15.116:0/1164733960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb15c012270 con 0x55b0c1cb1c60 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.444+0000 7fb14dffb640 1 -- 172.21.15.116:0/1164733960 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fb160034630 con 0x55b0c1cb1c60 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.444+0000 7fb16f967740 10 client.25979 fetch_fsmap learned FSMap version 45 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.444+0000 7fb16f967740 1 -- 172.21.15.116:0/1164733960 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55b0c1cbd560 con 0x55b0c1cb1c60 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.445+0000 7fb14dffb640 1 -- 172.21.15.116:0/1164733960 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb15c012270 con 0x55b0c1cb1c60 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.445+0000 7fb14dffb640 1 -- 172.21.15.116:0/1164733960 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fb1600695a0 con 0x55b0c1cb1c60 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.445+0000 7fb16f967740 10 client.25979 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.445+0000 7fb16f967740 10 client.25979 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.445+0000 7fb16f967740 20 client.25979 populate_metadata read hostname 'smithi116' 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.445+0000 7fb16f967740 1 --2- 172.21.15.116:0/1164733960 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b0c1d39630 0x55b0c1d3ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.445+0000 7fb16f967740 4 client.25979 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.445+0000 7fb16f967740 1 -- 172.21.15.116:0/1164733960 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b0c1c18470 con 0x55b0c1d39630 2024-03-19T01:23:33.449 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.445+0000 7fb16659c640 1 --2- 172.21.15.116:0/1164733960 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b0c1d39630 0x55b0c1d3ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.446+0000 7fb14dffb640 10 client.25979 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:33.450 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.446+0000 7fb16659c640 1 --2- 172.21.15.116:0/1164733960 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b0c1d39630 0x55b0c1d3ba50 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=723152033f3b3ce4 server_cookie=7d98c96cfd3ec995 in_seq=0 out_seq=0 2024-03-19T01:23:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb14dffb640 1 -- 172.21.15.116:0/1164733960 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b0c1c18470 con 0x55b0c1d39630 2024-03-19T01:23:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb14dffb640 10 client.25979 handle_command_reply: tid=0 2024-03-19T01:23:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb16f967740 1 client.25979 shutdown 2024-03-19T01:23:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb16f967740 1 -- 172.21.15.116:0/1164733960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cb1c60 msgr2=0x55b0c1cbbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb16f967740 1 --2- 172.21.15.116:0/1164733960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cbbc60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb160009be0 tx=0x7fb160034e40 comp rx=0 tx=0).stop 2024-03-19T01:23:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb16f967740 1 -- 172.21.15.116:0/1164733960 shutdown_connections 2024-03-19T01:23:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb16f967740 1 --2- 172.21.15.116:0/1164733960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55b0c1cb1c60 0x55b0c1cbbc60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb16f967740 1 --2- 172.21.15.116:0/1164733960 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55b0c1d39630 0x55b0c1d3ba50 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.458 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb16f967740 1 --2- 172.21.15.116:0/1164733960 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55b0c1cacaf0 0x55b0c1cbaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb16f967740 1 --2- 172.21.15.116:0/1164733960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b0c1cac100 0x55b0c1cba760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb16f967740 1 -- 172.21.15.116:0/1164733960 >> 172.21.15.116:0/1164733960 conn(0x55b0c1ca7670 msgr2=0x55b0c1cb1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb14dffb640 0 client.25979 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb16f967740 1 -- 172.21.15.116:0/1164733960 shutdown_connections 2024-03-19T01:23:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb16f967740 1 -- 172.21.15.116:0/1164733960 wait complete. 2024-03-19T01:23:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.457+0000 7fb16f967740 20 client.25979 trim_cache size 0 max 16384 2024-03-19T01:23:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:23:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:23:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:23:33.459 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:23:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.458+0000 7fb16cc9a640 1 -- 172.21.15.116:0/3141581883 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb1500a29a0 msgr2=0x7fb1500a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.458+0000 7fb16cc9a640 1 --2- 172.21.15.116:0/3141581883 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb1500a29a0 0x7fb1500a4e60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb160033ff0 tx=0x7fb1600098c0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.458+0000 7fb16cc9a640 1 -- 172.21.15.116:0/3141581883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1681000b0 msgr2=0x7fb16819e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.458+0000 7fb16cc9a640 1 --2- 172.21.15.116:0/3141581883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1681000b0 0x7fb16819e010 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb15800e9f0 tx=0x7fb15800edf0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.460 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.458+0000 7fb16cc9a640 1 -- 172.21.15.116:0/3141581883 shutdown_connections 2024-03-19T01:23:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.458+0000 7fb16cc9a640 1 --2- 172.21.15.116:0/3141581883 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb1500a29a0 0x7fb1500a4e60 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.458+0000 7fb16cc9a640 1 --2- 172.21.15.116:0/3141581883 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb1681053d0 0x7fb1681a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.458+0000 7fb16cc9a640 1 --2- 172.21.15.116:0/3141581883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb168102a20 0x7fb16819e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.458+0000 7fb16cc9a640 1 --2- 172.21.15.116:0/3141581883 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb1681000b0 0x7fb16819e010 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.458+0000 7fb16cc9a640 1 -- 172.21.15.116:0/3141581883 >> 172.21.15.116:0/3141581883 conn(0x7fb1680fbcb0 msgr2=0x7fb168100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.458+0000 7fb16cc9a640 1 -- 172.21.15.116:0/3141581883 shutdown_connections 2024-03-19T01:23:33.461 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:33.458+0000 7fb16cc9a640 1 -- 172.21.15.116:0/3141581883 wait complete. 2024-03-19T01:23:33.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:33 smithi156 ceph-mon[66799]: pgmap v679: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 8.4 MiB/s wr, 645 op/s 2024-03-19T01:23:35.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:35 smithi162 ceph-mon[67178]: pgmap v680: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.9 MiB/s wr, 608 op/s 2024-03-19T01:23:35.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:35 smithi116 ceph-mon[62701]: pgmap v680: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.9 MiB/s wr, 608 op/s 2024-03-19T01:23:35.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:35 smithi156 ceph-mon[66799]: pgmap v680: 97 pgs: 97 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.9 MiB/s wr, 608 op/s 2024-03-19T01:23:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 650.0 s, 1446.4 tps, lat 68.448 ms stddev 80.819 2024-03-19T01:23:37.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:37 smithi162 ceph-mon[67178]: pgmap v681: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 739 op/s 2024-03-19T01:23:37.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:37 smithi116 ceph-mon[62701]: pgmap v681: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 739 op/s 2024-03-19T01:23:37.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:37 smithi156 ceph-mon[66799]: pgmap v681: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.3 MiB/s wr, 739 op/s 2024-03-19T01:23:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:39 smithi162 ceph-mon[67178]: pgmap v682: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.4 MiB/s wr, 646 op/s 2024-03-19T01:23:39.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:39 smithi116 ceph-mon[62701]: pgmap v682: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.4 MiB/s wr, 646 op/s 2024-03-19T01:23:39.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:39 smithi156 ceph-mon[66799]: pgmap v682: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.4 MiB/s wr, 646 op/s 2024-03-19T01:23:41.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:41 smithi162 ceph-mon[67178]: pgmap v683: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 704 op/s 2024-03-19T01:23:41.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:41 smithi116 ceph-mon[62701]: pgmap v683: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 704 op/s 2024-03-19T01:23:41.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:41 smithi156 ceph-mon[66799]: pgmap v683: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.9 MiB/s wr, 704 op/s 2024-03-19T01:23:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 655.0 s, 1446.2 tps, lat 69.919 ms stddev 84.154 2024-03-19T01:23:43.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:43 smithi162 ceph-mon[67178]: pgmap v684: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.0 MiB/s wr, 632 op/s 2024-03-19T01:23:43.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:43 smithi116 ceph-mon[62701]: pgmap v684: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.0 MiB/s wr, 632 op/s 2024-03-19T01:23:43.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:43 smithi156 ceph-mon[66799]: pgmap v684: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.0 MiB/s wr, 632 op/s 2024-03-19T01:23:45.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:45 smithi162 ceph-mon[67178]: pgmap v685: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 6.2 MiB/s wr, 582 op/s 2024-03-19T01:23:45.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:45 smithi116 ceph-mon[62701]: pgmap v685: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 6.2 MiB/s wr, 582 op/s 2024-03-19T01:23:45.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:45 smithi156 ceph-mon[66799]: pgmap v685: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 6.2 MiB/s wr, 582 op/s 2024-03-19T01:23:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 660.0 s, 1668.8 tps, lat 60.279 ms stddev 66.860 2024-03-19T01:23:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.571+0000 7fe82d844640 1 -- 172.21.15.116:0/4027030295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82810c260 msgr2=0x7fe82810e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.571+0000 7fe82d844640 1 --2- 172.21.15.116:0/4027030295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82810c260 0x7fe82810e650 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009900 tx=0x7fe81c02bc30 comp rx=0 tx=0).stop 2024-03-19T01:23:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.571+0000 7fe82d844640 1 -- 172.21.15.116:0/4027030295 shutdown_connections 2024-03-19T01:23:46.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.571+0000 7fe82d844640 1 --2- 172.21.15.116:0/4027030295 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe82810ebe0 0x7fe828111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.571+0000 7fe82d844640 1 --2- 172.21.15.116:0/4027030295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82810c260 0x7fe82810e650 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.571+0000 7fe82d844640 1 --2- 172.21.15.116:0/4027030295 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe828109930 0x7fe82810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.571+0000 7fe82d844640 1 -- 172.21.15.116:0/4027030295 >> 172.21.15.116:0/4027030295 conn(0x7fe82806ea20 msgr2=0x7fe828070e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:46.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.571+0000 7fe82d844640 1 -- 172.21.15.116:0/4027030295 shutdown_connections 2024-03-19T01:23:46.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.571+0000 7fe82d844640 1 -- 172.21.15.116:0/4027030295 wait complete. 2024-03-19T01:23:46.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.572+0000 7fe82d844640 1 Processor -- start 2024-03-19T01:23:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.572+0000 7fe82d844640 1 -- start start 2024-03-19T01:23:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.572+0000 7fe82d844640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe828109930 0x7fe82810b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.572+0000 7fe82d844640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe82810c260 0x7fe828109ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.572+0000 7fe82d844640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82810ebe0 0x7fe82810a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe82d844640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe828114bd0 con 0x7fe82810ebe0 2024-03-19T01:23:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe82d844640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe828114a50 con 0x7fe82810c260 2024-03-19T01:23:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe8267fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe82810c260 0x7fe828109ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe82d844640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe828114d50 con 0x7fe828109930 2024-03-19T01:23:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe8277fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82810ebe0 0x7fe82810a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe8267fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe82810c260 0x7fe828109ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:40916/0 (socket says 172.21.15.116:40916) 2024-03-19T01:23:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe8267fc640 1 -- 172.21.15.116:0/2595351835 learned_addr learned my addr 172.21.15.116:0/2595351835 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe826ffd640 1 --2- 172.21.15.116:0/2595351835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe828109930 0x7fe82810b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe8267fc640 1 -- 172.21.15.116:0/2595351835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe828109930 msgr2=0x7fe82810b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe8267fc640 1 --2- 172.21.15.116:0/2595351835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe828109930 0x7fe82810b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe8267fc640 1 -- 172.21.15.116:0/2595351835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82810ebe0 msgr2=0x7fe82810a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe8267fc640 1 --2- 172.21.15.116:0/2595351835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82810ebe0 0x7fe82810a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe8267fc640 1 -- 172.21.15.116:0/2595351835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe82810aca0 con 0x7fe82810c260 2024-03-19T01:23:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.573+0000 7fe8267fc640 1 --2- 172.21.15.116:0/2595351835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe82810c260 0x7fe828109ed0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009230 tx=0x7fe81c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.574+0000 7fe82c842640 1 -- 172.21.15.116:0/2595351835 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c041070 con 0x7fe82810c260 2024-03-19T01:23:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.574+0000 7fe82c842640 1 -- 172.21.15.116:0/2595351835 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe81c004450 con 0x7fe82810c260 2024-03-19T01:23:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.574+0000 7fe82c842640 1 -- 172.21.15.116:0/2595351835 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c003e20 con 0x7fe82810c260 2024-03-19T01:23:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.575+0000 7fe82d844640 1 -- 172.21.15.116:0/2595351835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8281aa370 con 0x7fe82810c260 2024-03-19T01:23:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.575+0000 7fe82d844640 1 -- 172.21.15.116:0/2595351835 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8281aa7b0 con 0x7fe82810c260 2024-03-19T01:23:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.578+0000 7fe82c842640 1 -- 172.21.15.116:0/2595351835 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fe81c044070 con 0x7fe82810c260 2024-03-19T01:23:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.578+0000 7fe83042f740 1 Processor -- start 2024-03-19T01:23:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.579+0000 7fe83042f740 1 -- start start 2024-03-19T01:23:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.579+0000 7fe82c842640 1 --2- 172.21.15.116:0/2595351835 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe8100a2a10 0x7fe8100a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.579+0000 7fe82c842640 1 -- 172.21.15.116:0/2595351835 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe81c0f0050 con 0x7fe82810c260 2024-03-19T01:23:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.579+0000 7fe8277fe640 1 --2- 172.21.15.116:0/2595351835 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe8100a2a10 0x7fe8100a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.579+0000 7fe83042f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6d910 0x5607e2f6dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.579+0000 7fe83042f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f6e300 0x5607e2f72e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.579+0000 7fe83042f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f75900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.579+0000 7fe83042f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5607e2ec9be0 con 0x5607e2f6d910 2024-03-19T01:23:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.579+0000 7fe83042f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5607e2e72960 con 0x5607e2f6e300 2024-03-19T01:23:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.579+0000 7fe83042f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5607e2ec5c80 con 0x5607e2f73470 2024-03-19T01:23:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.580+0000 7fe8277fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f75900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.580+0000 7fe826ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6d910 0x5607e2f6dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.580+0000 7fe826ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6d910 0x5607e2f6dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48656/0 (socket says 172.21.15.116:48656) 2024-03-19T01:23:46.582 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.580+0000 7fe826ffd640 1 -- 172.21.15.116:0/2831280797 learned_addr learned my addr 172.21.15.116:0/2831280797 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.580+0000 7fe826ffd640 1 -- 172.21.15.116:0/2831280797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 msgr2=0x5607e2f75900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.580+0000 7fe826ffd640 1 --2- 172.21.15.116:0/2831280797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f75900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.580+0000 7fe826ffd640 1 -- 172.21.15.116:0/2831280797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f6e300 msgr2=0x5607e2f72e00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:23:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.580+0000 7fe826ffd640 1 --2- 172.21.15.116:0/2831280797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f6e300 0x5607e2f72e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.580+0000 7fe826ffd640 1 -- 172.21.15.116:0/2831280797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607e2e69d20 con 0x5607e2f6d910 2024-03-19T01:23:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.581+0000 7fe826ffd640 1 --2- 172.21.15.116:0/2831280797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6d910 0x5607e2f6dd30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe818009c50 tx=0x7fe818011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40786f4b43c4d568 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.581+0000 7fe80dffb640 1 -- 172.21.15.116:0/2831280797 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe818004230 con 0x5607e2f6d910 2024-03-19T01:23:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.582+0000 7fe80dffb640 1 -- 172.21.15.116:0/2831280797 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8180043d0 con 0x5607e2f6d910 2024-03-19T01:23:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.582+0000 7fe80dffb640 1 -- 172.21.15.116:0/2831280797 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8180198a0 con 0x5607e2f6d910 2024-03-19T01:23:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.583+0000 7fe83042f740 1 -- 172.21.15.116:0/2831280797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6d910 msgr2=0x5607e2f6dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.583+0000 7fe83042f740 1 --2- 172.21.15.116:0/2831280797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6d910 0x5607e2f6dd30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe818009c50 tx=0x7fe818011810 comp rx=0 tx=0).stop 2024-03-19T01:23:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.585+0000 7fe8277fe640 1 --2- 172.21.15.116:0/2595351835 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe8100a2a10 0x7fe8100a4ed0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe820005fd0 tx=0x7fe8200118f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.585+0000 7fe83042f740 1 -- 172.21.15.116:0/2831280797 shutdown_connections 2024-03-19T01:23:46.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.585+0000 7fe83042f740 1 --2- 172.21.15.116:0/2831280797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f75900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.585+0000 7fe83042f740 1 --2- 172.21.15.116:0/2831280797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f6e300 0x5607e2f72e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.585+0000 7fe83042f740 1 --2- 172.21.15.116:0/2831280797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6d910 0x5607e2f6dd30 secure :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe818009c50 tx=0x7fe818011810 comp rx=0 tx=0).stop 2024-03-19T01:23:46.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.585+0000 7fe83042f740 1 -- 172.21.15.116:0/2831280797 >> 172.21.15.116:0/2831280797 conn(0x5607e2f60670 msgr2=0x5607e2f60a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.587+0000 7fe83042f740 1 -- 172.21.15.116:0/2831280797 shutdown_connections 2024-03-19T01:23:46.590 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.590+0000 7fe83042f740 1 -- 172.21.15.116:0/2831280797 wait complete. 2024-03-19T01:23:46.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.590+0000 7fe83042f740 1 -- start start 2024-03-19T01:23:46.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.590+0000 7fe83042f740 1 Processor -- start 2024-03-19T01:23:46.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.591+0000 7fe83042f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 0x5607e2f77b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.591+0000 7fe83042f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f73470 0x5607e2f78070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.591+0000 7fe83042f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f7d0a0 0x5607e2f7f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.591+0000 7fe83042f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5607e2f76420 con 0x5607e2f6e300 2024-03-19T01:23:46.591 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.591+0000 7fe83042f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5607e2e85db0 con 0x5607e2f73470 2024-03-19T01:23:46.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.591+0000 7fe83042f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5607e2f765a0 con 0x5607e2f7d0a0 2024-03-19T01:23:46.592 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.591+0000 7fe83042f740 10 client.0 _ll_register_callbacks cb 0x5607e2e8ae70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:23:46.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.594+0000 7fe8277fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f7d0a0 0x5607e2f7f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.594+0000 7fe8267fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f73470 0x5607e2f78070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.594 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.594+0000 7fe8267fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f73470 0x5607e2f78070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:38526/0 (socket says 172.21.15.116:38526) 2024-03-19T01:23:46.595 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.594+0000 7fe8267fc640 1 -- 172.21.15.116:0/1138460370 learned_addr learned my addr 172.21.15.116:0/1138460370 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.600+0000 7fe8277fe640 1 -- 172.21.15.116:0/1138460370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f73470 msgr2=0x5607e2f78070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.600+0000 7fe8277fe640 1 --2- 172.21.15.116:0/1138460370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f73470 0x5607e2f78070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.600+0000 7fe8277fe640 1 -- 172.21.15.116:0/1138460370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 msgr2=0x5607e2f77b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.600+0000 7fe8277fe640 1 --2- 172.21.15.116:0/1138460370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 0x5607e2f77b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.600+0000 7fe8277fe640 1 -- 172.21.15.116:0/1138460370 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607e2e84c60 con 0x5607e2f7d0a0 2024-03-19T01:23:46.602 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.602+0000 7fe8277fe640 1 --2- 172.21.15.116:0/1138460370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f7d0a0 0x5607e2f7f490 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe820011df0 tx=0x7fe820004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=663f7d2c0b6963d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.604+0000 7fe806ffd640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:23:46.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.604+0000 7fe806ffd640 1 -- 172.21.15.116:0/1138460370 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe820026430 con 0x5607e2f7d0a0 2024-03-19T01:23:46.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.604+0000 7fe806ffd640 1 -- 172.21.15.116:0/1138460370 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe8200265d0 con 0x5607e2f7d0a0 2024-03-19T01:23:46.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.604+0000 7fe83042f740 1 -- 172.21.15.116:0/1138460370 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8180043d0 con 0x5607e2f7d0a0 2024-03-19T01:23:46.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.609+0000 7fe806ffd640 1 -- 172.21.15.116:0/1138460370 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe8200267d0 con 0x5607e2f7d0a0 2024-03-19T01:23:46.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.610+0000 7fe83042f740 10 client.25984 fetch_fsmap learned FSMap version 45 2024-03-19T01:23:46.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.610+0000 7fe83042f740 1 -- 172.21.15.116:0/1138460370 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5607e2bde210 con 0x5607e2f7d0a0 2024-03-19T01:23:46.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.613+0000 7fe806ffd640 1 -- 172.21.15.116:0/1138460370 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe8180043d0 con 0x5607e2f7d0a0 2024-03-19T01:23:46.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.613+0000 7fe806ffd640 1 -- 172.21.15.116:0/1138460370 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fe82002fb80 con 0x5607e2f7d0a0 2024-03-19T01:23:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.614+0000 7fe83042f740 10 client.25984 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:23:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.614+0000 7fe83042f740 10 client.25984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:23:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.614+0000 7fe83042f740 20 client.25984 populate_metadata read hostname 'smithi116' 2024-03-19T01:23:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.614+0000 7fe83042f740 1 --2- 172.21.15.116:0/1138460370 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5607e2f69210 0x5607e2f6b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.614+0000 7fe83042f740 4 client.25984 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:23:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.614+0000 7fe83042f740 1 -- 172.21.15.116:0/1138460370 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607e2ed7120 con 0x5607e2f69210 2024-03-19T01:23:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.619+0000 7fe826ffd640 1 --2- 172.21.15.116:0/1138460370 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5607e2f69210 0x5607e2f6b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.631+0000 7fe826ffd640 1 --2- 172.21.15.116:0/1138460370 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5607e2f69210 0x5607e2f6b660 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b19feaf82d799713 server_cookie=b74faa1ac116496 in_seq=0 out_seq=0 2024-03-19T01:23:46.634 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.632+0000 7fe806ffd640 10 client.25984 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.634+0000 7fe806ffd640 1 -- 172.21.15.116:0/1138460370 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5607e2ed7120 con 0x5607e2f69210 2024-03-19T01:23:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.634+0000 7fe806ffd640 10 client.25984 handle_command_reply: tid=0 2024-03-19T01:23:46.636 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.634+0000 7fe83042f740 1 client.25984 shutdown 2024-03-19T01:23:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.637+0000 7fe83042f740 1 -- 172.21.15.116:0/1138460370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f7d0a0 msgr2=0x5607e2f7f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.637 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.637+0000 7fe83042f740 1 --2- 172.21.15.116:0/1138460370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f7d0a0 0x5607e2f7f490 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe820011df0 tx=0x7fe820004ab0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.643+0000 7fe83042f740 1 -- 172.21.15.116:0/1138460370 shutdown_connections 2024-03-19T01:23:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.643+0000 7fe83042f740 1 --2- 172.21.15.116:0/1138460370 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5607e2f69210 0x5607e2f6b660 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.643+0000 7fe83042f740 1 --2- 172.21.15.116:0/1138460370 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f7d0a0 0x5607e2f7f490 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.643+0000 7fe83042f740 1 --2- 172.21.15.116:0/1138460370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f73470 0x5607e2f78070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.643+0000 7fe83042f740 1 --2- 172.21.15.116:0/1138460370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 0x5607e2f77b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.643+0000 7fe83042f740 1 -- 172.21.15.116:0/1138460370 >> 172.21.15.116:0/1138460370 conn(0x5607e2f60670 msgr2=0x5607e2f5fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:46.643 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.643+0000 7fe806ffd640 0 client.25984 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:46.648 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.647+0000 7fe83042f740 1 -- 172.21.15.116:0/1138460370 shutdown_connections 2024-03-19T01:23:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.653+0000 7fe83042f740 1 -- 172.21.15.116:0/1138460370 wait complete. 2024-03-19T01:23:46.653 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.653+0000 7fe83042f740 20 client.25984 trim_cache size 0 max 16384 2024-03-19T01:23:46.684 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.684+0000 7fe83042f740 1 Processor -- start 2024-03-19T01:23:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.684+0000 7fe83042f740 1 -- start start 2024-03-19T01:23:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.684+0000 7fe83042f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 0x5607e2f63fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.684+0000 7fe83042f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f5f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.684+0000 7fe83042f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f7d0a0 0x5607e2f5f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.684+0000 7fe83042f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe8281af3b0 con 0x5607e2f6e300 2024-03-19T01:23:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.684+0000 7fe83042f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe8281b0ef0 con 0x5607e2f7d0a0 2024-03-19T01:23:46.685 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.684+0000 7fe83042f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe8281afa80 con 0x5607e2f73470 2024-03-19T01:23:46.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.684+0000 7fe8267fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f5f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.684+0000 7fe8267fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f5f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48494/0 (socket says 172.21.15.116:48494) 2024-03-19T01:23:46.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.684+0000 7fe8267fc640 1 -- 172.21.15.116:0/2510228204 learned_addr learned my addr 172.21.15.116:0/2510228204 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:46.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.684+0000 7fe8277fe640 1 --2- 172.21.15.116:0/2510228204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f7d0a0 0x5607e2f5f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.685+0000 7fe826ffd640 1 --2- 172.21.15.116:0/2510228204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 0x5607e2f63fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.685+0000 7fe8267fc640 1 -- 172.21.15.116:0/2510228204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f7d0a0 msgr2=0x5607e2f5f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:46.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.685+0000 7fe8267fc640 1 --2- 172.21.15.116:0/2510228204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f7d0a0 0x5607e2f5f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.686 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.685+0000 7fe8267fc640 1 -- 172.21.15.116:0/2510228204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 msgr2=0x5607e2f63fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:46.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.685+0000 7fe8267fc640 1 --2- 172.21.15.116:0/2510228204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 0x5607e2f63fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.685+0000 7fe8267fc640 1 -- 172.21.15.116:0/2510228204 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607e2f7f800 con 0x5607e2f73470 2024-03-19T01:23:46.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.685+0000 7fe8267fc640 1 --2- 172.21.15.116:0/2510228204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f5f220 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe81c000c00 tx=0x7fe81c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1190960210c2c4f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.685+0000 7fe80d7fa640 1 -- 172.21.15.116:0/2510228204 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c0d5070 con 0x5607e2f73470 2024-03-19T01:23:46.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.686+0000 7fe80d7fa640 1 -- 172.21.15.116:0/2510228204 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe81c0bd0d0 con 0x5607e2f73470 2024-03-19T01:23:46.687 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.686+0000 7fe80d7fa640 1 -- 172.21.15.116:0/2510228204 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c0b9040 con 0x5607e2f73470 2024-03-19T01:23:46.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.686+0000 7fe83042f740 1 -- 172.21.15.116:0/2510228204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 msgr2=0x5607e2f5f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 --2- 172.21.15.116:0/2510228204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f5f220 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe81c000c00 tx=0x7fe81c005fd0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 -- 172.21.15.116:0/2510228204 shutdown_connections 2024-03-19T01:23:46.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 --2- 172.21.15.116:0/2510228204 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f7d0a0 0x5607e2f5f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 --2- 172.21.15.116:0/2510228204 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f5f220 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 --2- 172.21.15.116:0/2510228204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 0x5607e2f63fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 -- 172.21.15.116:0/2510228204 >> 172.21.15.116:0/2510228204 conn(0x5607e2f60670 msgr2=0x5607e2f638d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:46.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 -- 172.21.15.116:0/2510228204 shutdown_connections 2024-03-19T01:23:46.690 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 -- 172.21.15.116:0/2510228204 wait complete. 2024-03-19T01:23:46.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 -- start start 2024-03-19T01:23:46.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 Processor -- start 2024-03-19T01:23:46.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 0x5607e2f9cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f9d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f7d0a0 0x5607e2fa04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe8281b0820 con 0x5607e2f6e300 2024-03-19T01:23:46.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe8281b0150 con 0x5607e2f7d0a0 2024-03-19T01:23:46.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fe8281ad3b0 con 0x5607e2f73470 2024-03-19T01:23:46.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.687+0000 7fe83042f740 10 client.0 _ll_register_callbacks cb 0x7fe82000f700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:23:46.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.688+0000 7fe8267fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f9d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.688+0000 7fe826ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 0x5607e2f9cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.691 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.688+0000 7fe8267fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f9d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48502/0 (socket says 172.21.15.116:48502) 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.688+0000 7fe8267fc640 1 -- 172.21.15.116:0/1812601418 learned_addr learned my addr 172.21.15.116:0/1812601418 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.688+0000 7fe8277fe640 1 --2- 172.21.15.116:0/1812601418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f7d0a0 0x5607e2fa04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.688+0000 7fe826ffd640 1 -- 172.21.15.116:0/1812601418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 msgr2=0x5607e2f9d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.688+0000 7fe826ffd640 1 --2- 172.21.15.116:0/1812601418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f9d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.688+0000 7fe826ffd640 1 -- 172.21.15.116:0/1812601418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f7d0a0 msgr2=0x5607e2fa04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.688+0000 7fe826ffd640 1 --2- 172.21.15.116:0/1812601418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f7d0a0 0x5607e2fa04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.688+0000 7fe826ffd640 1 -- 172.21.15.116:0/1812601418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607e2caa620 con 0x5607e2f6e300 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.688+0000 7fe826ffd640 1 --2- 172.21.15.116:0/1812601418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 0x5607e2f9cfe0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x5607e2f61610 tx=0x7fe818005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afef49dccdfa328a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.688+0000 7fe806ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe806ffd640 1 -- 172.21.15.116:0/1812601418 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81801f040 con 0x5607e2f6e300 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe806ffd640 1 -- 172.21.15.116:0/1812601418 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fe818005240 con 0x5607e2f6e300 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe83042f740 1 -- 172.21.15.116:0/1812601418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe81c0bd0d0 con 0x5607e2f6e300 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe806ffd640 1 -- 172.21.15.116:0/1812601418 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fe818036070 con 0x5607e2f6e300 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe83042f740 10 client.16146 fetch_fsmap learned FSMap version 45 2024-03-19T01:23:46.692 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe83042f740 1 -- 172.21.15.116:0/1812601418 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x5607e2ff1ef0 con 0x5607e2f6e300 2024-03-19T01:23:46.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe806ffd640 1 -- 172.21.15.116:0/1812601418 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe81c0bd0d0 con 0x5607e2f6e300 2024-03-19T01:23:46.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe806ffd640 1 -- 172.21.15.116:0/1812601418 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7fe818018070 con 0x5607e2f6e300 2024-03-19T01:23:46.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe83042f740 10 client.16146 fetch_fsmap finished waiting for FSMap version 45 2024-03-19T01:23:46.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe83042f740 10 client.16146 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:23:46.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe83042f740 20 client.16146 populate_metadata read hostname 'smithi116' 2024-03-19T01:23:46.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe83042f740 1 --2- 172.21.15.116:0/1812601418 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5607e2ff2690 0x5607e2ff4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:46.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe83042f740 4 client.16146 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:23:46.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.689+0000 7fe83042f740 1 -- 172.21.15.116:0/1812601418 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5607e2ed1470 con 0x5607e2ff2690 2024-03-19T01:23:46.693 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.690+0000 7fe8267fc640 1 --2- 172.21.15.116:0/1812601418 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5607e2ff2690 0x5607e2ff4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:46.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.690+0000 7fe8267fc640 1 --2- 172.21.15.116:0/1812601418 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5607e2ff2690 0x5607e2ff4ab0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6bc1e9f961047d4 server_cookie=c10377fc1169f951 in_seq=0 out_seq=0 2024-03-19T01:23:46.694 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.690+0000 7fe806ffd640 10 client.16146 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:46.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.699+0000 7fe806ffd640 1 -- 172.21.15.116:0/1812601418 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5607e2ed1470 con 0x5607e2ff2690 2024-03-19T01:23:46.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.699+0000 7fe806ffd640 10 client.16146 handle_command_reply: tid=0 2024-03-19T01:23:46.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.699+0000 7fe83042f740 1 client.16146 shutdown 2024-03-19T01:23:46.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.699+0000 7fe83042f740 1 -- 172.21.15.116:0/1812601418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 msgr2=0x5607e2f9cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.699+0000 7fe83042f740 1 --2- 172.21.15.116:0/1812601418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 0x5607e2f9cfe0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x5607e2f61610 tx=0x7fe818005ce0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.699+0000 7fe83042f740 1 -- 172.21.15.116:0/1812601418 shutdown_connections 2024-03-19T01:23:46.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.699+0000 7fe83042f740 1 --2- 172.21.15.116:0/1812601418 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5607e2f7d0a0 0x5607e2fa04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.699+0000 7fe83042f740 1 --2- 172.21.15.116:0/1812601418 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5607e2ff2690 0x5607e2ff4ab0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.699+0000 7fe83042f740 1 --2- 172.21.15.116:0/1812601418 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5607e2f73470 0x5607e2f9d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.699+0000 7fe83042f740 1 --2- 172.21.15.116:0/1812601418 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5607e2f6e300 0x5607e2f9cfe0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.699+0000 7fe83042f740 1 -- 172.21.15.116:0/1812601418 >> 172.21.15.116:0/1812601418 conn(0x5607e2f60670 msgr2=0x5607e2f638d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:46.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.700+0000 7fe806ffd640 0 client.16146 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:46.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.700+0000 7fe83042f740 1 -- 172.21.15.116:0/1812601418 shutdown_connections 2024-03-19T01:23:46.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.700+0000 7fe83042f740 1 -- 172.21.15.116:0/1812601418 wait complete. 2024-03-19T01:23:46.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.700+0000 7fe83042f740 20 client.16146 trim_cache size 0 max 16384 2024-03-19T01:23:46.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:23:46.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:23:46.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:23:46.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:23:46.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.701+0000 7fe82d844640 1 -- 172.21.15.116:0/2595351835 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe8100a2a10 msgr2=0x7fe8100a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.701+0000 7fe82d844640 1 --2- 172.21.15.116:0/2595351835 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe8100a2a10 0x7fe8100a4ed0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe820005fd0 tx=0x7fe8200118f0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.701+0000 7fe82d844640 1 -- 172.21.15.116:0/2595351835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe82810c260 msgr2=0x7fe828109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.701+0000 7fe82d844640 1 --2- 172.21.15.116:0/2595351835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe82810c260 0x7fe828109ed0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009230 tx=0x7fe81c02c650 comp rx=0 tx=0).stop 2024-03-19T01:23:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.701+0000 7fe82d844640 1 -- 172.21.15.116:0/2595351835 shutdown_connections 2024-03-19T01:23:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.701+0000 7fe82d844640 1 --2- 172.21.15.116:0/2595351835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82810ebe0 0x7fe82810a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.701+0000 7fe82d844640 1 --2- 172.21.15.116:0/2595351835 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fe8100a2a10 0x7fe8100a4ed0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.701+0000 7fe82d844640 1 --2- 172.21.15.116:0/2595351835 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe82810c260 0x7fe828109ed0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.701+0000 7fe82d844640 1 --2- 172.21.15.116:0/2595351835 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fe828109930 0x7fe82810b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.701+0000 7fe82d844640 1 -- 172.21.15.116:0/2595351835 >> 172.21.15.116:0/2595351835 conn(0x7fe82806ea20 msgr2=0x7fe828108d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:46.703 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.701+0000 7fe82d844640 1 -- 172.21.15.116:0/2595351835 shutdown_connections 2024-03-19T01:23:46.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:46.701+0000 7fe82d844640 1 -- 172.21.15.116:0/2595351835 wait complete. 2024-03-19T01:23:47.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:47 smithi162 ceph-mon[67178]: pgmap v686: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.3 MiB/s wr, 691 op/s 2024-03-19T01:23:47.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:47 smithi116 ceph-mon[62701]: pgmap v686: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.3 MiB/s wr, 691 op/s 2024-03-19T01:23:47.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:47 smithi156 ceph-mon[66799]: pgmap v686: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.3 MiB/s wr, 691 op/s 2024-03-19T01:23:49.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:49 smithi162 ceph-mon[67178]: pgmap v687: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.2 MiB/s wr, 586 op/s 2024-03-19T01:23:49.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:49 smithi116 ceph-mon[62701]: pgmap v687: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.2 MiB/s wr, 586 op/s 2024-03-19T01:23:49.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:49 smithi156 ceph-mon[66799]: pgmap v687: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.2 MiB/s wr, 586 op/s 2024-03-19T01:23:51.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:51 smithi162 ceph-mon[67178]: pgmap v688: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.9 MiB/s wr, 656 op/s 2024-03-19T01:23:51.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:51 smithi116 ceph-mon[62701]: pgmap v688: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.9 MiB/s wr, 656 op/s 2024-03-19T01:23:51.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:51 smithi156 ceph-mon[66799]: pgmap v688: 97 pgs: 97 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 6.9 MiB/s wr, 656 op/s 2024-03-19T01:23:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 665.0 s, 1656.2 tps, lat 60.343 ms stddev 74.513 2024-03-19T01:23:52.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:52 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:23:52.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:52 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:23:52.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:52 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:23:53.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:53 smithi162 ceph-mon[67178]: pgmap v689: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.7 MiB/s wr, 612 op/s 2024-03-19T01:23:53.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:53 smithi116 ceph-mon[62701]: pgmap v689: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.7 MiB/s wr, 612 op/s 2024-03-19T01:23:53.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:53 smithi156 ceph-mon[66799]: pgmap v689: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.7 MiB/s wr, 612 op/s 2024-03-19T01:23:55.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:55 smithi162 ceph-mon[67178]: pgmap v690: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.5 MiB/s wr, 577 op/s 2024-03-19T01:23:55.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:55 smithi116 ceph-mon[62701]: pgmap v690: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.5 MiB/s wr, 577 op/s 2024-03-19T01:23:55.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:55 smithi156 ceph-mon[66799]: pgmap v690: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.5 MiB/s wr, 577 op/s 2024-03-19T01:23:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 670.0 s, 1558.2 tps, lat 64.126 ms stddev 70.420 2024-03-19T01:23:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:57 smithi162 ceph-mon[67178]: pgmap v691: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.5 MiB/s wr, 687 op/s 2024-03-19T01:23:57.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:57 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:23:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:57 smithi116 ceph-mon[62701]: pgmap v691: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.5 MiB/s wr, 687 op/s 2024-03-19T01:23:57.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:57 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:23:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:57 smithi156 ceph-mon[66799]: pgmap v691: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.5 MiB/s wr, 687 op/s 2024-03-19T01:23:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:57 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:23:59.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:23:59 smithi162 ceph-mon[67178]: pgmap v692: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.7 MiB/s wr, 621 op/s 2024-03-19T01:23:59.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:23:59 smithi116 ceph-mon[62701]: pgmap v692: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.7 MiB/s wr, 621 op/s 2024-03-19T01:23:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:23:59 smithi156 ceph-mon[66799]: pgmap v692: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.7 MiB/s wr, 621 op/s 2024-03-19T01:23:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.830+0000 7fbdf2d37640 1 -- 172.21.15.116:0/1101392962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbdec101930 msgr2=0x7fbdec101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.830+0000 7fbdf2d37640 1 --2- 172.21.15.116:0/1101392962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbdec101930 0x7fbdec101db0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009860 tx=0x7fbde002bb70 comp rx=0 tx=0).stop 2024-03-19T01:23:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.830+0000 7fbdf2d37640 1 -- 172.21.15.116:0/1101392962 shutdown_connections 2024-03-19T01:23:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.830+0000 7fbdf2d37640 1 --2- 172.21.15.116:0/1101392962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdec1022f0 0x7fbdec108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.830+0000 7fbdf2d37640 1 --2- 172.21.15.116:0/1101392962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbdec101930 0x7fbdec101db0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.830+0000 7fbdf2d37640 1 --2- 172.21.15.116:0/1101392962 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdec100740 0x7fbdec100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.830+0000 7fbdf2d37640 1 -- 172.21.15.116:0/1101392962 >> 172.21.15.116:0/1101392962 conn(0x7fbdec0fbcb0 msgr2=0x7fbdec0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:59.837 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.831+0000 7fbdf2d37640 1 -- 172.21.15.116:0/1101392962 shutdown_connections 2024-03-19T01:23:59.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.831+0000 7fbdf2d37640 1 -- 172.21.15.116:0/1101392962 wait complete. 2024-03-19T01:23:59.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.832+0000 7fbdf2d37640 1 Processor -- start 2024-03-19T01:23:59.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.832+0000 7fbdf2d37640 1 -- start start 2024-03-19T01:23:59.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.832+0000 7fbdf2d37640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbdec100740 0x7fbdec1959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.840 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.832+0000 7fbdf2d37640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdec101930 0x7fbdec195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.832+0000 7fbdf2d37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdec1022f0 0x7fbdec19cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.832+0000 7fbdf2d37640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec10c410 con 0x7fbdec1022f0 2024-03-19T01:23:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.832+0000 7fbdf2d37640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec10c290 con 0x7fbdec100740 2024-03-19T01:23:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.832+0000 7fbdf2d37640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec10c590 con 0x7fbdec101930 2024-03-19T01:23:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.833+0000 7fbdf0ad3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbdec100740 0x7fbdec1959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.833+0000 7fbdf12d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdec1022f0 0x7fbdec19cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.833+0000 7fbdf12d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdec1022f0 0x7fbdec19cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60948/0 (socket says 172.21.15.116:60948) 2024-03-19T01:23:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.833+0000 7fbdf12d4640 1 -- 172.21.15.116:0/723438882 learned_addr learned my addr 172.21.15.116:0/723438882 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.833+0000 7fbdebfff640 1 --2- 172.21.15.116:0/723438882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdec101930 0x7fbdec195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.833+0000 7fbdf12d4640 1 -- 172.21.15.116:0/723438882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdec101930 msgr2=0x7fbdec195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.833+0000 7fbdf12d4640 1 --2- 172.21.15.116:0/723438882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdec101930 0x7fbdec195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.833+0000 7fbdf12d4640 1 -- 172.21.15.116:0/723438882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbdec100740 msgr2=0x7fbdec1959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.841 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.833+0000 7fbdf12d4640 1 --2- 172.21.15.116:0/723438882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbdec100740 0x7fbdec1959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.833+0000 7fbdf12d4640 1 -- 172.21.15.116:0/723438882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdec19d5c0 con 0x7fbdec1022f0 2024-03-19T01:23:59.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.834+0000 7fbdf12d4640 1 --2- 172.21.15.116:0/723438882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdec1022f0 0x7fbdec19cf50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009fd0 tx=0x7fbde400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.834+0000 7fbde9ffb640 1 -- 172.21.15.116:0/723438882 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde4004b20 con 0x7fbdec1022f0 2024-03-19T01:23:59.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.834+0000 7fbde9ffb640 1 -- 172.21.15.116:0/723438882 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbde4003e50 con 0x7fbdec1022f0 2024-03-19T01:23:59.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.834+0000 7fbdf2d37640 1 -- 172.21.15.116:0/723438882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdec19d8b0 con 0x7fbdec1022f0 2024-03-19T01:23:59.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.834+0000 7fbde9ffb640 1 -- 172.21.15.116:0/723438882 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde400b630 con 0x7fbdec1022f0 2024-03-19T01:23:59.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.835+0000 7fbdf2d37640 1 -- 172.21.15.116:0/723438882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdec19ddd0 con 0x7fbdec1022f0 2024-03-19T01:23:59.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.837+0000 7fbde9ffb640 1 -- 172.21.15.116:0/723438882 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fbde401d020 con 0x7fbdec1022f0 2024-03-19T01:23:59.842 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.838+0000 7fbdf59fa740 1 Processor -- start 2024-03-19T01:23:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.839+0000 7fbdf59fa740 1 -- start start 2024-03-19T01:23:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.839+0000 7fbdf59fa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 0x564e5835d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.847 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.839+0000 7fbdf59fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835daf0 0x564e583625f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.839+0000 7fbdf59fa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e58362c60 0x564e583650f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.839+0000 7fbdf59fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564e582c1be0 con 0x564e5835daf0 2024-03-19T01:23:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.839+0000 7fbdf59fa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564e5826a960 con 0x564e5835d100 2024-03-19T01:23:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.839+0000 7fbdf59fa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564e582bdc80 con 0x564e58362c60 2024-03-19T01:23:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.839+0000 7fbdebfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 0x564e5835d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.839+0000 7fbdebfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e58362c60 0x564e583650f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.839+0000 7fbdebfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 0x564e5835d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:55862/0 (socket says 172.21.15.116:55862) 2024-03-19T01:23:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.839+0000 7fbdebfff640 1 -- 172.21.15.116:0/2255441252 learned_addr learned my addr 172.21.15.116:0/2255441252 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.839+0000 7fbdf0ad3640 1 --2- 172.21.15.116:0/2255441252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835daf0 0x564e583625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.840+0000 7fbdebfff640 1 -- 172.21.15.116:0/2255441252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e58362c60 msgr2=0x564e583650f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:59.848 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.840+0000 7fbdebfff640 1 --2- 172.21.15.116:0/2255441252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e58362c60 0x564e583650f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.840+0000 7fbdebfff640 1 -- 172.21.15.116:0/2255441252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835daf0 msgr2=0x564e583625f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.840+0000 7fbdebfff640 1 --2- 172.21.15.116:0/2255441252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835daf0 0x564e583625f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.840+0000 7fbdebfff640 1 -- 172.21.15.116:0/2255441252 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e58261d20 con 0x564e5835d100 2024-03-19T01:23:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.840+0000 7fbdf0ad3640 1 --2- 172.21.15.116:0/2255441252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835daf0 0x564e583625f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.840+0000 7fbde9ffb640 1 --2- 172.21.15.116:0/723438882 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fbdd40a29f0 0x7fbdd40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.840+0000 7fbdebfff640 1 --2- 172.21.15.116:0/2255441252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 0x564e5835d520 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbde0005b60 tx=0x7fbde0035f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c92dc940955b9c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.840+0000 7fbdf0ad3640 1 --2- 172.21.15.116:0/723438882 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fbdd40a29f0 0x7fbdd40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.840+0000 7fbdd2ffd640 1 -- 172.21.15.116:0/2255441252 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde0039070 con 0x564e5835d100 2024-03-19T01:23:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.840+0000 7fbdd2ffd640 1 -- 172.21.15.116:0/2255441252 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbde003a040 con 0x564e5835d100 2024-03-19T01:23:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.840+0000 7fbdd2ffd640 1 -- 172.21.15.116:0/2255441252 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde00052d0 con 0x564e5835d100 2024-03-19T01:23:59.849 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.841+0000 7fbde9ffb640 1 -- 172.21.15.116:0/723438882 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbde40ceab0 con 0x7fbdec1022f0 2024-03-19T01:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.841+0000 7fbdf59fa740 1 -- 172.21.15.116:0/2255441252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 msgr2=0x564e5835d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.841+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/2255441252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 0x564e5835d520 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbde0005b60 tx=0x7fbde0035f80 comp rx=0 tx=0).stop 2024-03-19T01:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.841+0000 7fbdf0ad3640 1 --2- 172.21.15.116:0/723438882 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fbdd40a29f0 0x7fbdd40a4eb0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x564e5835def0 tx=0x7fbddc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.841+0000 7fbdf59fa740 1 -- 172.21.15.116:0/2255441252 shutdown_connections 2024-03-19T01:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.841+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/2255441252 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e58362c60 0x564e583650f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.841+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/2255441252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835daf0 0x564e583625f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.841+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/2255441252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 0x564e5835d520 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.841+0000 7fbdf59fa740 1 -- 172.21.15.116:0/2255441252 >> 172.21.15.116:0/2255441252 conn(0x564e58358670 msgr2=0x564e58358a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.841+0000 7fbdf59fa740 1 -- 172.21.15.116:0/2255441252 shutdown_connections 2024-03-19T01:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.841+0000 7fbdf59fa740 1 -- 172.21.15.116:0/2255441252 wait complete. 2024-03-19T01:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdf59fa740 1 -- start start 2024-03-19T01:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdf59fa740 1 Processor -- start 2024-03-19T01:23:59.850 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdf59fa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e5835d100 0x564e58367130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdf59fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835daf0 0x564e58367670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdf59fa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e58362c60 0x564e5836e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdf59fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x564e58365bb0 con 0x564e5835daf0 2024-03-19T01:23:59.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdf59fa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564e5827ddb0 con 0x564e58362c60 2024-03-19T01:23:59.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdf59fa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x564e58365d30 con 0x564e5835d100 2024-03-19T01:23:59.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdf59fa740 10 client.0 _ll_register_callbacks cb 0x564e58282e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:23:59.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835daf0 0x564e58367670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdf0ad3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e5835d100 0x564e58367130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdf0ad3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e5835d100 0x564e58367130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51036/0 (socket says 172.21.15.116:51036) 2024-03-19T01:23:59.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdf0ad3640 1 -- 172.21.15.116:0/1283702568 learned_addr learned my addr 172.21.15.116:0/1283702568 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:59.851 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdf12d4640 1 --2- 172.21.15.116:0/1283702568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e58362c60 0x564e5836e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdebfff640 1 -- 172.21.15.116:0/1283702568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e5835d100 msgr2=0x564e58367130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:59.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdebfff640 1 --2- 172.21.15.116:0/1283702568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e5835d100 0x564e58367130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdebfff640 1 -- 172.21.15.116:0/1283702568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e58362c60 msgr2=0x564e5836e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:59.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.842+0000 7fbdebfff640 1 --2- 172.21.15.116:0/1283702568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e58362c60 0x564e5836e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.843+0000 7fbdebfff640 1 -- 172.21.15.116:0/1283702568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e5827cc60 con 0x564e5835daf0 2024-03-19T01:23:59.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.843+0000 7fbdf0ad3640 1 --2- 172.21.15.116:0/1283702568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e5835d100 0x564e58367130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:23:59.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.843+0000 7fbdebfff640 1 --2- 172.21.15.116:0/1283702568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835daf0 0x564e58367670 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbde00095f0 tx=0x7fbde0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7e84a03070d5c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.843+0000 7fbdcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:23:59.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.843+0000 7fbdcbfff640 1 -- 172.21.15.116:0/1283702568 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde004a070 con 0x564e5835daf0 2024-03-19T01:23:59.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.843+0000 7fbdcbfff640 1 -- 172.21.15.116:0/1283702568 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbde0005020 con 0x564e5835daf0 2024-03-19T01:23:59.852 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.844+0000 7fbdf59fa740 1 -- 172.21.15.116:0/1283702568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbde003a040 con 0x564e5835daf0 2024-03-19T01:23:59.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.844+0000 7fbdcbfff640 1 -- 172.21.15.116:0/1283702568 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fbde0054050 con 0x564e5835daf0 2024-03-19T01:23:59.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.845+0000 7fbdf59fa740 10 client.16158 fetch_fsmap learned FSMap version 47 2024-03-19T01:23:59.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.845+0000 7fbdf59fa740 1 -- 172.21.15.116:0/1283702568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x564e57fd6210 con 0x564e5835daf0 2024-03-19T01:23:59.853 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.845+0000 7fbdcbfff640 1 -- 172.21.15.116:0/1283702568 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbde003a040 con 0x564e5835daf0 2024-03-19T01:23:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.845+0000 7fbdcbfff640 1 -- 172.21.15.116:0/1283702568 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7fbde002c9a0 con 0x564e5835daf0 2024-03-19T01:23:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.846+0000 7fbdf59fa740 10 client.16158 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:23:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.846+0000 7fbdf59fa740 10 client.16158 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:23:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.846+0000 7fbdf59fa740 20 client.16158 populate_metadata read hostname 'smithi116' 2024-03-19T01:23:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.846+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/1283702568 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e58372ba0 0x564e58374f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.846+0000 7fbdf59fa740 4 client.16158 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:23:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.846+0000 7fbdf59fa740 1 -- 172.21.15.116:0/1283702568 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e582cf120 con 0x564e58372ba0 2024-03-19T01:23:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.846+0000 7fbdf0ad3640 1 --2- 172.21.15.116:0/1283702568 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e58372ba0 0x564e58374f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.855 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.847+0000 7fbdf0ad3640 1 --2- 172.21.15.116:0/1283702568 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e58372ba0 0x564e58374f90 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50062c884a9411a4 server_cookie=6e9759fceace33d1 in_seq=0 out_seq=0 2024-03-19T01:23:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.847+0000 7fbdcbfff640 10 client.16158 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdcbfff640 1 -- 172.21.15.116:0/1283702568 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564e582cf120 con 0x564e58372ba0 2024-03-19T01:23:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdcbfff640 10 client.16158 handle_command_reply: tid=0 2024-03-19T01:23:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdf59fa740 1 client.16158 shutdown 2024-03-19T01:23:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdf59fa740 1 -- 172.21.15.116:0/1283702568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835daf0 msgr2=0x564e58367670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/1283702568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835daf0 0x564e58367670 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fbde00095f0 tx=0x7fbde0004270 comp rx=0 tx=0).stop 2024-03-19T01:23:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdf59fa740 1 -- 172.21.15.116:0/1283702568 shutdown_connections 2024-03-19T01:23:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/1283702568 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e58362c60 0x564e5836e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/1283702568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835daf0 0x564e58367670 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/1283702568 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e58372ba0 0x564e58374f90 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/1283702568 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e5835d100 0x564e58367130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdf59fa740 1 -- 172.21.15.116:0/1283702568 >> 172.21.15.116:0/1283702568 conn(0x564e58358670 msgr2=0x564e583598c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:59.856 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdcbfff640 0 client.16158 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdf59fa740 1 -- 172.21.15.116:0/1283702568 shutdown_connections 2024-03-19T01:23:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdf59fa740 1 -- 172.21.15.116:0/1283702568 wait complete. 2024-03-19T01:23:59.857 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.848+0000 7fbdf59fa740 20 client.16158 trim_cache size 0 max 16384 2024-03-19T01:23:59.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.866+0000 7fbdf59fa740 1 Processor -- start 2024-03-19T01:23:59.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.866+0000 7fbdf59fa740 1 -- start start 2024-03-19T01:23:59.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.866+0000 7fbdf59fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835d100 0x564e5835bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.866+0000 7fbdf59fa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835daf0 0x564e5836e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.869 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.866+0000 7fbdf59fa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e58362c60 0x564e58379670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.866+0000 7fbdf59fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec0737b0 con 0x564e5835d100 2024-03-19T01:23:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.866+0000 7fbdf59fa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec1c2090 con 0x564e5835daf0 2024-03-19T01:23:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.866+0000 7fbdf59fa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec1c0b60 con 0x564e58362c60 2024-03-19T01:23:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdf0ad3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835d100 0x564e5835bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdf0ad3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835d100 0x564e5835bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60974/0 (socket says 172.21.15.116:60974) 2024-03-19T01:23:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdf0ad3640 1 -- 172.21.15.116:0/4166501249 learned_addr learned my addr 172.21.15.116:0/4166501249 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdebfff640 1 --2- 172.21.15.116:0/4166501249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835daf0 0x564e5836e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdf12d4640 1 --2- 172.21.15.116:0/4166501249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e58362c60 0x564e58379670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdf0ad3640 1 -- 172.21.15.116:0/4166501249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e58362c60 msgr2=0x564e58379670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdf0ad3640 1 --2- 172.21.15.116:0/4166501249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e58362c60 0x564e58379670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdf0ad3640 1 -- 172.21.15.116:0/4166501249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835daf0 msgr2=0x564e5836e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdf0ad3640 1 --2- 172.21.15.116:0/4166501249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835daf0 0x564e5836e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.870 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdf0ad3640 1 -- 172.21.15.116:0/4166501249 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e580a2620 con 0x564e5835d100 2024-03-19T01:23:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdf0ad3640 1 --2- 172.21.15.116:0/4166501249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835d100 0x564e5835bfe0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fbddc009d50 tx=0x7fbddc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985600380afd6ee server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.871 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdd27fc640 1 -- 172.21.15.116:0/4166501249 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbddc004030 con 0x564e5835d100 2024-03-19T01:23:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdd27fc640 1 -- 172.21.15.116:0/4166501249 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbddc02e690 con 0x564e5835d100 2024-03-19T01:23:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.867+0000 7fbdd27fc640 1 -- 172.21.15.116:0/4166501249 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbddc003bd0 con 0x564e5835d100 2024-03-19T01:23:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.869+0000 7fbdf59fa740 1 -- 172.21.15.116:0/4166501249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835d100 msgr2=0x564e5835bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.869+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/4166501249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835d100 0x564e5835bfe0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fbddc009d50 tx=0x7fbddc017000 comp rx=0 tx=0).stop 2024-03-19T01:23:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.869+0000 7fbdf59fa740 1 -- 172.21.15.116:0/4166501249 shutdown_connections 2024-03-19T01:23:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.869+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/4166501249 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e58362c60 0x564e58379670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.869+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/4166501249 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835daf0 0x564e5836e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.869+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/4166501249 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e5835d100 0x564e5835bfe0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.869+0000 7fbdf59fa740 1 -- 172.21.15.116:0/4166501249 >> 172.21.15.116:0/4166501249 conn(0x564e58358670 msgr2=0x564e5835b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.869+0000 7fbdf59fa740 1 -- 172.21.15.116:0/4166501249 shutdown_connections 2024-03-19T01:23:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.869+0000 7fbdf59fa740 1 -- 172.21.15.116:0/4166501249 wait complete. 2024-03-19T01:23:59.872 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.869+0000 7fbdf59fa740 1 -- start start 2024-03-19T01:23:59.873 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.869+0000 7fbdf59fa740 1 Processor -- start 2024-03-19T01:23:59.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf59fa740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 0x564e5836c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf59fa740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e5835daf0 0x564e5836c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf59fa740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e58362c60 0x564e5836d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf59fa740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec1c19c0 con 0x564e58362c60 2024-03-19T01:23:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf59fa740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec1c12f0 con 0x564e5835d100 2024-03-19T01:23:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf59fa740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbdec19e500 con 0x564e5835daf0 2024-03-19T01:23:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf0ad3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 0x564e5836c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf0ad3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 0x564e5836c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:55876/0 (socket says 172.21.15.116:55876) 2024-03-19T01:23:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf0ad3640 1 -- 172.21.15.116:0/3407988280 learned_addr learned my addr 172.21.15.116:0/3407988280 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:23:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf59fa740 10 client.0 _ll_register_callbacks cb 0x7fbddc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:23:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdebfff640 1 --2- 172.21.15.116:0/3407988280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e5835daf0 0x564e5836c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.882 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf12d4640 1 --2- 172.21.15.116:0/3407988280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e58362c60 0x564e5836d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf0ad3640 1 -- 172.21.15.116:0/3407988280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e5835daf0 msgr2=0x564e5836c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf0ad3640 1 --2- 172.21.15.116:0/3407988280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e5835daf0 0x564e5836c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf0ad3640 1 -- 172.21.15.116:0/3407988280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e58362c60 msgr2=0x564e5836d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:23:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf0ad3640 1 --2- 172.21.15.116:0/3407988280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e58362c60 0x564e5836d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.871+0000 7fbdf0ad3640 1 -- 172.21.15.116:0/3407988280 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e5836def0 con 0x564e5835d100 2024-03-19T01:23:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.872+0000 7fbdf0ad3640 1 --2- 172.21.15.116:0/3407988280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 0x564e5836c3b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbddc01e7b0 tx=0x7fbddc02e840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6241aa2201360d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:23:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.872+0000 7fbdcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:23:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.872+0000 7fbdcbfff640 1 -- 172.21.15.116:0/3407988280 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbddc03b070 con 0x564e5835d100 2024-03-19T01:23:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.872+0000 7fbdcbfff640 1 -- 172.21.15.116:0/3407988280 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbddc005040 con 0x564e5835d100 2024-03-19T01:23:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.872+0000 7fbdf59fa740 1 -- 172.21.15.116:0/3407988280 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbddc02e690 con 0x564e5835d100 2024-03-19T01:23:59.883 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.872+0000 7fbdcbfff640 1 -- 172.21.15.116:0/3407988280 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fbddc005240 con 0x564e5835d100 2024-03-19T01:23:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.872+0000 7fbdf59fa740 10 client.26012 fetch_fsmap learned FSMap version 47 2024-03-19T01:23:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.872+0000 7fbdf59fa740 1 -- 172.21.15.116:0/3407988280 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x564e583d9240 con 0x564e5835d100 2024-03-19T01:23:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.872+0000 7fbdcbfff640 1 -- 172.21.15.116:0/3407988280 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbddc02e690 con 0x564e5835d100 2024-03-19T01:23:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.873+0000 7fbdcbfff640 1 -- 172.21.15.116:0/3407988280 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7fbddc040460 con 0x564e5835d100 2024-03-19T01:23:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.873+0000 7fbdf59fa740 10 client.26012 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:23:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.873+0000 7fbdf59fa740 10 client.26012 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:23:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.873+0000 7fbdf59fa740 20 client.26012 populate_metadata read hostname 'smithi116' 2024-03-19T01:23:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.873+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/3407988280 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e583d9bb0 0x564e583dbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:23:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.873+0000 7fbdf59fa740 4 client.26012 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:23:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.873+0000 7fbdf59fa740 1 -- 172.21.15.116:0/3407988280 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564e582c9470 con 0x564e583d9bb0 2024-03-19T01:23:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.873+0000 7fbdebfff640 1 --2- 172.21.15.116:0/3407988280 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e583d9bb0 0x564e583dbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:23:59.884 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.874+0000 7fbdcbfff640 10 client.26012 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:59.885 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.874+0000 7fbdebfff640 1 --2- 172.21.15.116:0/3407988280 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e583d9bb0 0x564e583dbfd0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25a31027d2678bd2 server_cookie=e7f7a8c648f7db4f in_seq=0 out_seq=0 2024-03-19T01:23:59.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.882+0000 7fbdcbfff640 1 -- 172.21.15.116:0/3407988280 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564e582c9470 con 0x564e583d9bb0 2024-03-19T01:23:59.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.882+0000 7fbdcbfff640 10 client.26012 handle_command_reply: tid=0 2024-03-19T01:23:59.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.882+0000 7fbdf59fa740 1 client.26012 shutdown 2024-03-19T01:23:59.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.882+0000 7fbdf59fa740 1 -- 172.21.15.116:0/3407988280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 msgr2=0x564e5836c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.882+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/3407988280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 0x564e5836c3b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbddc01e7b0 tx=0x7fbddc02e840 comp rx=0 tx=0).stop 2024-03-19T01:23:59.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.882+0000 7fbdf59fa740 1 -- 172.21.15.116:0/3407988280 shutdown_connections 2024-03-19T01:23:59.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.882+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/3407988280 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x564e58362c60 0x564e5836d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.882+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/3407988280 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x564e583d9bb0 0x564e583dbfd0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.882+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/3407988280 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x564e5835daf0 0x564e5836c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.882+0000 7fbdf59fa740 1 --2- 172.21.15.116:0/3407988280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564e5835d100 0x564e5836c3b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.882+0000 7fbdf59fa740 1 -- 172.21.15.116:0/3407988280 >> 172.21.15.116:0/3407988280 conn(0x564e58358670 msgr2=0x564e5835c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:59.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.882+0000 7fbdcbfff640 0 client.26012 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:23:59.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.883+0000 7fbdf59fa740 1 -- 172.21.15.116:0/3407988280 shutdown_connections 2024-03-19T01:23:59.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.883+0000 7fbdf59fa740 1 -- 172.21.15.116:0/3407988280 wait complete. 2024-03-19T01:23:59.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.883+0000 7fbdf59fa740 20 client.26012 trim_cache size 0 max 16384 2024-03-19T01:23:59.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:23:59.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:23:59.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:23:59.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:23:59.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.884+0000 7fbdf2d37640 1 -- 172.21.15.116:0/723438882 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fbdd40a29f0 msgr2=0x7fbdd40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.884+0000 7fbdf2d37640 1 --2- 172.21.15.116:0/723438882 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fbdd40a29f0 0x7fbdd40a4eb0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x564e5835def0 tx=0x7fbddc005f90 comp rx=0 tx=0).stop 2024-03-19T01:23:59.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.884+0000 7fbdf2d37640 1 -- 172.21.15.116:0/723438882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdec1022f0 msgr2=0x7fbdec19cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:23:59.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.884+0000 7fbdf2d37640 1 --2- 172.21.15.116:0/723438882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdec1022f0 0x7fbdec19cf50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fbde4009fd0 tx=0x7fbde400c800 comp rx=0 tx=0).stop 2024-03-19T01:23:59.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.884+0000 7fbdf2d37640 1 -- 172.21.15.116:0/723438882 shutdown_connections 2024-03-19T01:23:59.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.884+0000 7fbdf2d37640 1 --2- 172.21.15.116:0/723438882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbdec1022f0 0x7fbdec19cf50 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.884+0000 7fbdf2d37640 1 --2- 172.21.15.116:0/723438882 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbdec101930 0x7fbdec195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.884+0000 7fbdf2d37640 1 --2- 172.21.15.116:0/723438882 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fbdd40a29f0 0x7fbdd40a4eb0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.884+0000 7fbdf2d37640 1 --2- 172.21.15.116:0/723438882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbdec100740 0x7fbdec1959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:23:59.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.884+0000 7fbdf2d37640 1 -- 172.21.15.116:0/723438882 >> 172.21.15.116:0/723438882 conn(0x7fbdec0fbcb0 msgr2=0x7fbdec0fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:23:59.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.885+0000 7fbdf2d37640 1 -- 172.21.15.116:0/723438882 shutdown_connections 2024-03-19T01:23:59.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:23:59.885+0000 7fbdf2d37640 1 -- 172.21.15.116:0/723438882 wait complete. 2024-03-19T01:24:01.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:01 smithi162 ceph-mon[67178]: pgmap v693: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.0 MiB/s wr, 595 op/s 2024-03-19T01:24:01.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:01 smithi116 ceph-mon[62701]: pgmap v693: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.0 MiB/s wr, 595 op/s 2024-03-19T01:24:01.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:01 smithi156 ceph-mon[66799]: pgmap v693: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 6.0 MiB/s wr, 595 op/s 2024-03-19T01:24:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 675.0 s, 2090.4 tps, lat 47.563 ms stddev 58.288 2024-03-19T01:24:03.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:03 smithi162 ceph-mon[67178]: pgmap v694: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 MiB/s wr, 580 op/s 2024-03-19T01:24:03.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:03 smithi116 ceph-mon[62701]: pgmap v694: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 MiB/s wr, 580 op/s 2024-03-19T01:24:03.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:03 smithi156 ceph-mon[66799]: pgmap v694: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 MiB/s wr, 580 op/s 2024-03-19T01:24:05.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:05 smithi162 ceph-mon[67178]: pgmap v695: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 578 op/s 2024-03-19T01:24:05.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:05 smithi116 ceph-mon[62701]: pgmap v695: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 578 op/s 2024-03-19T01:24:05.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:05 smithi156 ceph-mon[66799]: pgmap v695: 97 pgs: 97 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 MiB/s wr, 578 op/s 2024-03-19T01:24:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 680.0 s, 2274.4 tps, lat 44.494 ms stddev 53.989 2024-03-19T01:24:07.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:07 smithi162 ceph-mon[67178]: pgmap v696: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 741 op/s 2024-03-19T01:24:07.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:07 smithi116 ceph-mon[62701]: pgmap v696: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 741 op/s 2024-03-19T01:24:07.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:07 smithi156 ceph-mon[66799]: pgmap v696: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 741 op/s 2024-03-19T01:24:09.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:09 smithi162 ceph-mon[67178]: pgmap v697: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8.5 MiB/s wr, 664 op/s 2024-03-19T01:24:09.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:09 smithi116 ceph-mon[62701]: pgmap v697: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8.5 MiB/s wr, 664 op/s 2024-03-19T01:24:09.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:09 smithi156 ceph-mon[66799]: pgmap v697: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 8.5 MiB/s wr, 664 op/s 2024-03-19T01:24:11.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:11 smithi162 ceph-mon[67178]: pgmap v698: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.9 MiB/s wr, 686 op/s 2024-03-19T01:24:11.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:11 smithi116 ceph-mon[62701]: pgmap v698: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.9 MiB/s wr, 686 op/s 2024-03-19T01:24:11.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:11 smithi156 ceph-mon[66799]: pgmap v698: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 8.9 MiB/s wr, 686 op/s 2024-03-19T01:24:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 685.0 s, 1707.2 tps, lat 58.564 ms stddev 68.625 2024-03-19T01:24:12.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:12 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:24:12.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:12 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:24:12.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:12 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:24:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.014+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2890866447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d50101950 msgr2=0x7f7d50101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.014+0000 7f7d57bb2640 1 --2- 172.21.15.116:0/2890866447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d50101950 0x7f7d50101dd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009900 tx=0x7f7d4002bc30 comp rx=0 tx=0).stop 2024-03-19T01:24:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.014+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2890866447 shutdown_connections 2024-03-19T01:24:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.014+0000 7f7d57bb2640 1 --2- 172.21.15.116:0/2890866447 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d50102310 0x7f7d50108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.014+0000 7f7d57bb2640 1 --2- 172.21.15.116:0/2890866447 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d50101950 0x7f7d50101dd0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.015 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.014+0000 7f7d57bb2640 1 --2- 172.21.15.116:0/2890866447 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d50100760 0x7f7d50100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.014+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2890866447 >> 172.21.15.116:0/2890866447 conn(0x7f7d500fbcf0 msgr2=0x7f7d500fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.015+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2890866447 shutdown_connections 2024-03-19T01:24:13.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.015+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2890866447 wait complete. 2024-03-19T01:24:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.015+0000 7f7d57bb2640 1 Processor -- start 2024-03-19T01:24:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.015+0000 7f7d57bb2640 1 -- start start 2024-03-19T01:24:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d57bb2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d50100760 0x7f7d50197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d57bb2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d50101950 0x7f7d50198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d57bb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d50102310 0x7f7d5019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d5594e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d50100760 0x7f7d50197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d57bb2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5010d500 con 0x7f7d50102310 2024-03-19T01:24:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d5594e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d50100760 0x7f7d50197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:37692/0 (socket says 172.21.15.116:37692) 2024-03-19T01:24:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d5514d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d50101950 0x7f7d50198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d5594e640 1 -- 172.21.15.116:0/2656398166 learned_addr learned my addr 172.21.15.116:0/2656398166 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2656398166 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5010d380 con 0x7f7d50100760 2024-03-19T01:24:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2656398166 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5010d680 con 0x7f7d50101950 2024-03-19T01:24:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d5614f640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d50102310 0x7f7d5019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d5514d640 1 -- 172.21.15.116:0/2656398166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d50100760 msgr2=0x7f7d50197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d5514d640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d50100760 0x7f7d50197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d5514d640 1 -- 172.21.15.116:0/2656398166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d50102310 msgr2=0x7f7d5019f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d5514d640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d50102310 0x7f7d5019f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d5514d640 1 -- 172.21.15.116:0/2656398166 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5019f710 con 0x7f7d50101950 2024-03-19T01:24:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.016+0000 7f7d5614f640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d50102310 0x7f7d5019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:24:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.017+0000 7f7d5514d640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d50101950 0x7f7d50198030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009230 tx=0x7f7d4002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.017+0000 7f7d46ffd640 1 -- 172.21.15.116:0/2656398166 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40041070 con 0x7f7d50101950 2024-03-19T01:24:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.017+0000 7f7d46ffd640 1 -- 172.21.15.116:0/2656398166 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d40004450 con 0x7f7d50101950 2024-03-19T01:24:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.017+0000 7f7d46ffd640 1 -- 172.21.15.116:0/2656398166 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d40003df0 con 0x7f7d50101950 2024-03-19T01:24:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.017+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2656398166 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d5019f9a0 con 0x7f7d50101950 2024-03-19T01:24:13.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.017+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2656398166 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d5019fd80 con 0x7f7d50101950 2024-03-19T01:24:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.020+0000 7f7d46ffd640 1 -- 172.21.15.116:0/2656398166 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7d40044070 con 0x7f7d50101950 2024-03-19T01:24:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.021+0000 7f7d46ffd640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7d3c0a2920 0x7f7d3c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.021+0000 7f7d5a79f740 1 Processor -- start 2024-03-19T01:24:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.021+0000 7f7d46ffd640 1 -- 172.21.15.116:0/2656398166 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7d400f0450 con 0x7f7d50101950 2024-03-19T01:24:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.021+0000 7f7d5a79f740 1 -- start start 2024-03-19T01:24:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.021+0000 7f7d5a79f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c097a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.021+0000 7f7d5a79f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c09d70 0x55a109c0c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.021+0000 7f7d5a79f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c0ced0 0x55a109c1ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.021+0000 7f7d5a79f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a109b72be0 con 0x55a109c09380 2024-03-19T01:24:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.021+0000 7f7d5a79f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a109b1b960 con 0x55a109c0ced0 2024-03-19T01:24:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.021+0000 7f7d5a79f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a109b6ec80 con 0x55a109c09d70 2024-03-19T01:24:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.021+0000 7f7d5594e640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7d3c0a2920 0x7f7d3c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.022+0000 7f7d5614f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c0ced0 0x55a109c1ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.022+0000 7f7d5614f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c0ced0 0x55a109c1ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:37698/0 (socket says 172.21.15.116:37698) 2024-03-19T01:24:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.022+0000 7f7d5614f640 1 -- 172.21.15.116:0/1911339362 learned_addr learned my addr 172.21.15.116:0/1911339362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.022+0000 7f7d5594e640 1 --2- 172.21.15.116:0/1911339362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.022+0000 7f7d5614f640 1 -- 172.21.15.116:0/1911339362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c09d70 msgr2=0x55a109c0c860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:24:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.022+0000 7f7d5614f640 1 --2- 172.21.15.116:0/1911339362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c09d70 0x55a109c0c860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.022+0000 7f7d5614f640 1 -- 172.21.15.116:0/1911339362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 msgr2=0x55a109c097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.022+0000 7f7d5614f640 1 --2- 172.21.15.116:0/1911339362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c097a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.022+0000 7f7d5614f640 1 -- 172.21.15.116:0/1911339362 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a109b12d20 con 0x55a109c0ced0 2024-03-19T01:24:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.022+0000 7f7d5594e640 1 --2- 172.21.15.116:0/1911339362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.022+0000 7f7d5614f640 1 --2- 172.21.15.116:0/1911339362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c0ced0 0x55a109c1ea40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7d480117e0 tx=0x7f7d48011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd06c5751b040a25 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.022+0000 7f7d5594e640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7d3c0a2920 0x7f7d3c0a4de0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x55a109c0a5d0 tx=0x7f7d4c0098f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.023+0000 7f7d3bfff640 1 -- 172.21.15.116:0/1911339362 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d48011310 con 0x55a109c0ced0 2024-03-19T01:24:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.023+0000 7f7d3bfff640 1 -- 172.21.15.116:0/1911339362 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d48003e50 con 0x55a109c0ced0 2024-03-19T01:24:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.023+0000 7f7d3bfff640 1 -- 172.21.15.116:0/1911339362 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d480045f0 con 0x55a109c0ced0 2024-03-19T01:24:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.024+0000 7f7d5a79f740 1 -- 172.21.15.116:0/1911339362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c0ced0 msgr2=0x55a109c1ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.024+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/1911339362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c0ced0 0x55a109c1ea40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7d480117e0 tx=0x7f7d48011ca0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.024+0000 7f7d5a79f740 1 -- 172.21.15.116:0/1911339362 shutdown_connections 2024-03-19T01:24:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.024+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/1911339362 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c0ced0 0x55a109c1ea40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.024+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/1911339362 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c09d70 0x55a109c0c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.024+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/1911339362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c097a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.024+0000 7f7d5a79f740 1 -- 172.21.15.116:0/1911339362 >> 172.21.15.116:0/1911339362 conn(0x55a109c0da80 msgr2=0x55a109c0de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.024+0000 7f7d5a79f740 1 -- 172.21.15.116:0/1911339362 shutdown_connections 2024-03-19T01:24:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.024+0000 7f7d5a79f740 1 -- 172.21.15.116:0/1911339362 wait complete. 2024-03-19T01:24:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.025+0000 7f7d5a79f740 1 -- start start 2024-03-19T01:24:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.025+0000 7f7d5a79f740 1 Processor -- start 2024-03-19T01:24:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5a79f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c18100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5a79f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09d70 0x55a109c18640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5a79f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c0ced0 0x55a109c1d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5a79f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55a109c12610 con 0x55a109c09380 2024-03-19T01:24:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5a79f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55a109b2edb0 con 0x55a109c09d70 2024-03-19T01:24:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5594e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c18100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5514d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09d70 0x55a109c18640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5594e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c18100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59466/0 (socket says 172.21.15.116:59466) 2024-03-19T01:24:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5614f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c0ced0 0x55a109c1d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5594e640 1 -- 172.21.15.116:0/341067638 learned_addr learned my addr 172.21.15.116:0/341067638 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5a79f740 1 -- 172.21.15.116:0/341067638 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55a109c12790 con 0x55a109c0ced0 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5a79f740 10 client.0 _ll_register_callbacks cb 0x55a109b33e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5594e640 1 -- 172.21.15.116:0/341067638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c0ced0 msgr2=0x55a109c1d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5594e640 1 --2- 172.21.15.116:0/341067638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c0ced0 0x55a109c1d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5594e640 1 -- 172.21.15.116:0/341067638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09d70 msgr2=0x55a109c18640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5594e640 1 --2- 172.21.15.116:0/341067638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09d70 0x55a109c18640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5594e640 1 -- 172.21.15.116:0/341067638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a109b2dc60 con 0x55a109c09380 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.026+0000 7f7d5514d640 1 --2- 172.21.15.116:0/341067638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09d70 0x55a109c18640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.027+0000 7f7d5594e640 1 --2- 172.21.15.116:0/341067638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c18100 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009870 tx=0x7f7d4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3940921a976e892f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.027+0000 7f7d397fa640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.028+0000 7f7d397fa640 1 -- 172.21.15.116:0/341067638 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c023070 con 0x55a109c09380 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.028+0000 7f7d397fa640 1 -- 172.21.15.116:0/341067638 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7d4c01f9d0 con 0x55a109c09380 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.028+0000 7f7d5a79f740 1 -- 172.21.15.116:0/341067638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d48003e50 con 0x55a109c09380 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.028+0000 7f7d397fa640 1 -- 172.21.15.116:0/341067638 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f7d4c013e60 con 0x55a109c09380 2024-03-19T01:24:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.028+0000 7f7d5a79f740 10 client.16173 fetch_fsmap learned FSMap version 47 2024-03-19T01:24:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.028+0000 7f7d5a79f740 1 -- 172.21.15.116:0/341067638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a109887210 con 0x55a109c09380 2024-03-19T01:24:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.028+0000 7f7d397fa640 1 -- 172.21.15.116:0/341067638 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d4c01fbd0 con 0x55a109c09380 2024-03-19T01:24:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.029+0000 7f7d397fa640 1 -- 172.21.15.116:0/341067638 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f7d4c032420 con 0x55a109c09380 2024-03-19T01:24:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.029+0000 7f7d5a79f740 10 client.16173 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:24:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.029+0000 7f7d5a79f740 10 client.16173 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:24:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.029+0000 7f7d5a79f740 20 client.16173 populate_metadata read hostname 'smithi116' 2024-03-19T01:24:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.029+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/341067638 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a109c34fa0 0x55a109c37410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.029+0000 7f7d5a79f740 4 client.16173 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:24:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.029+0000 7f7d5a79f740 1 -- 172.21.15.116:0/341067638 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7d48003e50 con 0x55a109c34fa0 2024-03-19T01:24:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.029+0000 7f7d5514d640 1 --2- 172.21.15.116:0/341067638 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a109c34fa0 0x55a109c37410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.029+0000 7f7d5514d640 1 --2- 172.21.15.116:0/341067638 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a109c34fa0 0x55a109c37410 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22146058473187b1 server_cookie=ef15ea63534a72a4 in_seq=0 out_seq=0 2024-03-19T01:24:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.030+0000 7f7d397fa640 10 client.16173 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.030+0000 7f7d397fa640 1 -- 172.21.15.116:0/341067638 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f7d48003e50 con 0x55a109c34fa0 2024-03-19T01:24:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.030+0000 7f7d397fa640 10 client.16173 handle_command_reply: tid=0 2024-03-19T01:24:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.030+0000 7f7d5a79f740 1 client.16173 shutdown 2024-03-19T01:24:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.030+0000 7f7d5a79f740 1 -- 172.21.15.116:0/341067638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 msgr2=0x55a109c18100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.030+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/341067638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c18100 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c009870 tx=0x7f7d4c0046c0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.031+0000 7f7d5a79f740 1 -- 172.21.15.116:0/341067638 shutdown_connections 2024-03-19T01:24:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.031+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/341067638 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a109c34fa0 0x55a109c37410 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.031+0000 7f7d397fa640 0 client.16173 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.031+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/341067638 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c0ced0 0x55a109c1d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.031+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/341067638 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09d70 0x55a109c18640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.031+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/341067638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c18100 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.031+0000 7f7d5a79f740 1 -- 172.21.15.116:0/341067638 >> 172.21.15.116:0/341067638 conn(0x55a109c0da80 msgr2=0x55a109c0ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.031+0000 7f7d5a79f740 1 -- 172.21.15.116:0/341067638 shutdown_connections 2024-03-19T01:24:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.031+0000 7f7d5a79f740 1 -- 172.21.15.116:0/341067638 wait complete. 2024-03-19T01:24:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.031+0000 7f7d5a79f740 20 client.16173 trim_cache size 0 max 16384 2024-03-19T01:24:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5a79f740 1 Processor -- start 2024-03-19T01:24:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5a79f740 1 -- start start 2024-03-19T01:24:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5a79f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09380 0x55a109c17d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.053 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5a79f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c09d70 0x55a109c18250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5a79f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c0ced0 0x55a109c12b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5a79f740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d501a6ad0 con 0x55a109c0ced0 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5a79f740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7d501a86d0 con 0x55a109c09380 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5a79f740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7d501a71a0 con 0x55a109c09d70 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5614f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c0ced0 0x55a109c12b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5514d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c09d70 0x55a109c18250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5614f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c0ced0 0x55a109c12b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59482/0 (socket says 172.21.15.116:59482) 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5514d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c09d70 0x55a109c18250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47438/0 (socket says 172.21.15.116:47438) 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5614f640 1 -- 172.21.15.116:0/4225511296 learned_addr learned my addr 172.21.15.116:0/4225511296 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5594e640 1 --2- 172.21.15.116:0/4225511296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09380 0x55a109c17d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5514d640 1 -- 172.21.15.116:0/4225511296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09380 msgr2=0x55a109c17d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5514d640 1 --2- 172.21.15.116:0/4225511296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09380 0x55a109c17d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5514d640 1 -- 172.21.15.116:0/4225511296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c0ced0 msgr2=0x55a109c12b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:13.054 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5514d640 1 --2- 172.21.15.116:0/4225511296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c0ced0 0x55a109c12b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.052+0000 7f7d5514d640 1 -- 172.21.15.116:0/4225511296 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a109953620 con 0x55a109c09d70 2024-03-19T01:24:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.053+0000 7f7d5514d640 1 --2- 172.21.15.116:0/4225511296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c09d70 0x55a109c18250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x55a109c0a7e0 tx=0x7f7d40009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f8357ce11e6d1a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.053+0000 7f7d3b7fe640 1 -- 172.21.15.116:0/4225511296 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4004b040 con 0x55a109c09d70 2024-03-19T01:24:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.053+0000 7f7d3b7fe640 1 -- 172.21.15.116:0/4225511296 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d40004c80 con 0x55a109c09d70 2024-03-19T01:24:13.055 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.053+0000 7f7d3b7fe640 1 -- 172.21.15.116:0/4225511296 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d400f00b0 con 0x55a109c09d70 2024-03-19T01:24:13.058 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.054+0000 7f7d5a79f740 1 -- 172.21.15.116:0/4225511296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c09d70 msgr2=0x55a109c18250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.058 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.054+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/4225511296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c09d70 0x55a109c18250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x55a109c0a7e0 tx=0x7f7d40009260 comp rx=0 tx=0).stop 2024-03-19T01:24:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.054+0000 7f7d5a79f740 1 -- 172.21.15.116:0/4225511296 shutdown_connections 2024-03-19T01:24:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.054+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/4225511296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c0ced0 0x55a109c12b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.054+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/4225511296 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c09d70 0x55a109c18250 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.054+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/4225511296 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09380 0x55a109c17d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.054+0000 7f7d5a79f740 1 -- 172.21.15.116:0/4225511296 >> 172.21.15.116:0/4225511296 conn(0x55a109c0da80 msgr2=0x55a109c04940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.054+0000 7f7d5a79f740 1 -- 172.21.15.116:0/4225511296 shutdown_connections 2024-03-19T01:24:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.054+0000 7f7d5a79f740 1 -- 172.21.15.116:0/4225511296 wait complete. 2024-03-19T01:24:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.054+0000 7f7d5a79f740 1 -- start start 2024-03-19T01:24:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.054+0000 7f7d5a79f740 1 Processor -- start 2024-03-19T01:24:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.055+0000 7f7d5a79f740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c484d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.055+0000 7f7d5a79f740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09d70 0x55a109c46bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.059 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.055+0000 7f7d5a79f740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c0ced0 0x55a109c47140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.055+0000 7f7d5514d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09d70 0x55a109c46bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.055+0000 7f7d5514d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09d70 0x55a109c46bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:37726/0 (socket says 172.21.15.116:37726) 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.055+0000 7f7d5a79f740 1 -- 172.21.15.116:0/2240358469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d501a8000 con 0x55a109c09380 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.055+0000 7f7d5a79f740 1 -- 172.21.15.116:0/2240358469 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7d501a7930 con 0x55a109c09d70 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.055+0000 7f7d5a79f740 1 -- 172.21.15.116:0/2240358469 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7d501a4ab0 con 0x55a109c0ced0 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.055+0000 7f7d5594e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.056+0000 7f7d5a79f740 10 client.0 _ll_register_callbacks cb 0x7f7d4c009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.055+0000 7f7d5514d640 1 -- 172.21.15.116:0/2240358469 learned_addr learned my addr 172.21.15.116:0/2240358469 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.055+0000 7f7d5614f640 1 --2- 172.21.15.116:0/2240358469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c0ced0 0x55a109c47140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.056+0000 7f7d5514d640 1 -- 172.21.15.116:0/2240358469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c0ced0 msgr2=0x55a109c47140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.056+0000 7f7d5514d640 1 --2- 172.21.15.116:0/2240358469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c0ced0 0x55a109c47140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.056+0000 7f7d5514d640 1 -- 172.21.15.116:0/2240358469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 msgr2=0x55a109c484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.056+0000 7f7d5514d640 1 --2- 172.21.15.116:0/2240358469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.056+0000 7f7d5514d640 1 -- 172.21.15.116:0/2240358469 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a109c47970 con 0x55a109c09d70 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.056+0000 7f7d5514d640 1 --2- 172.21.15.116:0/2240358469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09d70 0x55a109c46bc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d40006010 tx=0x7f7d4002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f22d050b0a9b9f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.056+0000 7f7d397fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.056+0000 7f7d397fa640 1 -- 172.21.15.116:0/2240358469 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d4004b040 con 0x55a109c09d70 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.056+0000 7f7d397fa640 1 -- 172.21.15.116:0/2240358469 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7d400c50e0 con 0x55a109c09d70 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.056+0000 7f7d5a79f740 1 -- 172.21.15.116:0/2240358469 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d40004c80 con 0x55a109c09d70 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.057+0000 7f7d397fa640 1 -- 172.21.15.116:0/2240358469 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f7d400a1070 con 0x55a109c09d70 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.057+0000 7f7d5a79f740 10 client.26030 fetch_fsmap learned FSMap version 47 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.057+0000 7f7d5a79f740 1 -- 172.21.15.116:0/2240358469 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a109c9be80 con 0x55a109c09d70 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.057+0000 7f7d397fa640 1 -- 172.21.15.116:0/2240358469 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d400c5570 con 0x55a109c09d70 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.057+0000 7f7d397fa640 1 -- 172.21.15.116:0/2240358469 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f7d400d4030 con 0x55a109c09d70 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.057+0000 7f7d5a79f740 10 client.26030 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.057+0000 7f7d5a79f740 10 client.26030 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.057+0000 7f7d5a79f740 20 client.26030 populate_metadata read hostname 'smithi116' 2024-03-19T01:24:13.061 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.057+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/2240358469 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a109c34fa0 0x55a109c9e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.057+0000 7f7d5a79f740 4 client.26030 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:24:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.057+0000 7f7d5a79f740 1 -- 172.21.15.116:0/2240358469 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f7d40004c80 con 0x55a109c34fa0 2024-03-19T01:24:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.057+0000 7f7d5594e640 1 --2- 172.21.15.116:0/2240358469 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a109c34fa0 0x55a109c9e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.058+0000 7f7d5594e640 1 --2- 172.21.15.116:0/2240358469 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a109c34fa0 0x55a109c9e630 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eab7f18ef9cbc33c server_cookie=c8c8c2fac5d1ea5a in_seq=0 out_seq=0 2024-03-19T01:24:13.062 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.058+0000 7f7d397fa640 10 client.26030 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.067+0000 7f7d397fa640 1 -- 172.21.15.116:0/2240358469 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f7d40004c80 con 0x55a109c34fa0 2024-03-19T01:24:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.067+0000 7f7d397fa640 10 client.26030 handle_command_reply: tid=0 2024-03-19T01:24:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.067+0000 7f7d5a79f740 1 client.26030 shutdown 2024-03-19T01:24:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.068+0000 7f7d5a79f740 1 -- 172.21.15.116:0/2240358469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09d70 msgr2=0x55a109c46bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.068+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/2240358469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09d70 0x55a109c46bc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d40006010 tx=0x7f7d4002bc30 comp rx=0 tx=0).stop 2024-03-19T01:24:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.068+0000 7f7d5a79f740 1 -- 172.21.15.116:0/2240358469 shutdown_connections 2024-03-19T01:24:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.068+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/2240358469 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55a109c34fa0 0x55a109c9e630 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.068+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/2240358469 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55a109c0ced0 0x55a109c47140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.068+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/2240358469 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55a109c09d70 0x55a109c46bc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.068+0000 7f7d5a79f740 1 --2- 172.21.15.116:0/2240358469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55a109c09380 0x55a109c484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.068+0000 7f7d5a79f740 1 -- 172.21.15.116:0/2240358469 >> 172.21.15.116:0/2240358469 conn(0x55a109c0da80 msgr2=0x55a109c17cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.068+0000 7f7d397fa640 0 client.26030 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.068+0000 7f7d5a79f740 1 -- 172.21.15.116:0/2240358469 shutdown_connections 2024-03-19T01:24:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.068+0000 7f7d5a79f740 1 -- 172.21.15.116:0/2240358469 wait complete. 2024-03-19T01:24:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.068+0000 7f7d5a79f740 20 client.26030 trim_cache size 0 max 16384 2024-03-19T01:24:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:24:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:24:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:24:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:24:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.069+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2656398166 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7d3c0a2920 msgr2=0x7f7d3c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.069+0000 7f7d57bb2640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7d3c0a2920 0x7f7d3c0a4de0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x55a109c0a5d0 tx=0x7f7d4c0098f0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.069+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2656398166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d50101950 msgr2=0x7f7d50198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.069+0000 7f7d57bb2640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d50101950 0x7f7d50198030 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7d40009230 tx=0x7f7d4002c650 comp rx=0 tx=0).stop 2024-03-19T01:24:13.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.069+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2656398166 shutdown_connections 2024-03-19T01:24:13.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.069+0000 7f7d57bb2640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d50102310 0x7f7d5019f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.069+0000 7f7d57bb2640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7d50101950 0x7f7d50198030 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.069+0000 7f7d57bb2640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7d3c0a2920 0x7f7d3c0a4de0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.069+0000 7f7d57bb2640 1 --2- 172.21.15.116:0/2656398166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d50100760 0x7f7d50197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:13.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.069+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2656398166 >> 172.21.15.116:0/2656398166 conn(0x7f7d500fbcf0 msgr2=0x7f7d50104c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:13.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.069+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2656398166 shutdown_connections 2024-03-19T01:24:13.072 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:13.069+0000 7f7d57bb2640 1 -- 172.21.15.116:0/2656398166 wait complete. 2024-03-19T01:24:13.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:13 smithi162 ceph-mon[67178]: pgmap v699: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 736 op/s 2024-03-19T01:24:13.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:13 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:13.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:13 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:13.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:13 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:13.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:13 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:13.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:13 smithi116 ceph-mon[62701]: pgmap v699: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 736 op/s 2024-03-19T01:24:13.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:13 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:13.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:13 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:13.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:13 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:13.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:13 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:13.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:13 smithi156 ceph-mon[66799]: pgmap v699: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 736 op/s 2024-03-19T01:24:13.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:13 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:13.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:13 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:13.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:13 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:13.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:13 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:24:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:24:14.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:24:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:24:14.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:24:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:24:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:24:15.375 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:15 smithi156 ceph-mon[66799]: pgmap v700: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 663 op/s 2024-03-19T01:24:15.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:15 smithi162 ceph-mon[67178]: pgmap v700: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 663 op/s 2024-03-19T01:24:15.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:15 smithi116 ceph-mon[62701]: pgmap v700: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 MiB/s wr, 663 op/s 2024-03-19T01:24:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 690.0 s, 1519.4 tps, lat 65.270 ms stddev 74.368 2024-03-19T01:24:17.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:17 smithi162 ceph-mon[67178]: pgmap v701: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.8 MiB/s wr, 717 op/s 2024-03-19T01:24:17.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:17 smithi116 ceph-mon[62701]: pgmap v701: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.8 MiB/s wr, 717 op/s 2024-03-19T01:24:17.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:17 smithi156 ceph-mon[66799]: pgmap v701: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.8 MiB/s wr, 717 op/s 2024-03-19T01:24:19.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:19 smithi162 ceph-mon[67178]: pgmap v702: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 612 op/s 2024-03-19T01:24:19.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:19 smithi116 ceph-mon[62701]: pgmap v702: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 612 op/s 2024-03-19T01:24:19.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:19 smithi156 ceph-mon[66799]: pgmap v702: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.1 MiB/s wr, 612 op/s 2024-03-19T01:24:21.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:21 smithi162 ceph-mon[67178]: pgmap v703: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.2 MiB/s wr, 632 op/s 2024-03-19T01:24:21.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:21 smithi116 ceph-mon[62701]: pgmap v703: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.2 MiB/s wr, 632 op/s 2024-03-19T01:24:21.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:21 smithi156 ceph-mon[66799]: pgmap v703: 97 pgs: 97 active+clean; 1.6 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.2 MiB/s wr, 632 op/s 2024-03-19T01:24:21.545 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 695.0 s, 1447.6 tps, lat 69.577 ms stddev 74.764 2024-03-19T01:24:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:23 smithi162 ceph-mon[67178]: pgmap v704: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.7 MiB/s wr, 607 op/s 2024-03-19T01:24:23.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:23 smithi116 ceph-mon[62701]: pgmap v704: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.7 MiB/s wr, 607 op/s 2024-03-19T01:24:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:23 smithi156 ceph-mon[66799]: pgmap v704: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.7 MiB/s wr, 607 op/s 2024-03-19T01:24:25.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:25 smithi162 ceph-mon[67178]: pgmap v705: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.5 MiB/s wr, 566 op/s 2024-03-19T01:24:25.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:25 smithi116 ceph-mon[62701]: pgmap v705: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.5 MiB/s wr, 566 op/s 2024-03-19T01:24:25.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:25 smithi156 ceph-mon[66799]: pgmap v705: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.5 MiB/s wr, 566 op/s 2024-03-19T01:24:26.189 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.189+0000 7f3c6d834640 1 -- 172.21.15.116:0/2058262852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c68100760 msgr2=0x7f3c68100b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.189+0000 7f3c6d834640 1 --2- 172.21.15.116:0/2058262852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c68100760 0x7f3c68100b80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f3c58009900 tx=0x7f3c5802bc30 comp rx=0 tx=0).stop 2024-03-19T01:24:26.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.191+0000 7f3c6d834640 1 -- 172.21.15.116:0/2058262852 shutdown_connections 2024-03-19T01:24:26.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.191+0000 7f3c6d834640 1 --2- 172.21.15.116:0/2058262852 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c68102310 0x7f3c68108ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.191+0000 7f3c6d834640 1 --2- 172.21.15.116:0/2058262852 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c68101950 0x7f3c68101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.191+0000 7f3c6d834640 1 --2- 172.21.15.116:0/2058262852 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c68100760 0x7f3c68100b80 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.191+0000 7f3c6d834640 1 -- 172.21.15.116:0/2058262852 >> 172.21.15.116:0/2058262852 conn(0x7f3c680fbcb0 msgr2=0x7f3c680fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:26.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.191+0000 7f3c6d834640 1 -- 172.21.15.116:0/2058262852 shutdown_connections 2024-03-19T01:24:26.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.191+0000 7f3c6d834640 1 -- 172.21.15.116:0/2058262852 wait complete. 2024-03-19T01:24:26.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.192+0000 7f3c6d834640 1 Processor -- start 2024-03-19T01:24:26.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.192+0000 7f3c6d834640 1 -- start start 2024-03-19T01:24:26.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.192+0000 7f3c6d834640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c68100760 0x7f3c6810b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.192+0000 7f3c6d834640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c68101950 0x7f3c68109720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.192+0000 7f3c6d834640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c68102310 0x7f3c68109c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.192+0000 7f3c6d834640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6810d560 con 0x7f3c68100760 2024-03-19T01:24:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.192+0000 7f3c6d834640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6810d3e0 con 0x7f3c68102310 2024-03-19T01:24:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.192+0000 7f3c6d834640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6810d6e0 con 0x7f3c68101950 2024-03-19T01:24:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.192+0000 7f3c66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c68100760 0x7f3c6810b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.192+0000 7f3c66ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c68100760 0x7f3c6810b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35782/0 (socket says 172.21.15.116:35782) 2024-03-19T01:24:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.192+0000 7f3c66ffd640 1 -- 172.21.15.116:0/1914427497 learned_addr learned my addr 172.21.15.116:0/1914427497 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.192+0000 7f3c677fe640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c68102310 0x7f3c68109c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.192+0000 7f3c5ffff640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c68101950 0x7f3c68109720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.193+0000 7f3c66ffd640 1 -- 172.21.15.116:0/1914427497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c68101950 msgr2=0x7f3c68109720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.193+0000 7f3c66ffd640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c68101950 0x7f3c68109720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.193+0000 7f3c66ffd640 1 -- 172.21.15.116:0/1914427497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c68102310 msgr2=0x7f3c68109c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.195 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.193+0000 7f3c66ffd640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c68102310 0x7f3c68109c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.193+0000 7f3c66ffd640 1 -- 172.21.15.116:0/1914427497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6810a4f0 con 0x7f3c68100760 2024-03-19T01:24:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.193+0000 7f3c5ffff640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c68101950 0x7f3c68109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.193+0000 7f3c66ffd640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c68100760 0x7f3c6810b0e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3c58000c00 tx=0x7f3c5802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.193+0000 7f3c6c832640 1 -- 172.21.15.116:0/1914427497 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c58043020 con 0x7f3c68100760 2024-03-19T01:24:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.193+0000 7f3c6d834640 1 -- 172.21.15.116:0/1914427497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c681a1c70 con 0x7f3c68100760 2024-03-19T01:24:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.193+0000 7f3c6c832640 1 -- 172.21.15.116:0/1914427497 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c580054e0 con 0x7f3c68100760 2024-03-19T01:24:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.193+0000 7f3c6c832640 1 -- 172.21.15.116:0/1914427497 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c58003c40 con 0x7f3c68100760 2024-03-19T01:24:26.196 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.193+0000 7f3c6d834640 1 -- 172.21.15.116:0/1914427497 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c681a21c0 con 0x7f3c68100760 2024-03-19T01:24:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.196+0000 7f3c6c832640 1 -- 172.21.15.116:0/1914427497 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f3c58046020 con 0x7f3c68100760 2024-03-19T01:24:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.196+0000 7f3c7041c740 1 Processor -- start 2024-03-19T01:24:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.196+0000 7f3c7041c740 1 -- start start 2024-03-19T01:24:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.197+0000 7f3c7041c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873d910 0x560e2873dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.197+0000 7f3c7041c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e2873e300 0x560e28742e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.197+0000 7f3c7041c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e28743470 0x560e28745900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.197+0000 7f3c7041c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560e28699be0 con 0x560e2873d910 2024-03-19T01:24:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.197+0000 7f3c7041c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560e28642960 con 0x560e2873e300 2024-03-19T01:24:26.197 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.197+0000 7f3c7041c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560e28695c80 con 0x560e28743470 2024-03-19T01:24:26.198 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.197+0000 7f3c677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873d910 0x560e2873dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.197+0000 7f3c66ffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e2873e300 0x560e28742e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.197+0000 7f3c677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873d910 0x560e2873dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35790/0 (socket says 172.21.15.116:35790) 2024-03-19T01:24:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.197+0000 7f3c677fe640 1 -- 172.21.15.116:0/4071542749 learned_addr learned my addr 172.21.15.116:0/4071542749 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.198+0000 7f3c677fe640 1 -- 172.21.15.116:0/4071542749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e28743470 msgr2=0x560e28745900 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:24:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.198+0000 7f3c677fe640 1 --2- 172.21.15.116:0/4071542749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e28743470 0x560e28745900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.198+0000 7f3c677fe640 1 -- 172.21.15.116:0/4071542749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e2873e300 msgr2=0x560e28742e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.198+0000 7f3c677fe640 1 --2- 172.21.15.116:0/4071542749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e2873e300 0x560e28742e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.198+0000 7f3c677fe640 1 -- 172.21.15.116:0/4071542749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e28639d20 con 0x560e2873d910 2024-03-19T01:24:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.198+0000 7f3c677fe640 1 --2- 172.21.15.116:0/4071542749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873d910 0x560e2873dd30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009f50 tx=0x7f3c600118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4018924299f39711 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.198+0000 7f3c6c832640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3c540a2920 0x7f3c540a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.198+0000 7f3c5dffb640 1 -- 172.21.15.116:0/4071542749 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c60004b20 con 0x560e2873d910 2024-03-19T01:24:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.198+0000 7f3c66ffd640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3c540a2920 0x7f3c540a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.199+0000 7f3c5dffb640 1 -- 172.21.15.116:0/4071542749 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c60004cc0 con 0x560e2873d910 2024-03-19T01:24:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.199+0000 7f3c5dffb640 1 -- 172.21.15.116:0/4071542749 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c60005020 con 0x560e2873d910 2024-03-19T01:24:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.199+0000 7f3c6c832640 1 -- 172.21.15.116:0/1914427497 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f3c580f0b30 con 0x7f3c68100760 2024-03-19T01:24:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.200+0000 7f3c66ffd640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3c540a2920 0x7f3c540a4de0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3c580bd3e0 tx=0x7f3c580f44f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.200+0000 7f3c7041c740 1 -- 172.21.15.116:0/4071542749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873d910 msgr2=0x560e2873dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.200+0000 7f3c7041c740 1 --2- 172.21.15.116:0/4071542749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873d910 0x560e2873dd30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009f50 tx=0x7f3c600118d0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.200+0000 7f3c7041c740 1 -- 172.21.15.116:0/4071542749 shutdown_connections 2024-03-19T01:24:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.200+0000 7f3c7041c740 1 --2- 172.21.15.116:0/4071542749 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e28743470 0x560e28745900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.200+0000 7f3c7041c740 1 --2- 172.21.15.116:0/4071542749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e2873e300 0x560e28742e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.200+0000 7f3c7041c740 1 --2- 172.21.15.116:0/4071542749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873d910 0x560e2873dd30 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.204 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.200+0000 7f3c7041c740 1 -- 172.21.15.116:0/4071542749 >> 172.21.15.116:0/4071542749 conn(0x560e28730670 msgr2=0x560e28730a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.200+0000 7f3c7041c740 1 -- 172.21.15.116:0/4071542749 shutdown_connections 2024-03-19T01:24:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.200+0000 7f3c7041c740 1 -- 172.21.15.116:0/4071542749 wait complete. 2024-03-19T01:24:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.200+0000 7f3c7041c740 1 -- start start 2024-03-19T01:24:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c7041c740 1 Processor -- start 2024-03-19T01:24:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c7041c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e2873d910 0x560e287478e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c7041c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 0x560e28747e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c7041c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e28743470 0x560e2874ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c7041c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560e28746470 con 0x560e2873e300 2024-03-19T01:24:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c7041c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560e28655db0 con 0x560e2873d910 2024-03-19T01:24:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c7041c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560e287465f0 con 0x560e28743470 2024-03-19T01:24:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c5ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 0x560e28747e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c7041c740 10 client.0 _ll_register_callbacks cb 0x560e2865ae70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:24:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c5ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 0x560e28747e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35798/0 (socket says 172.21.15.116:35798) 2024-03-19T01:24:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c677fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e2873d910 0x560e287478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c5ffff640 1 -- 172.21.15.116:0/3434494541 learned_addr learned my addr 172.21.15.116:0/3434494541 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c677fe640 1 --2- 172.21.15.116:0/3434494541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e28743470 0x560e2874ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c5ffff640 1 -- 172.21.15.116:0/3434494541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e28743470 msgr2=0x560e2874ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c5ffff640 1 --2- 172.21.15.116:0/3434494541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e28743470 0x560e2874ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c5ffff640 1 -- 172.21.15.116:0/3434494541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e2873d910 msgr2=0x560e287478e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c5ffff640 1 --2- 172.21.15.116:0/3434494541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e2873d910 0x560e287478e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c5ffff640 1 -- 172.21.15.116:0/3434494541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e28654c60 con 0x560e2873e300 2024-03-19T01:24:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.201+0000 7f3c677fe640 1 --2- 172.21.15.116:0/3434494541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e2873d910 0x560e287478e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:24:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.202+0000 7f3c5ffff640 1 --2- 172.21.15.116:0/3434494541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 0x560e28747e20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3c50011480 tx=0x7f3c50011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76605b8a86b41734 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.202+0000 7f3c46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:24:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.202+0000 7f3c46ffd640 1 -- 172.21.15.116:0/3434494541 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c50012980 con 0x560e2873e300 2024-03-19T01:24:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.202+0000 7f3c46ffd640 1 -- 172.21.15.116:0/3434494541 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f3c50012b20 con 0x560e2873e300 2024-03-19T01:24:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.202+0000 7f3c7041c740 1 -- 172.21.15.116:0/3434494541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c60004cc0 con 0x560e2873e300 2024-03-19T01:24:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.202+0000 7f3c46ffd640 1 -- 172.21.15.116:0/3434494541 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f3c50012d20 con 0x560e2873e300 2024-03-19T01:24:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.202+0000 7f3c7041c740 10 client.16191 fetch_fsmap learned FSMap version 47 2024-03-19T01:24:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.202+0000 7f3c7041c740 1 -- 172.21.15.116:0/3434494541 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x560e283ae210 con 0x560e2873e300 2024-03-19T01:24:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.202+0000 7f3c46ffd640 1 -- 172.21.15.116:0/3434494541 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c60004cc0 con 0x560e2873e300 2024-03-19T01:24:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.202+0000 7f3c46ffd640 1 -- 172.21.15.116:0/3434494541 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f3c50024570 con 0x560e2873e300 2024-03-19T01:24:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.203+0000 7f3c7041c740 10 client.16191 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:24:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.203+0000 7f3c7041c740 10 client.16191 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:24:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.203+0000 7f3c7041c740 20 client.16191 populate_metadata read hostname 'smithi116' 2024-03-19T01:24:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.203+0000 7f3c7041c740 1 --2- 172.21.15.116:0/3434494541 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560e287537c0 0x560e28755c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.203+0000 7f3c7041c740 4 client.16191 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:24:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.203+0000 7f3c7041c740 1 -- 172.21.15.116:0/3434494541 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e286a7120 con 0x560e287537c0 2024-03-19T01:24:26.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.204+0000 7f3c677fe640 1 --2- 172.21.15.116:0/3434494541 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560e287537c0 0x560e28755c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.205+0000 7f3c46ffd640 10 client.16191 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.205+0000 7f3c677fe640 1 --2- 172.21.15.116:0/3434494541 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560e287537c0 0x560e28755c20 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=760582c0ef825018 server_cookie=618c24fa69704a91 in_seq=0 out_seq=0 2024-03-19T01:24:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.205+0000 7f3c46ffd640 1 -- 172.21.15.116:0/3434494541 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560e286a7120 con 0x560e287537c0 2024-03-19T01:24:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.205+0000 7f3c46ffd640 10 client.16191 handle_command_reply: tid=0 2024-03-19T01:24:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.205+0000 7f3c7041c740 1 client.16191 shutdown 2024-03-19T01:24:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.206+0000 7f3c7041c740 1 -- 172.21.15.116:0/3434494541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 msgr2=0x560e28747e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.206+0000 7f3c7041c740 1 --2- 172.21.15.116:0/3434494541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 0x560e28747e20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3c50011480 tx=0x7f3c50011940 comp rx=0 tx=0).stop 2024-03-19T01:24:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.206+0000 7f3c7041c740 1 -- 172.21.15.116:0/3434494541 shutdown_connections 2024-03-19T01:24:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.206+0000 7f3c7041c740 1 --2- 172.21.15.116:0/3434494541 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560e287537c0 0x560e28755c20 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.206+0000 7f3c7041c740 1 --2- 172.21.15.116:0/3434494541 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e28743470 0x560e2874ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.206+0000 7f3c7041c740 1 --2- 172.21.15.116:0/3434494541 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 0x560e28747e20 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.206+0000 7f3c46ffd640 0 client.16191 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:26.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.206+0000 7f3c7041c740 1 --2- 172.21.15.116:0/3434494541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e2873d910 0x560e287478e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.206+0000 7f3c7041c740 1 -- 172.21.15.116:0/3434494541 >> 172.21.15.116:0/3434494541 conn(0x560e28730670 msgr2=0x560e287318c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:26.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.206+0000 7f3c7041c740 1 -- 172.21.15.116:0/3434494541 shutdown_connections 2024-03-19T01:24:26.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.206+0000 7f3c7041c740 1 -- 172.21.15.116:0/3434494541 wait complete. 2024-03-19T01:24:26.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.206+0000 7f3c7041c740 20 client.16191 trim_cache size 0 max 16384 2024-03-19T01:24:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.227+0000 7f3c7041c740 1 Processor -- start 2024-03-19T01:24:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.227+0000 7f3c7041c740 1 -- start start 2024-03-19T01:24:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.227+0000 7f3c7041c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e2873d910 0x560e28746f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.227+0000 7f3c7041c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 0x560e2874ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.227+0000 7f3c7041c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e28743470 0x560e287478e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.227+0000 7f3c7041c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3c681a6be0 con 0x560e2873e300 2024-03-19T01:24:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.227+0000 7f3c7041c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3c681a87e0 con 0x560e28743470 2024-03-19T01:24:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.227+0000 7f3c7041c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3c681a72b0 con 0x560e2873d910 2024-03-19T01:24:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c5ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 0x560e2874ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c677fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e2873d910 0x560e28746f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c5ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 0x560e2874ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35812/0 (socket says 172.21.15.116:35812) 2024-03-19T01:24:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c5ffff640 1 -- 172.21.15.116:0/1970808484 learned_addr learned my addr 172.21.15.116:0/1970808484 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c677fe640 1 --2- 172.21.15.116:0/1970808484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e28743470 0x560e287478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c5ffff640 1 -- 172.21.15.116:0/1970808484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e2873d910 msgr2=0x560e28746f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c5ffff640 1 --2- 172.21.15.116:0/1970808484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e2873d910 0x560e28746f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c5ffff640 1 -- 172.21.15.116:0/1970808484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e28743470 msgr2=0x560e287478e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c5ffff640 1 --2- 172.21.15.116:0/1970808484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e28743470 0x560e287478e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c5ffff640 1 -- 172.21.15.116:0/1970808484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e2847a620 con 0x560e2873e300 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c677fe640 1 --2- 172.21.15.116:0/1970808484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e2873d910 0x560e28746f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c5ffff640 1 --2- 172.21.15.116:0/1970808484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 0x560e2874ee20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3c50024ab0 tx=0x7f3c50009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea505a03594d91ff server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c5d7fa640 1 -- 172.21.15.116:0/1970808484 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c50029040 con 0x560e2873e300 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c5d7fa640 1 -- 172.21.15.116:0/1970808484 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c500243b0 con 0x560e2873e300 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.228+0000 7f3c5d7fa640 1 -- 172.21.15.116:0/1970808484 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c500053d0 con 0x560e2873e300 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.229+0000 7f3c7041c740 1 -- 172.21.15.116:0/1970808484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 msgr2=0x560e2874ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.229+0000 7f3c7041c740 1 --2- 172.21.15.116:0/1970808484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 0x560e2874ee20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3c50024ab0 tx=0x7f3c50009d00 comp rx=0 tx=0).stop 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.229+0000 7f3c7041c740 1 -- 172.21.15.116:0/1970808484 shutdown_connections 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.229+0000 7f3c7041c740 1 --2- 172.21.15.116:0/1970808484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e28743470 0x560e287478e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.229+0000 7f3c7041c740 1 --2- 172.21.15.116:0/1970808484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873e300 0x560e2874ee20 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.229+0000 7f3c7041c740 1 --2- 172.21.15.116:0/1970808484 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e2873d910 0x560e28746f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.229+0000 7f3c7041c740 1 -- 172.21.15.116:0/1970808484 >> 172.21.15.116:0/1970808484 conn(0x560e28730670 msgr2=0x560e28730a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.229+0000 7f3c7041c740 1 -- 172.21.15.116:0/1970808484 shutdown_connections 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.229+0000 7f3c7041c740 1 -- 172.21.15.116:0/1970808484 wait complete. 2024-03-19T01:24:26.233 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.229+0000 7f3c7041c740 1 -- start start 2024-03-19T01:24:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.230+0000 7f3c7041c740 1 Processor -- start 2024-03-19T01:24:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.230+0000 7f3c7041c740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873d910 0x560e2874c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.230+0000 7f3c7041c740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e2873e300 0x560e2874c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.230+0000 7f3c7041c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e28743470 0x560e2874d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.230+0000 7f3c7041c740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3c681a8110 con 0x560e2873d910 2024-03-19T01:24:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.230+0000 7f3c7041c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3c681a7a40 con 0x560e28743470 2024-03-19T01:24:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.230+0000 7f3c7041c740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3c681a4bc0 con 0x560e2873e300 2024-03-19T01:24:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.230+0000 7f3c7041c740 10 client.0 _ll_register_callbacks cb 0x7f3c50009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:24:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.230+0000 7f3c5ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e2873e300 0x560e2874c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.231+0000 7f3c5ffff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e2873e300 0x560e2874c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55682/0 (socket says 172.21.15.116:55682) 2024-03-19T01:24:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.231+0000 7f3c5ffff640 1 -- 172.21.15.116:0/1582860996 learned_addr learned my addr 172.21.15.116:0/1582860996 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:26.234 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.231+0000 7f3c677fe640 1 --2- 172.21.15.116:0/1582860996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873d910 0x560e2874c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.235 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.231+0000 7f3c677fe640 1 --2- 172.21.15.116:0/1582860996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e28743470 0x560e2874d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.231+0000 7f3c677fe640 1 -- 172.21.15.116:0/1582860996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e2873e300 msgr2=0x560e2874c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.231+0000 7f3c677fe640 1 --2- 172.21.15.116:0/1582860996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e2873e300 0x560e2874c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.231+0000 7f3c677fe640 1 -- 172.21.15.116:0/1582860996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873d910 msgr2=0x560e2874c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:26.236 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.231+0000 7f3c677fe640 1 --2- 172.21.15.116:0/1582860996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873d910 0x560e2874c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.231+0000 7f3c677fe640 1 -- 172.21.15.116:0/1582860996 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e2874dbe0 con 0x560e28743470 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.232+0000 7f3c677fe640 1 --2- 172.21.15.116:0/1582860996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e28743470 0x560e2874d5f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3c60012fd0 tx=0x7f3c600096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f579a146b2b3c175 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.232+0000 7f3c46ffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.232+0000 7f3c46ffd640 1 -- 172.21.15.116:0/1582860996 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c60003970 con 0x560e28743470 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.232+0000 7f3c46ffd640 1 -- 172.21.15.116:0/1582860996 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f3c60016040 con 0x560e28743470 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.232+0000 7f3c7041c740 1 -- 172.21.15.116:0/1582860996 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c500243b0 con 0x560e28743470 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.232+0000 7f3c46ffd640 1 -- 172.21.15.116:0/1582860996 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f3c60003b10 con 0x560e28743470 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.232+0000 7f3c7041c740 10 client.26048 fetch_fsmap learned FSMap version 47 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.232+0000 7f3c7041c740 1 -- 172.21.15.116:0/1582860996 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x560e2874eef0 con 0x560e28743470 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.232+0000 7f3c46ffd640 1 -- 172.21.15.116:0/1582860996 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c500243b0 con 0x560e28743470 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.233+0000 7f3c46ffd640 1 -- 172.21.15.116:0/1582860996 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f3c6003a440 con 0x560e28743470 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.233+0000 7f3c7041c740 10 client.26048 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.233+0000 7f3c7041c740 10 client.26048 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:24:26.237 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.233+0000 7f3c7041c740 20 client.26048 populate_metadata read hostname 'smithi116' 2024-03-19T01:24:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.233+0000 7f3c7041c740 1 --2- 172.21.15.116:0/1582860996 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560e287537c0 0x560e287ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.233+0000 7f3c7041c740 4 client.26048 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:24:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.233+0000 7f3c7041c740 1 -- 172.21.15.116:0/1582860996 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560e286a1470 con 0x560e287537c0 2024-03-19T01:24:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.233+0000 7f3c5ffff640 1 --2- 172.21.15.116:0/1582860996 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560e287537c0 0x560e287ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.234+0000 7f3c46ffd640 10 client.26048 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:26.238 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.234+0000 7f3c5ffff640 1 --2- 172.21.15.116:0/1582860996 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560e287537c0 0x560e287ccf70 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e01e87766831bf2 server_cookie=42d995daaa956bd4 in_seq=0 out_seq=0 2024-03-19T01:24:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c46ffd640 1 -- 172.21.15.116:0/1582860996 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560e286a1470 con 0x560e287537c0 2024-03-19T01:24:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c46ffd640 10 client.26048 handle_command_reply: tid=0 2024-03-19T01:24:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c7041c740 1 client.26048 shutdown 2024-03-19T01:24:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c7041c740 1 -- 172.21.15.116:0/1582860996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e28743470 msgr2=0x560e2874d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c7041c740 1 --2- 172.21.15.116:0/1582860996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e28743470 0x560e2874d5f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3c60012fd0 tx=0x7f3c600096c0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c7041c740 1 -- 172.21.15.116:0/1582860996 shutdown_connections 2024-03-19T01:24:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c7041c740 1 --2- 172.21.15.116:0/1582860996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560e28743470 0x560e2874d5f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c7041c740 1 --2- 172.21.15.116:0/1582860996 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560e287537c0 0x560e287ccf70 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.244 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c7041c740 1 --2- 172.21.15.116:0/1582860996 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560e2873e300 0x560e2874c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c46ffd640 0 client.26048 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c7041c740 1 --2- 172.21.15.116:0/1582860996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560e2873d910 0x560e2874c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c7041c740 1 -- 172.21.15.116:0/1582860996 >> 172.21.15.116:0/1582860996 conn(0x560e28730670 msgr2=0x560e287478c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c7041c740 1 -- 172.21.15.116:0/1582860996 shutdown_connections 2024-03-19T01:24:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c7041c740 1 -- 172.21.15.116:0/1582860996 wait complete. 2024-03-19T01:24:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.243+0000 7f3c7041c740 20 client.26048 trim_cache size 0 max 16384 2024-03-19T01:24:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:24:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:24:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:24:26.245 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:24:26.246 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.244+0000 7f3c6d834640 1 -- 172.21.15.116:0/1914427497 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3c540a2920 msgr2=0x7f3c540a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.244+0000 7f3c6d834640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3c540a2920 0x7f3c540a4de0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3c580bd3e0 tx=0x7f3c580f44f0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.244+0000 7f3c6d834640 1 -- 172.21.15.116:0/1914427497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c68100760 msgr2=0x7f3c6810b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.244+0000 7f3c6d834640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c68100760 0x7f3c6810b0e0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f3c58000c00 tx=0x7f3c5802c650 comp rx=0 tx=0).stop 2024-03-19T01:24:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.244+0000 7f3c6d834640 1 -- 172.21.15.116:0/1914427497 shutdown_connections 2024-03-19T01:24:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.244+0000 7f3c6d834640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3c540a2920 0x7f3c540a4de0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.244+0000 7f3c6d834640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3c68102310 0x7f3c68109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.244+0000 7f3c6d834640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3c68101950 0x7f3c68109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.244+0000 7f3c6d834640 1 --2- 172.21.15.116:0/1914427497 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c68100760 0x7f3c6810b0e0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.244+0000 7f3c6d834640 1 -- 172.21.15.116:0/1914427497 >> 172.21.15.116:0/1914427497 conn(0x7f3c680fbcb0 msgr2=0x7f3c68104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.244+0000 7f3c6d834640 1 -- 172.21.15.116:0/1914427497 shutdown_connections 2024-03-19T01:24:26.247 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:26.244+0000 7f3c6d834640 1 -- 172.21.15.116:0/1914427497 wait complete. 2024-03-19T01:24:26.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 700.0 s, 1974.0 tps, lat 50.135 ms stddev 58.295 2024-03-19T01:24:27.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:27 smithi162 ceph-mon[67178]: pgmap v706: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 MiB/s wr, 669 op/s 2024-03-19T01:24:27.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:27 smithi116 ceph-mon[62701]: pgmap v706: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 MiB/s wr, 669 op/s 2024-03-19T01:24:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:27 smithi156 ceph-mon[66799]: pgmap v706: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 MiB/s wr, 669 op/s 2024-03-19T01:24:29.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:29 smithi162 ceph-mon[67178]: pgmap v707: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.7 MiB/s wr, 596 op/s 2024-03-19T01:24:29.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:29 smithi116 ceph-mon[62701]: pgmap v707: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.7 MiB/s wr, 596 op/s 2024-03-19T01:24:29.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:29 smithi156 ceph-mon[66799]: pgmap v707: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.7 MiB/s wr, 596 op/s 2024-03-19T01:24:31.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:31 smithi162 ceph-mon[67178]: pgmap v708: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.4 MiB/s wr, 581 op/s 2024-03-19T01:24:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:31 smithi116 ceph-mon[62701]: pgmap v708: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.4 MiB/s wr, 581 op/s 2024-03-19T01:24:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:31 smithi156 ceph-mon[66799]: pgmap v708: 97 pgs: 97 active+clean; 1.6 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.4 MiB/s wr, 581 op/s 2024-03-19T01:24:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 705.0 s, 1593.4 tps, lat 63.479 ms stddev 70.993 2024-03-19T01:24:33.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:33 smithi162 ceph-mon[67178]: pgmap v709: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.0 MiB/s wr, 620 op/s 2024-03-19T01:24:33.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:33 smithi116 ceph-mon[62701]: pgmap v709: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.0 MiB/s wr, 620 op/s 2024-03-19T01:24:33.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:33 smithi156 ceph-mon[66799]: pgmap v709: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.0 MiB/s wr, 620 op/s 2024-03-19T01:24:35.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:35 smithi116 ceph-mon[62701]: pgmap v710: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.9 MiB/s wr, 590 op/s 2024-03-19T01:24:35.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:35 smithi156 ceph-mon[66799]: pgmap v710: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.9 MiB/s wr, 590 op/s 2024-03-19T01:24:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:35 smithi162 ceph-mon[67178]: pgmap v710: 97 pgs: 97 active+clean; 1.6 GiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.9 MiB/s wr, 590 op/s 2024-03-19T01:24:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 710.0 s, 2422.4 tps, lat 40.942 ms stddev 49.409 2024-03-19T01:24:37.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:37 smithi156 ceph-mon[66799]: pgmap v711: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.7 MiB/s wr, 732 op/s 2024-03-19T01:24:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:37 smithi162 ceph-mon[67178]: pgmap v711: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.7 MiB/s wr, 732 op/s 2024-03-19T01:24:37.656 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:37 smithi116 ceph-mon[62701]: pgmap v711: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.7 MiB/s wr, 732 op/s 2024-03-19T01:24:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.382+0000 7fbc9e3f3640 1 -- 172.21.15.116:0/874601131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc981022f0 msgr2=0x7fbc98108b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.382+0000 7fbc9e3f3640 1 --2- 172.21.15.116:0/874601131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc981022f0 0x7fbc98108b80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009be0 tx=0x7fbc9002bef0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.382+0000 7fbc9e3f3640 1 -- 172.21.15.116:0/874601131 shutdown_connections 2024-03-19T01:24:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.382+0000 7fbc9e3f3640 1 --2- 172.21.15.116:0/874601131 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc981022f0 0x7fbc98108b80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.382+0000 7fbc9e3f3640 1 --2- 172.21.15.116:0/874601131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc98101930 0x7fbc98101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.382+0000 7fbc9e3f3640 1 --2- 172.21.15.116:0/874601131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc98100740 0x7fbc98100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.382+0000 7fbc9e3f3640 1 -- 172.21.15.116:0/874601131 >> 172.21.15.116:0/874601131 conn(0x7fbc980fbcb0 msgr2=0x7fbc980fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.382+0000 7fbc9e3f3640 1 -- 172.21.15.116:0/874601131 shutdown_connections 2024-03-19T01:24:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.382+0000 7fbc9e3f3640 1 -- 172.21.15.116:0/874601131 wait complete. 2024-03-19T01:24:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.383+0000 7fbc9e3f3640 1 Processor -- start 2024-03-19T01:24:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.383+0000 7fbc9e3f3640 1 -- start start 2024-03-19T01:24:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.383+0000 7fbc9e3f3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc98100740 0x7fbc98196960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.383+0000 7fbc9e3f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc98101930 0x7fbc98196ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.383+0000 7fbc9e3f3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc981022f0 0x7fbc9819df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.383+0000 7fbc9e3f3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9810c3a0 con 0x7fbc98101930 2024-03-19T01:24:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.383+0000 7fbc9e3f3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9810c220 con 0x7fbc981022f0 2024-03-19T01:24:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.383+0000 7fbc9e3f3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9810c520 con 0x7fbc98100740 2024-03-19T01:24:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc98101930 0x7fbc98196ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc97fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc98100740 0x7fbc98196960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc97fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc98100740 0x7fbc98196960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:32884/0 (socket says 172.21.15.116:32884) 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc98101930 0x7fbc98196ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58962/0 (socket says 172.21.15.116:58962) 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc97fff640 1 -- 172.21.15.116:0/2345391898 learned_addr learned my addr 172.21.15.116:0/2345391898 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc9c990640 1 --2- 172.21.15.116:0/2345391898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc981022f0 0x7fbc9819df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc97fff640 1 -- 172.21.15.116:0/2345391898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc981022f0 msgr2=0x7fbc9819df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc97fff640 1 --2- 172.21.15.116:0/2345391898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc981022f0 0x7fbc9819df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc97fff640 1 -- 172.21.15.116:0/2345391898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc98101930 msgr2=0x7fbc98196ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc97fff640 1 --2- 172.21.15.116:0/2345391898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc98101930 0x7fbc98196ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc97fff640 1 -- 172.21.15.116:0/2345391898 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9819e580 con 0x7fbc98100740 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc97fff640 1 --2- 172.21.15.116:0/2345391898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc98100740 0x7fbc98196960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800e9f0 tx=0x7fbc8800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc957fa640 1 -- 172.21.15.116:0/2345391898 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8800ccc0 con 0x7fbc98100740 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc9e3f3640 1 -- 172.21.15.116:0/2345391898 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc9819e870 con 0x7fbc98100740 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc957fa640 1 -- 172.21.15.116:0/2345391898 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc880035f0 con 0x7fbc98100740 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.384+0000 7fbc9e3f3640 1 -- 172.21.15.116:0/2345391898 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc9819ee10 con 0x7fbc98100740 2024-03-19T01:24:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.385+0000 7fbc957fa640 1 -- 172.21.15.116:0/2345391898 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8800ce60 con 0x7fbc98100740 2024-03-19T01:24:39.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.388+0000 7fbc957fa640 1 -- 172.21.15.116:0/2345391898 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fbc880037a0 con 0x7fbc98100740 2024-03-19T01:24:39.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.388+0000 7fbca0fd9740 1 Processor -- start 2024-03-19T01:24:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.388+0000 7fbca0fd9740 1 -- start start 2024-03-19T01:24:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.388+0000 7fbca0fd9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d228f00 0x56282d229320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.388+0000 7fbca0fd9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d22de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.388+0000 7fbca0fd9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d22e4f0 0x56282d230960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.388+0000 7fbca0fd9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56282d184be0 con 0x56282d228f00 2024-03-19T01:24:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.388+0000 7fbca0fd9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56282d12d960 con 0x56282d2298f0 2024-03-19T01:24:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.388+0000 7fbca0fd9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56282d180c80 con 0x56282d22e4f0 2024-03-19T01:24:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.388+0000 7fbc9c990640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d228f00 0x56282d229320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.388+0000 7fbc9c990640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d228f00 0x56282d229320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58964/0 (socket says 172.21.15.116:58964) 2024-03-19T01:24:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.389+0000 7fbc9c990640 1 -- 172.21.15.116:0/2839572285 learned_addr learned my addr 172.21.15.116:0/2839572285 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.389+0000 7fbc957fa640 1 --2- 172.21.15.116:0/2345391898 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fbc800a28a0 0x7fbc800a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.389+0000 7fbc977fe640 1 --2- 172.21.15.116:0/2839572285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d22e4f0 0x56282d230960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.389+0000 7fbc957fa640 1 -- 172.21.15.116:0/2345391898 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbc880996c0 con 0x7fbc98100740 2024-03-19T01:24:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.389+0000 7fbc9c990640 1 --2- 172.21.15.116:0/2345391898 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fbc800a28a0 0x7fbc800a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.390+0000 7fbc9c990640 1 -- 172.21.15.116:0/2839572285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d22e4f0 msgr2=0x56282d230960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.390+0000 7fbc9c990640 1 --2- 172.21.15.116:0/2839572285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d22e4f0 0x56282d230960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.390+0000 7fbc97fff640 1 --2- 172.21.15.116:0/2839572285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d22de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.390+0000 7fbc9c990640 1 -- 172.21.15.116:0/2839572285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 msgr2=0x56282d22de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.390+0000 7fbc9c990640 1 --2- 172.21.15.116:0/2839572285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d22de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.390+0000 7fbc9c990640 1 -- 172.21.15.116:0/2839572285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56282d124d20 con 0x56282d228f00 2024-03-19T01:24:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.390+0000 7fbc977fe640 1 --2- 172.21.15.116:0/2839572285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d22e4f0 0x56282d230960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:24:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.390+0000 7fbc97fff640 1 --2- 172.21.15.116:0/2839572285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d22de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.390+0000 7fbc9c990640 1 --2- 172.21.15.116:0/2839572285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d228f00 0x56282d229320 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbc90004ab0 tx=0x7fbc90033ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae14b5beeb13000 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.390+0000 7fbc7e7fc640 1 -- 172.21.15.116:0/2839572285 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc900337a0 con 0x56282d228f00 2024-03-19T01:24:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.391+0000 7fbc9c990640 1 --2- 172.21.15.116:0/2345391898 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fbc800a28a0 0x7fbc800a4d60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbc90048000 tx=0x7fbc9004a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:39.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.391+0000 7fbc7e7fc640 1 -- 172.21.15.116:0/2839572285 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc9002cc60 con 0x56282d228f00 2024-03-19T01:24:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.391+0000 7fbc7e7fc640 1 -- 172.21.15.116:0/2839572285 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc90034690 con 0x56282d228f00 2024-03-19T01:24:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.392+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2839572285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d228f00 msgr2=0x56282d229320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:39.394 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.392+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2839572285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d228f00 0x56282d229320 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fbc90004ab0 tx=0x7fbc90033ee0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.392+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2839572285 shutdown_connections 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.392+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2839572285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d22e4f0 0x56282d230960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.392+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2839572285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d22de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.392+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2839572285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d228f00 0x56282d229320 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.392+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2839572285 >> 172.21.15.116:0/2839572285 conn(0x56282d21b5f0 msgr2=0x56282d21b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.392+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2839572285 shutdown_connections 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.392+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2839572285 wait complete. 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.392+0000 7fbca0fd9740 1 -- start start 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbca0fd9740 1 Processor -- start 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbca0fd9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d228f00 0x56282d2438c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbca0fd9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d243e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbca0fd9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d24ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbca0fd9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56282d2423d0 con 0x56282d22e4f0 2024-03-19T01:24:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbca0fd9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56282d140db0 con 0x56282d2298f0 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbca0fd9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56282d242550 con 0x56282d228f00 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbca0fd9740 10 client.0 _ll_register_callbacks cb 0x56282d145e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbc97fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d228f00 0x56282d2438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbc977fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d243e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbc977fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d243e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:52402/0 (socket says 172.21.15.116:52402) 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbc977fe640 1 -- 172.21.15.116:0/2081839120 learned_addr learned my addr 172.21.15.116:0/2081839120 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbc9c990640 1 --2- 172.21.15.116:0/2081839120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d24ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbc977fe640 1 -- 172.21.15.116:0/2081839120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d228f00 msgr2=0x56282d2438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbc977fe640 1 --2- 172.21.15.116:0/2081839120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d228f00 0x56282d2438c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbc977fe640 1 -- 172.21.15.116:0/2081839120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 msgr2=0x56282d24ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbc977fe640 1 --2- 172.21.15.116:0/2081839120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d24ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbc977fe640 1 -- 172.21.15.116:0/2081839120 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56282d13fc60 con 0x56282d2298f0 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.393+0000 7fbc9c990640 1 --2- 172.21.15.116:0/2081839120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d24ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.394+0000 7fbc977fe640 1 --2- 172.21.15.116:0/2081839120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d243e00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c012720 tx=0x7fbc8c012be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5c1fc5bee3961ef server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.394+0000 7fbc777fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:24:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.394+0000 7fbc777fe640 1 -- 172.21.15.116:0/2081839120 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c004050 con 0x56282d2298f0 2024-03-19T01:24:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.394+0000 7fbc777fe640 1 -- 172.21.15.116:0/2081839120 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbc8c0041f0 con 0x56282d2298f0 2024-03-19T01:24:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.394+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2081839120 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc9002cc60 con 0x56282d2298f0 2024-03-19T01:24:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.394+0000 7fbc777fe640 1 -- 172.21.15.116:0/2081839120 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fbc8c0043f0 con 0x56282d2298f0 2024-03-19T01:24:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.394+0000 7fbca0fd9740 10 client.26057 fetch_fsmap learned FSMap version 47 2024-03-19T01:24:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.394+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2081839120 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x56282ce99210 con 0x56282d2298f0 2024-03-19T01:24:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.395+0000 7fbc777fe640 1 -- 172.21.15.116:0/2081839120 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc9002cc60 con 0x56282d2298f0 2024-03-19T01:24:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.395+0000 7fbc777fe640 1 -- 172.21.15.116:0/2081839120 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7fbc8c003ca0 con 0x56282d2298f0 2024-03-19T01:24:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.395+0000 7fbca0fd9740 10 client.26057 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:24:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.395+0000 7fbca0fd9740 10 client.26057 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:24:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.395+0000 7fbca0fd9740 20 client.26057 populate_metadata read hostname 'smithi116' 2024-03-19T01:24:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.395+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2081839120 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56282d24f640 0x56282d251a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.395+0000 7fbca0fd9740 4 client.26057 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:24:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.395+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2081839120 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56282d192120 con 0x56282d24f640 2024-03-19T01:24:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.395+0000 7fbc97fff640 1 --2- 172.21.15.116:0/2081839120 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56282d24f640 0x56282d251a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.396+0000 7fbc777fe640 10 client.26057 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.396+0000 7fbc97fff640 1 --2- 172.21.15.116:0/2081839120 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56282d24f640 0x56282d251a90 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12cf602f8e2a712d server_cookie=5e42cba56d85f5a0 in_seq=0 out_seq=0 2024-03-19T01:24:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.396+0000 7fbc777fe640 1 -- 172.21.15.116:0/2081839120 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56282d192120 con 0x56282d24f640 2024-03-19T01:24:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.396+0000 7fbc777fe640 10 client.26057 handle_command_reply: tid=0 2024-03-19T01:24:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.396+0000 7fbca0fd9740 1 client.26057 shutdown 2024-03-19T01:24:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.397+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2081839120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 msgr2=0x56282d243e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.397+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2081839120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d243e00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c012720 tx=0x7fbc8c012be0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.397+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2081839120 shutdown_connections 2024-03-19T01:24:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.397+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2081839120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d24ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.397+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2081839120 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d243e00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.397+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2081839120 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56282d24f640 0x56282d251a90 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.397+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2081839120 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d228f00 0x56282d2438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.397+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2081839120 >> 172.21.15.116:0/2081839120 conn(0x56282d21b5f0 msgr2=0x56282d227d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.397+0000 7fbc777fe640 0 client.26057 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.397+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2081839120 shutdown_connections 2024-03-19T01:24:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.397+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2081839120 wait complete. 2024-03-19T01:24:39.401 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.397+0000 7fbca0fd9740 20 client.26057 trim_cache size 0 max 16384 2024-03-19T01:24:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbca0fd9740 1 Processor -- start 2024-03-19T01:24:39.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbca0fd9740 1 -- start start 2024-03-19T01:24:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbca0fd9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d228f00 0x56282d21aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbca0fd9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d2298f0 0x56282d24ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbca0fd9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d228410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbca0fd9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbc98072e30 con 0x56282d22e4f0 2024-03-19T01:24:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbca0fd9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9819ffc0 con 0x56282d228f00 2024-03-19T01:24:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbca0fd9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbc98073500 con 0x56282d2298f0 2024-03-19T01:24:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbc97fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d228f00 0x56282d21aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbc977fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d2298f0 0x56282d24ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbc9c990640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d228410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbc977fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d2298f0 0x56282d24ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:32912/0 (socket says 172.21.15.116:32912) 2024-03-19T01:24:39.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbc977fe640 1 -- 172.21.15.116:0/2363691735 learned_addr learned my addr 172.21.15.116:0/2363691735 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.416+0000 7fbc977fe640 1 -- 172.21.15.116:0/2363691735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d228f00 msgr2=0x56282d21aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.415+0000 7fbc97fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d228f00 0x56282d21aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:52418/0 (socket says 172.21.15.116:52418) 2024-03-19T01:24:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.416+0000 7fbc977fe640 1 --2- 172.21.15.116:0/2363691735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d228f00 0x56282d21aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.416+0000 7fbc977fe640 1 -- 172.21.15.116:0/2363691735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 msgr2=0x56282d228410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.416+0000 7fbc977fe640 1 --2- 172.21.15.116:0/2363691735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d228410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.416+0000 7fbc977fe640 1 -- 172.21.15.116:0/2363691735 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56282cf65620 con 0x56282d2298f0 2024-03-19T01:24:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.416+0000 7fbc9c990640 1 --2- 172.21.15.116:0/2363691735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d228410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:24:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.416+0000 7fbc977fe640 1 --2- 172.21.15.116:0/2363691735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d2298f0 0x56282d24ae00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c000c00 tx=0x7fbc8c0059b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=620acc15dd8fab86 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.416+0000 7fbc97fff640 1 --2- 172.21.15.116:0/2363691735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d228f00 0x56282d21aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:24:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.416+0000 7fbc7dffb640 1 -- 172.21.15.116:0/2363691735 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c0106a0 con 0x56282d2298f0 2024-03-19T01:24:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.416+0000 7fbc7dffb640 1 -- 172.21.15.116:0/2363691735 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc8c010840 con 0x56282d2298f0 2024-03-19T01:24:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.416+0000 7fbc7dffb640 1 -- 172.21.15.116:0/2363691735 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c0053a0 con 0x56282d2298f0 2024-03-19T01:24:39.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.417+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2363691735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d2298f0 msgr2=0x56282d24ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:39.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.417+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2363691735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d2298f0 0x56282d24ae00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c000c00 tx=0x7fbc8c0059b0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.418+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2363691735 shutdown_connections 2024-03-19T01:24:39.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.418+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2363691735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d228410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.418+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2363691735 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d2298f0 0x56282d24ae00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.418+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/2363691735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d228f00 0x56282d21aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.419+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2363691735 >> 172.21.15.116:0/2363691735 conn(0x56282d21b5f0 msgr2=0x56282d227d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.419+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2363691735 shutdown_connections 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.419+0000 7fbca0fd9740 1 -- 172.21.15.116:0/2363691735 wait complete. 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.419+0000 7fbca0fd9740 1 -- start start 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.420+0000 7fbca0fd9740 1 Processor -- start 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.420+0000 7fbca0fd9740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d228f00 0x56282d248a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.420+0000 7fbca0fd9740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d248fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.420+0000 7fbca0fd9740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d249fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.420+0000 7fbca0fd9740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9819f8f0 con 0x56282d22e4f0 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.420+0000 7fbca0fd9740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9819f220 con 0x56282d2298f0 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.420+0000 7fbca0fd9740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fbc98070e10 con 0x56282d228f00 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.420+0000 7fbc9c990640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d249fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.420+0000 7fbc9c990640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d249fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58976/0 (socket says 172.21.15.116:58976) 2024-03-19T01:24:39.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.420+0000 7fbc9c990640 1 -- 172.21.15.116:0/1562500794 learned_addr learned my addr 172.21.15.116:0/1562500794 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:39.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.420+0000 7fbc97fff640 1 --2- 172.21.15.116:0/1562500794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d228f00 0x56282d248a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.420+0000 7fbc977fe640 1 --2- 172.21.15.116:0/1562500794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d248fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.420+0000 7fbca0fd9740 10 client.0 _ll_register_callbacks cb 0x7fbc8c009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:24:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbc977fe640 1 -- 172.21.15.116:0/1562500794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d228f00 msgr2=0x56282d248a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbc977fe640 1 --2- 172.21.15.116:0/1562500794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d228f00 0x56282d248a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbc977fe640 1 -- 172.21.15.116:0/1562500794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 msgr2=0x56282d249fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbc977fe640 1 --2- 172.21.15.116:0/1562500794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d249fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbc977fe640 1 -- 172.21.15.116:0/1562500794 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56282d24a5c0 con 0x56282d2298f0 2024-03-19T01:24:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbc977fe640 1 --2- 172.21.15.116:0/1562500794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d248fc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x56282d251fa0 tx=0x7fbc8c012be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88437f75edbe916d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbc777fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:24:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbc777fe640 1 -- 172.21.15.116:0/1562500794 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c0106a0 con 0x56282d2298f0 2024-03-19T01:24:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbca0fd9740 1 -- 172.21.15.116:0/1562500794 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc8c010840 con 0x56282d2298f0 2024-03-19T01:24:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbc777fe640 1 -- 172.21.15.116:0/1562500794 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fbc8c005120 con 0x56282d2298f0 2024-03-19T01:24:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbc777fe640 1 -- 172.21.15.116:0/1562500794 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fbc8c03e070 con 0x56282d2298f0 2024-03-19T01:24:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbca0fd9740 10 client.26060 fetch_fsmap learned FSMap version 47 2024-03-19T01:24:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbca0fd9740 1 -- 172.21.15.116:0/1562500794 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x56282d2b5e10 con 0x56282d2298f0 2024-03-19T01:24:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbc777fe640 1 -- 172.21.15.116:0/1562500794 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc8c010840 con 0x56282d2298f0 2024-03-19T01:24:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.421+0000 7fbc777fe640 1 -- 172.21.15.116:0/1562500794 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7fbc8c004c00 con 0x56282d2298f0 2024-03-19T01:24:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.422+0000 7fbca0fd9740 10 client.26060 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:24:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.422+0000 7fbca0fd9740 10 client.26060 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:24:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.422+0000 7fbca0fd9740 20 client.26060 populate_metadata read hostname 'smithi116' 2024-03-19T01:24:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.422+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/1562500794 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56282d2b6790 0x56282d2b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.422+0000 7fbca0fd9740 4 client.26060 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:24:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.422+0000 7fbca0fd9740 1 -- 172.21.15.116:0/1562500794 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56282d18c470 con 0x56282d2b6790 2024-03-19T01:24:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.422+0000 7fbc97fff640 1 --2- 172.21.15.116:0/1562500794 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56282d2b6790 0x56282d2b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.422+0000 7fbc777fe640 10 client.26060 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:39.425 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.422+0000 7fbc97fff640 1 --2- 172.21.15.116:0/1562500794 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56282d2b6790 0x56282d2b8bb0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebd81cb3432ba8cb server_cookie=b369067d1ab09c5a in_seq=0 out_seq=0 2024-03-19T01:24:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.430+0000 7fbc777fe640 1 -- 172.21.15.116:0/1562500794 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56282d18c470 con 0x56282d2b6790 2024-03-19T01:24:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.430+0000 7fbc777fe640 10 client.26060 handle_command_reply: tid=0 2024-03-19T01:24:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.430+0000 7fbca0fd9740 1 client.26060 shutdown 2024-03-19T01:24:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.430+0000 7fbca0fd9740 1 -- 172.21.15.116:0/1562500794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 msgr2=0x56282d248fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.430+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/1562500794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d248fc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x56282d251fa0 tx=0x7fbc8c012be0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.430+0000 7fbca0fd9740 1 -- 172.21.15.116:0/1562500794 shutdown_connections 2024-03-19T01:24:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.430+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/1562500794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56282d22e4f0 0x56282d249fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.430+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/1562500794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56282d2298f0 0x56282d248fc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.430+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/1562500794 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56282d2b6790 0x56282d2b8bb0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.430+0000 7fbca0fd9740 1 --2- 172.21.15.116:0/1562500794 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56282d228f00 0x56282d248a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.430+0000 7fbc777fe640 0 client.26060 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:39.431 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.430+0000 7fbca0fd9740 1 -- 172.21.15.116:0/1562500794 >> 172.21.15.116:0/1562500794 conn(0x56282d21b5f0 msgr2=0x56282d228590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.430+0000 7fbca0fd9740 1 -- 172.21.15.116:0/1562500794 shutdown_connections 2024-03-19T01:24:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.431+0000 7fbca0fd9740 1 -- 172.21.15.116:0/1562500794 wait complete. 2024-03-19T01:24:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.431+0000 7fbca0fd9740 20 client.26060 trim_cache size 0 max 16384 2024-03-19T01:24:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:24:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:24:39.432 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:24:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:24:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.432+0000 7fbc9e3f3640 1 -- 172.21.15.116:0/2345391898 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fbc800a28a0 msgr2=0x7fbc800a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:39.433 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.433+0000 7fbc9e3f3640 1 --2- 172.21.15.116:0/2345391898 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fbc800a28a0 0x7fbc800a4d60 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbc90048000 tx=0x7fbc9004a000 comp rx=0 tx=0).stop 2024-03-19T01:24:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.433+0000 7fbc9e3f3640 1 -- 172.21.15.116:0/2345391898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc98100740 msgr2=0x7fbc98196960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.433+0000 7fbc9e3f3640 1 --2- 172.21.15.116:0/2345391898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc98100740 0x7fbc98196960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbc8800e9f0 tx=0x7fbc8800eeb0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.433+0000 7fbc9e3f3640 1 -- 172.21.15.116:0/2345391898 shutdown_connections 2024-03-19T01:24:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.433+0000 7fbc9e3f3640 1 --2- 172.21.15.116:0/2345391898 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fbc800a28a0 0x7fbc800a4d60 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.433+0000 7fbc9e3f3640 1 --2- 172.21.15.116:0/2345391898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc981022f0 0x7fbc9819df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.433+0000 7fbc9e3f3640 1 --2- 172.21.15.116:0/2345391898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc98101930 0x7fbc98196ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.433+0000 7fbc9e3f3640 1 --2- 172.21.15.116:0/2345391898 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fbc98100740 0x7fbc98196960 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.433+0000 7fbc9e3f3640 1 -- 172.21.15.116:0/2345391898 >> 172.21.15.116:0/2345391898 conn(0x7fbc980fbcb0 msgr2=0x7fbc98104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.433+0000 7fbc9e3f3640 1 -- 172.21.15.116:0/2345391898 shutdown_connections 2024-03-19T01:24:39.434 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:39.433+0000 7fbc9e3f3640 1 -- 172.21.15.116:0/2345391898 wait complete. 2024-03-19T01:24:39.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:39 smithi156 ceph-mon[66799]: pgmap v712: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.6 MiB/s wr, 647 op/s 2024-03-19T01:24:39.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:39 smithi162 ceph-mon[67178]: pgmap v712: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.6 MiB/s wr, 647 op/s 2024-03-19T01:24:39.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:39 smithi116 ceph-mon[62701]: pgmap v712: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 6.6 MiB/s wr, 647 op/s 2024-03-19T01:24:41.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:41 smithi156 ceph-mon[66799]: pgmap v713: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.6 MiB/s wr, 710 op/s 2024-03-19T01:24:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 715.0 s, 1676.0 tps, lat 59.852 ms stddev 65.114 2024-03-19T01:24:41.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:41 smithi162 ceph-mon[67178]: pgmap v713: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.6 MiB/s wr, 710 op/s 2024-03-19T01:24:41.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:41 smithi116 ceph-mon[62701]: pgmap v713: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.6 MiB/s wr, 710 op/s 2024-03-19T01:24:43.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:43 smithi156 ceph-mon[66799]: pgmap v714: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.0 MiB/s wr, 743 op/s 2024-03-19T01:24:43.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:43 smithi162 ceph-mon[67178]: pgmap v714: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.0 MiB/s wr, 743 op/s 2024-03-19T01:24:43.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:43 smithi116 ceph-mon[62701]: pgmap v714: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.0 MiB/s wr, 743 op/s 2024-03-19T01:24:45.450 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:45 smithi162 ceph-mon[67178]: pgmap v715: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.6 MiB/s wr, 638 op/s 2024-03-19T01:24:45.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:45 smithi156 ceph-mon[66799]: pgmap v715: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.6 MiB/s wr, 638 op/s 2024-03-19T01:24:45.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:45 smithi116 ceph-mon[62701]: pgmap v715: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 6.6 MiB/s wr, 638 op/s 2024-03-19T01:24:46.545 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 720.0 s, 1240.8 tps, lat 80.400 ms stddev 83.690 2024-03-19T01:24:47.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:47 smithi156 ceph-mon[66799]: pgmap v716: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 8.4 MiB/s wr, 784 op/s 2024-03-19T01:24:47.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:47 smithi162 ceph-mon[67178]: pgmap v716: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 8.4 MiB/s wr, 784 op/s 2024-03-19T01:24:47.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:47 smithi116 ceph-mon[62701]: pgmap v716: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 8.4 MiB/s wr, 784 op/s 2024-03-19T01:24:49.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:49 smithi156 ceph-mon[66799]: pgmap v717: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 6.8 MiB/s wr, 672 op/s 2024-03-19T01:24:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:49 smithi162 ceph-mon[67178]: pgmap v717: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 6.8 MiB/s wr, 672 op/s 2024-03-19T01:24:49.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:49 smithi116 ceph-mon[62701]: pgmap v717: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 6.8 MiB/s wr, 672 op/s 2024-03-19T01:24:51.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:51 smithi156 ceph-mon[66799]: pgmap v718: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 6.7 MiB/s wr, 668 op/s 2024-03-19T01:24:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 725.0 s, 1563.6 tps, lat 63.951 ms stddev 73.381 2024-03-19T01:24:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:51 smithi162 ceph-mon[67178]: pgmap v718: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 6.7 MiB/s wr, 668 op/s 2024-03-19T01:24:51.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:51 smithi116 ceph-mon[62701]: pgmap v718: 97 pgs: 97 active+clean; 1.6 GiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 6.7 MiB/s wr, 668 op/s 2024-03-19T01:24:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.557+0000 7f18ce344640 1 -- 172.21.15.116:0/3039111923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8101910 msgr2=0x7f18c8101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.557+0000 7f18ce344640 1 --2- 172.21.15.116:0/3039111923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8101910 0x7f18c8101d90 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f18bc009900 tx=0x7f18bc02bc30 comp rx=0 tx=0).stop 2024-03-19T01:24:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.557+0000 7f18ce344640 1 -- 172.21.15.116:0/3039111923 shutdown_connections 2024-03-19T01:24:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.557+0000 7f18ce344640 1 --2- 172.21.15.116:0/3039111923 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18c81022d0 0x7f18c8108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.557+0000 7f18ce344640 1 --2- 172.21.15.116:0/3039111923 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c8101910 0x7f18c8101d90 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.557+0000 7f18ce344640 1 --2- 172.21.15.116:0/3039111923 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18c8100720 0x7f18c8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.557+0000 7f18ce344640 1 -- 172.21.15.116:0/3039111923 >> 172.21.15.116:0/3039111923 conn(0x7f18c80fbc50 msgr2=0x7f18c80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.557+0000 7f18ce344640 1 -- 172.21.15.116:0/3039111923 shutdown_connections 2024-03-19T01:24:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.557+0000 7f18ce344640 1 -- 172.21.15.116:0/3039111923 wait complete. 2024-03-19T01:24:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.558+0000 7f18ce344640 1 Processor -- start 2024-03-19T01:24:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.558+0000 7f18ce344640 1 -- start start 2024-03-19T01:24:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.558+0000 7f18ce344640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18c8100720 0x7f18c8199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.559 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.558+0000 7f18ce344640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18c8101910 0x7f18c819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.558+0000 7f18ce344640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c81022d0 0x7f18c81a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.558+0000 7f18ce344640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f18c810b2e0 con 0x7f18c81022d0 2024-03-19T01:24:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.558+0000 7f18ce344640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f18c810b160 con 0x7f18c8101910 2024-03-19T01:24:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.558+0000 7f18ce344640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f18c810b460 con 0x7f18c8100720 2024-03-19T01:24:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.558+0000 7f18c7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18c8100720 0x7f18c8199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.558+0000 7f18cc8e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c81022d0 0x7f18c81a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.558+0000 7f18c7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18c8100720 0x7f18c8199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43056/0 (socket says 172.21.15.116:43056) 2024-03-19T01:24:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.558+0000 7f18c7fff640 1 -- 172.21.15.116:0/545399360 learned_addr learned my addr 172.21.15.116:0/545399360 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.558+0000 7f18c77fe640 1 --2- 172.21.15.116:0/545399360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18c8101910 0x7f18c819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.559+0000 7f18c77fe640 1 -- 172.21.15.116:0/545399360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18c8100720 msgr2=0x7f18c8199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.559+0000 7f18c77fe640 1 --2- 172.21.15.116:0/545399360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18c8100720 0x7f18c8199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.561 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.559+0000 7f18c77fe640 1 -- 172.21.15.116:0/545399360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c81022d0 msgr2=0x7f18c81a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.559+0000 7f18c77fe640 1 --2- 172.21.15.116:0/545399360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c81022d0 0x7f18c81a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.559+0000 7f18c77fe640 1 -- 172.21.15.116:0/545399360 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c81a18e0 con 0x7f18c8101910 2024-03-19T01:24:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.559+0000 7f18c77fe640 1 --2- 172.21.15.116:0/545399360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18c8101910 0x7f18c819a1e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f18bc036040 tx=0x7f18bc005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.559+0000 7f18c57fa640 1 -- 172.21.15.116:0/545399360 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc043020 con 0x7f18c8101910 2024-03-19T01:24:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.559+0000 7f18c57fa640 1 -- 172.21.15.116:0/545399360 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18bc0054e0 con 0x7f18c8101910 2024-03-19T01:24:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.559+0000 7f18ce344640 1 -- 172.21.15.116:0/545399360 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c81a1b70 con 0x7f18c8101910 2024-03-19T01:24:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.559+0000 7f18c57fa640 1 -- 172.21.15.116:0/545399360 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc003b30 con 0x7f18c8101910 2024-03-19T01:24:52.562 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.559+0000 7f18ce344640 1 -- 172.21.15.116:0/545399360 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c81a20e0 con 0x7f18c8101910 2024-03-19T01:24:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.562+0000 7f18c57fa640 1 -- 172.21.15.116:0/545399360 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f18bc046020 con 0x7f18c8101910 2024-03-19T01:24:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.562+0000 7f18c57fa640 1 --2- 172.21.15.116:0/545399360 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f18b00a29f0 0x7f18b00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.562+0000 7f18c57fa640 1 -- 172.21.15.116:0/545399360 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f18bc0f0aa0 con 0x7f18c8101910 2024-03-19T01:24:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.563+0000 7f18c7fff640 1 --2- 172.21.15.116:0/545399360 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f18b00a29f0 0x7f18b00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.564+0000 7f18c7fff640 1 --2- 172.21.15.116:0/545399360 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f18b00a29f0 0x7f18b00a4eb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f18c819ab70 tx=0x7f18b8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.565+0000 7f18d1005740 1 Processor -- start 2024-03-19T01:24:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.565+0000 7f18d1005740 1 -- start start 2024-03-19T01:24:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.566+0000 7f18d1005740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d0b100 0x562e41d0b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.566+0000 7f18d1005740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d105f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.566+0000 7f18d1005740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d10c60 0x562e41d130f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.566+0000 7f18c7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d0b100 0x562e41d0b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.566+0000 7f18c7fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d0b100 0x562e41d0b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:43064/0 (socket says 172.21.15.116:43064) 2024-03-19T01:24:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.566+0000 7f18c77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d105f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.566+0000 7f18c77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d105f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:33018/0 (socket says 172.21.15.116:33018) 2024-03-19T01:24:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.566+0000 7f18cc8e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d10c60 0x562e41d130f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.566+0000 7f18cc8e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d10c60 0x562e41d130f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38060/0 (socket says 172.21.15.116:38060) 2024-03-19T01:24:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.566+0000 7f18c7fff640 1 -- 172.21.15.116:0/3502727080 learned_addr learned my addr 172.21.15.116:0/3502727080 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.566+0000 7f18d1005740 1 -- 172.21.15.116:0/3502727080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562e41c6fbe0 con 0x562e41d10c60 2024-03-19T01:24:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.566+0000 7f18d1005740 1 -- 172.21.15.116:0/3502727080 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562e41c18960 con 0x562e41d0baf0 2024-03-19T01:24:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.566+0000 7f18d1005740 1 -- 172.21.15.116:0/3502727080 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562e41c6bc80 con 0x562e41d0b100 2024-03-19T01:24:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.567+0000 7f18c7fff640 1 -- 172.21.15.116:0/3502727080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 msgr2=0x562e41d105f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.567+0000 7f18c7fff640 1 --2- 172.21.15.116:0/3502727080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d105f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.567+0000 7f18c7fff640 1 -- 172.21.15.116:0/3502727080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d10c60 msgr2=0x562e41d130f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.567+0000 7f18c7fff640 1 --2- 172.21.15.116:0/3502727080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d10c60 0x562e41d130f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.567+0000 7f18c7fff640 1 -- 172.21.15.116:0/3502727080 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e41c0fd20 con 0x562e41d0b100 2024-03-19T01:24:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.567+0000 7f18c7fff640 1 --2- 172.21.15.116:0/3502727080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d0b100 0x562e41d0b520 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f18b8011fd0 tx=0x7f18b8014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa03eba5a6ba89ec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.567+0000 7f18ae7fc640 1 -- 172.21.15.116:0/3502727080 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b8003d70 con 0x562e41d0b100 2024-03-19T01:24:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.567+0000 7f18ae7fc640 1 -- 172.21.15.116:0/3502727080 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b8005500 con 0x562e41d0b100 2024-03-19T01:24:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.567+0000 7f18ae7fc640 1 -- 172.21.15.116:0/3502727080 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b801e660 con 0x562e41d0b100 2024-03-19T01:24:52.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.568+0000 7f18d1005740 1 -- 172.21.15.116:0/3502727080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d0b100 msgr2=0x562e41d0b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.568+0000 7f18d1005740 1 --2- 172.21.15.116:0/3502727080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d0b100 0x562e41d0b520 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f18b8011fd0 tx=0x7f18b8014000 comp rx=0 tx=0).stop 2024-03-19T01:24:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.569+0000 7f18d1005740 1 -- 172.21.15.116:0/3502727080 shutdown_connections 2024-03-19T01:24:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.569+0000 7f18d1005740 1 --2- 172.21.15.116:0/3502727080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d10c60 0x562e41d130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.569+0000 7f18d1005740 1 --2- 172.21.15.116:0/3502727080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.569+0000 7f18d1005740 1 --2- 172.21.15.116:0/3502727080 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d0b100 0x562e41d0b520 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.569+0000 7f18d1005740 1 -- 172.21.15.116:0/3502727080 >> 172.21.15.116:0/3502727080 conn(0x562e41d06670 msgr2=0x562e41d06a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.569+0000 7f18d1005740 1 -- 172.21.15.116:0/3502727080 shutdown_connections 2024-03-19T01:24:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.569+0000 7f18d1005740 1 -- 172.21.15.116:0/3502727080 wait complete. 2024-03-19T01:24:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.569+0000 7f18d1005740 1 -- start start 2024-03-19T01:24:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.569+0000 7f18d1005740 1 Processor -- start 2024-03-19T01:24:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.569+0000 7f18d1005740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0b100 0x562e41d15070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18d1005740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d0baf0 0x562e41d155b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18d1005740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d10c60 0x562e41d1c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18d1005740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x562e41d13c10 con 0x562e41d10c60 2024-03-19T01:24:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18d1005740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x562e41c2bdb0 con 0x562e41d0b100 2024-03-19T01:24:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18d1005740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x562e41d13d90 con 0x562e41d0baf0 2024-03-19T01:24:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18c7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0b100 0x562e41d15070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18d1005740 10 client.0 _ll_register_callbacks cb 0x562e41c30e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:24:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18c7fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0b100 0x562e41d15070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:33026/0 (socket says 172.21.15.116:33026) 2024-03-19T01:24:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18c7fff640 1 -- 172.21.15.116:0/3816946156 learned_addr learned my addr 172.21.15.116:0/3816946156 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18c77fe640 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d0baf0 0x562e41d155b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18cc8e1640 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d10c60 0x562e41d1c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18c7fff640 1 -- 172.21.15.116:0/3816946156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d0baf0 msgr2=0x562e41d155b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18c7fff640 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d0baf0 0x562e41d155b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18c7fff640 1 -- 172.21.15.116:0/3816946156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d10c60 msgr2=0x562e41d1c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18c7fff640 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d10c60 0x562e41d1c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18c7fff640 1 -- 172.21.15.116:0/3816946156 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e41c2ac60 con 0x562e41d0b100 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18cc8e1640 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d10c60 0x562e41d1c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18c7fff640 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0b100 0x562e41d15070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f18b8011fa0 tx=0x7f18b8004990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ef169bbfae632ec server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.570+0000 7f18a77fe640 1 -- 172.21.15.116:0/3816946156 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b8036060 con 0x562e41d0b100 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18a77fe640 1 -- 172.21.15.116:0/3816946156 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f18b80051e0 con 0x562e41d0b100 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18d1005740 1 -- 172.21.15.116:0/3816946156 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18b8005500 con 0x562e41d0b100 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18a77fe640 1 -- 172.21.15.116:0/3816946156 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f18b801e420 con 0x562e41d0b100 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18d1005740 10 client.26072 fetch_fsmap learned FSMap version 47 2024-03-19T01:24:52.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18d1005740 1 -- 172.21.15.116:0/3816946156 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562e41984210 con 0x562e41d0b100 2024-03-19T01:24:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18a77fe640 1 -- 172.21.15.116:0/3816946156 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b8005500 con 0x562e41d0b100 2024-03-19T01:24:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18a77fe640 1 -- 172.21.15.116:0/3816946156 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f18b803c5a0 con 0x562e41d0b100 2024-03-19T01:24:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18d1005740 10 client.26072 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:24:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18d1005740 10 client.26072 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:24:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18d1005740 20 client.26072 populate_metadata read hostname 'smithi116' 2024-03-19T01:24:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18d1005740 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562e41d20bd0 0x562e41d23000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18d1005740 4 client.26072 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:24:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18d1005740 1 -- 172.21.15.116:0/3816946156 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e41c7d120 con 0x562e41d20bd0 2024-03-19T01:24:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.571+0000 7f18c77fe640 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562e41d20bd0 0x562e41d23000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.572+0000 7f18a77fe640 10 client.26072 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.572+0000 7f18c77fe640 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562e41d20bd0 0x562e41d23000 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5a140e784d70782 server_cookie=322c66a5b95d54c0 in_seq=0 out_seq=0 2024-03-19T01:24:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.573+0000 7f18a77fe640 1 -- 172.21.15.116:0/3816946156 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562e41c7d120 con 0x562e41d20bd0 2024-03-19T01:24:52.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.573+0000 7f18a77fe640 10 client.26072 handle_command_reply: tid=0 2024-03-19T01:24:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.575+0000 7f18d1005740 1 client.26072 shutdown 2024-03-19T01:24:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.575+0000 7f18d1005740 1 -- 172.21.15.116:0/3816946156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0b100 msgr2=0x562e41d15070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.575+0000 7f18d1005740 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0b100 0x562e41d15070 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f18b8011fa0 tx=0x7f18b8004990 comp rx=0 tx=0).stop 2024-03-19T01:24:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.575+0000 7f18d1005740 1 -- 172.21.15.116:0/3816946156 shutdown_connections 2024-03-19T01:24:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.575+0000 7f18d1005740 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d10c60 0x562e41d1c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.575+0000 7f18d1005740 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562e41d20bd0 0x562e41d23000 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.575+0000 7f18a77fe640 0 client.26072 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.575+0000 7f18d1005740 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d0baf0 0x562e41d155b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.575+0000 7f18d1005740 1 --2- 172.21.15.116:0/3816946156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0b100 0x562e41d15070 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.575+0000 7f18d1005740 1 -- 172.21.15.116:0/3816946156 >> 172.21.15.116:0/3816946156 conn(0x562e41d06670 msgr2=0x562e41d078c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.575+0000 7f18d1005740 1 -- 172.21.15.116:0/3816946156 shutdown_connections 2024-03-19T01:24:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.575+0000 7f18d1005740 1 -- 172.21.15.116:0/3816946156 wait complete. 2024-03-19T01:24:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.575+0000 7f18d1005740 20 client.26072 trim_cache size 0 max 16384 2024-03-19T01:24:52.599 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.598+0000 7f18d1005740 1 Processor -- start 2024-03-19T01:24:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.598+0000 7f18d1005740 1 -- start start 2024-03-19T01:24:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.598+0000 7f18d1005740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d0b100 0x562e41d0c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.598+0000 7f18d1005740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d1c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.598+0000 7f18d1005740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d10c60 0x562e41d0c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.598+0000 7f18d1005740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81aaee0 con 0x562e41d0b100 2024-03-19T01:24:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.598+0000 7f18d1005740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81acae0 con 0x562e41d0baf0 2024-03-19T01:24:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.598+0000 7f18c77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d1c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.598+0000 7f18d1005740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81ab5b0 con 0x562e41d10c60 2024-03-19T01:24:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.598+0000 7f18c77fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d1c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:33034/0 (socket says 172.21.15.116:33034) 2024-03-19T01:24:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.598+0000 7f18c77fe640 1 -- 172.21.15.116:0/3733738753 learned_addr learned my addr 172.21.15.116:0/3733738753 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.598+0000 7f18c7fff640 1 --2- 172.21.15.116:0/3733738753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d0b100 0x562e41d0c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.598+0000 7f18cc8e1640 1 --2- 172.21.15.116:0/3733738753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d10c60 0x562e41d0c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.599+0000 7f18c77fe640 1 -- 172.21.15.116:0/3733738753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d10c60 msgr2=0x562e41d0c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.599+0000 7f18c77fe640 1 --2- 172.21.15.116:0/3733738753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d10c60 0x562e41d0c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.599+0000 7f18c77fe640 1 -- 172.21.15.116:0/3733738753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d0b100 msgr2=0x562e41d0c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.599+0000 7f18c77fe640 1 --2- 172.21.15.116:0/3733738753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d0b100 0x562e41d0c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.599+0000 7f18c77fe640 1 -- 172.21.15.116:0/3733738753 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e41a50620 con 0x562e41d0baf0 2024-03-19T01:24:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.599+0000 7f18c77fe640 1 --2- 172.21.15.116:0/3733738753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d1c5b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f18bc0b9d90 tx=0x7f18bc0ee3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a079d6dabc49137 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.599+0000 7f18adffb640 1 -- 172.21.15.116:0/3733738753 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc0bd3c0 con 0x562e41d0baf0 2024-03-19T01:24:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.599+0000 7f18adffb640 1 -- 172.21.15.116:0/3733738753 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18bc0c1510 con 0x562e41d0baf0 2024-03-19T01:24:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.599+0000 7f18adffb640 1 -- 172.21.15.116:0/3733738753 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc0b9a30 con 0x562e41d0baf0 2024-03-19T01:24:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.600+0000 7f18d1005740 1 -- 172.21.15.116:0/3733738753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 msgr2=0x562e41d1c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.600+0000 7f18d1005740 1 --2- 172.21.15.116:0/3733738753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d1c5b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f18bc0b9d90 tx=0x7f18bc0ee3f0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.600+0000 7f18d1005740 1 -- 172.21.15.116:0/3733738753 shutdown_connections 2024-03-19T01:24:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.600+0000 7f18d1005740 1 --2- 172.21.15.116:0/3733738753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d10c60 0x562e41d0c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.600+0000 7f18d1005740 1 --2- 172.21.15.116:0/3733738753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d1c5b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.600+0000 7f18d1005740 1 --2- 172.21.15.116:0/3733738753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d0b100 0x562e41d0c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.600+0000 7f18d1005740 1 -- 172.21.15.116:0/3733738753 >> 172.21.15.116:0/3733738753 conn(0x562e41d06670 msgr2=0x562e41d14c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.600+0000 7f18d1005740 1 -- 172.21.15.116:0/3733738753 shutdown_connections 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.600+0000 7f18d1005740 1 -- 172.21.15.116:0/3733738753 wait complete. 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18d1005740 1 -- start start 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18d1005740 1 Processor -- start 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18d1005740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d0b100 0x562e41d19ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18d1005740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d1a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18d1005740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d10c60 0x562e41d1b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18d1005740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81ac410 con 0x562e41d0b100 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18d1005740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81abd40 con 0x562e41d0baf0 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18d1005740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f18c81a8ec0 con 0x562e41d10c60 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18d1005740 10 client.0 _ll_register_callbacks cb 0x7f18c0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d0b100 0x562e41d19ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18c7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d0b100 0x562e41d19ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38080/0 (socket says 172.21.15.116:38080) 2024-03-19T01:24:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18c7fff640 1 -- 172.21.15.116:0/1973568942 learned_addr learned my addr 172.21.15.116:0/1973568942 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:24:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18c77fe640 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d1a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.601+0000 7f18cc8e1640 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d10c60 0x562e41d1b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.602+0000 7f18c77fe640 1 -- 172.21.15.116:0/1973568942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d10c60 msgr2=0x562e41d1b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.602+0000 7f18c77fe640 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d10c60 0x562e41d1b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.602+0000 7f18c77fe640 1 -- 172.21.15.116:0/1973568942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d0b100 msgr2=0x562e41d19ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:24:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.602+0000 7f18c77fe640 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d0b100 0x562e41d19ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.602+0000 7f18c77fe640 1 -- 172.21.15.116:0/1973568942 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e41d1bb30 con 0x562e41d0baf0 2024-03-19T01:24:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.602+0000 7f18c7fff640 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d0b100 0x562e41d19ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:24:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.602+0000 7f18c77fe640 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d1a530 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f18bc02c7b0 tx=0x7f18bc02c150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb60003ca9e2ca8f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:24:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.602+0000 7f18a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:24:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.603+0000 7f18a77fe640 1 -- 172.21.15.116:0/1973568942 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc0c5b60 con 0x562e41d0baf0 2024-03-19T01:24:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.603+0000 7f18a77fe640 1 -- 172.21.15.116:0/1973568942 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f18bc003cd0 con 0x562e41d0baf0 2024-03-19T01:24:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.603+0000 7f18d1005740 1 -- 172.21.15.116:0/1973568942 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f18bc0c1510 con 0x562e41d0baf0 2024-03-19T01:24:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.603+0000 7f18a77fe640 1 -- 172.21.15.116:0/1973568942 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f18bc0c1170 con 0x562e41d0baf0 2024-03-19T01:24:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.606+0000 7f18d1005740 10 client.26078 fetch_fsmap learned FSMap version 47 2024-03-19T01:24:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.606+0000 7f18d1005740 1 -- 172.21.15.116:0/1973568942 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562e41d8fa20 con 0x562e41d0baf0 2024-03-19T01:24:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.606+0000 7f18a77fe640 1 -- 172.21.15.116:0/1973568942 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18bc0c1510 con 0x562e41d0baf0 2024-03-19T01:24:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.606+0000 7f18a77fe640 1 -- 172.21.15.116:0/1973568942 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f18bc0c56d0 con 0x562e41d0baf0 2024-03-19T01:24:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.606+0000 7f18d1005740 10 client.26078 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:24:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.606+0000 7f18d1005740 10 client.26078 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:24:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.606+0000 7f18d1005740 20 client.26078 populate_metadata read hostname 'smithi116' 2024-03-19T01:24:52.610 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.606+0000 7f18d1005740 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562e41d20bd0 0x562e41d92400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:24:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.606+0000 7f18d1005740 4 client.26078 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:24:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.606+0000 7f18d1005740 1 -- 172.21.15.116:0/1973568942 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e41c77470 con 0x562e41d20bd0 2024-03-19T01:24:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.607+0000 7f18c7fff640 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562e41d20bd0 0x562e41d92400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:24:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.607+0000 7f18c7fff640 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562e41d20bd0 0x562e41d92400 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f109f721c65b9f36 server_cookie=43e583c592c773d5 in_seq=0 out_seq=0 2024-03-19T01:24:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.607+0000 7f18a77fe640 10 client.26078 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.616+0000 7f18a77fe640 1 -- 172.21.15.116:0/1973568942 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e41c77470 con 0x562e41d20bd0 2024-03-19T01:24:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.616+0000 7f18a77fe640 10 client.26078 handle_command_reply: tid=0 2024-03-19T01:24:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.616+0000 7f18d1005740 1 client.26078 shutdown 2024-03-19T01:24:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.617+0000 7f18d1005740 1 -- 172.21.15.116:0/1973568942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 msgr2=0x562e41d1a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.617+0000 7f18d1005740 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d1a530 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f18bc02c7b0 tx=0x7f18bc02c150 comp rx=0 tx=0).stop 2024-03-19T01:24:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.617+0000 7f18d1005740 1 -- 172.21.15.116:0/1973568942 shutdown_connections 2024-03-19T01:24:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.617+0000 7f18d1005740 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x562e41d20bd0 0x562e41d92400 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.617+0000 7f18d1005740 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x562e41d10c60 0x562e41d1b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.617+0000 7f18d1005740 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x562e41d0baf0 0x562e41d1a530 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.617+0000 7f18d1005740 1 --2- 172.21.15.116:0/1973568942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x562e41d0b100 0x562e41d19ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.617+0000 7f18a77fe640 0 client.26078 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:24:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.617+0000 7f18d1005740 1 -- 172.21.15.116:0/1973568942 >> 172.21.15.116:0/1973568942 conn(0x562e41d06670 msgr2=0x562e41d098d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18d1005740 1 -- 172.21.15.116:0/1973568942 shutdown_connections 2024-03-19T01:24:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18d1005740 1 -- 172.21.15.116:0/1973568942 wait complete. 2024-03-19T01:24:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18d1005740 20 client.26078 trim_cache size 0 max 16384 2024-03-19T01:24:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:24:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:24:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:24:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:24:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18ce344640 1 -- 172.21.15.116:0/545399360 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f18b00a29f0 msgr2=0x7f18b00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18ce344640 1 --2- 172.21.15.116:0/545399360 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f18b00a29f0 0x7f18b00a4eb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f18c819ab70 tx=0x7f18b8005f90 comp rx=0 tx=0).stop 2024-03-19T01:24:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18ce344640 1 -- 172.21.15.116:0/545399360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18c8101910 msgr2=0x7f18c819a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:24:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18ce344640 1 --2- 172.21.15.116:0/545399360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18c8101910 0x7f18c819a1e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f18bc036040 tx=0x7f18bc005a20 comp rx=0 tx=0).stop 2024-03-19T01:24:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18ce344640 1 -- 172.21.15.116:0/545399360 shutdown_connections 2024-03-19T01:24:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18ce344640 1 --2- 172.21.15.116:0/545399360 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18c81022d0 0x7f18c81a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18ce344640 1 --2- 172.21.15.116:0/545399360 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f18b00a29f0 0x7f18b00a4eb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18ce344640 1 --2- 172.21.15.116:0/545399360 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18c8101910 0x7f18c819a1e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18ce344640 1 --2- 172.21.15.116:0/545399360 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f18c8100720 0x7f18c8199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:24:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18ce344640 1 -- 172.21.15.116:0/545399360 >> 172.21.15.116:0/545399360 conn(0x7f18c80fbc50 msgr2=0x7f18c8104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:24:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18ce344640 1 -- 172.21.15.116:0/545399360 shutdown_connections 2024-03-19T01:24:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:24:52.618+0000 7f18ce344640 1 -- 172.21.15.116:0/545399360 wait complete. 2024-03-19T01:24:53.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:53 smithi156 ceph-mon[66799]: pgmap v719: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 5.9 MiB/s wr, 637 op/s 2024-03-19T01:24:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:53 smithi162 ceph-mon[67178]: pgmap v719: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 5.9 MiB/s wr, 637 op/s 2024-03-19T01:24:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:53 smithi116 ceph-mon[62701]: pgmap v719: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 5.9 MiB/s wr, 637 op/s 2024-03-19T01:24:55.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:55 smithi156 ceph-mon[66799]: pgmap v720: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 6.5 MiB/s wr, 648 op/s 2024-03-19T01:24:55.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:55 smithi162 ceph-mon[67178]: pgmap v720: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 6.5 MiB/s wr, 648 op/s 2024-03-19T01:24:55.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:55 smithi116 ceph-mon[62701]: pgmap v720: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 6.5 MiB/s wr, 648 op/s 2024-03-19T01:24:56.544 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 730.0 s, 1922.0 tps, lat 52.043 ms stddev 61.764 2024-03-19T01:24:57.456 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:57 smithi116 ceph-mon[62701]: pgmap v721: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6.9 MiB/s wr, 708 op/s 2024-03-19T01:24:57.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:57 smithi156 ceph-mon[66799]: pgmap v721: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6.9 MiB/s wr, 708 op/s 2024-03-19T01:24:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:57 smithi162 ceph-mon[67178]: pgmap v721: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6.9 MiB/s wr, 708 op/s 2024-03-19T01:24:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:24:59 smithi156 ceph-mon[66799]: pgmap v722: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.4 MiB/s wr, 599 op/s 2024-03-19T01:24:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:24:59 smithi162 ceph-mon[67178]: pgmap v722: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.4 MiB/s wr, 599 op/s 2024-03-19T01:24:59.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:24:59 smithi116 ceph-mon[62701]: pgmap v722: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 5.4 MiB/s wr, 599 op/s 2024-03-19T01:25:01.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:01 smithi156 ceph-mon[66799]: pgmap v723: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.2 MiB/s wr, 581 op/s 2024-03-19T01:25:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 735.0 s, 1421.0 tps, lat 70.098 ms stddev 72.589 2024-03-19T01:25:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:01 smithi162 ceph-mon[67178]: pgmap v723: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.2 MiB/s wr, 581 op/s 2024-03-19T01:25:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:01 smithi116 ceph-mon[62701]: pgmap v723: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 5.2 MiB/s wr, 581 op/s 2024-03-19T01:25:03.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:03 smithi156 ceph-mon[66799]: pgmap v724: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.0 MiB/s wr, 642 op/s 2024-03-19T01:25:03.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:03 smithi162 ceph-mon[67178]: pgmap v724: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.0 MiB/s wr, 642 op/s 2024-03-19T01:25:03.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:03 smithi116 ceph-mon[62701]: pgmap v724: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.0 MiB/s wr, 642 op/s 2024-03-19T01:25:05.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:05 smithi156 ceph-mon[66799]: pgmap v725: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.7 MiB/s wr, 599 op/s 2024-03-19T01:25:05.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:05 smithi162 ceph-mon[67178]: pgmap v725: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.7 MiB/s wr, 599 op/s 2024-03-19T01:25:05.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:05 smithi116 ceph-mon[62701]: pgmap v725: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.7 MiB/s wr, 599 op/s 2024-03-19T01:25:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.744+0000 7f5ea2ff4640 1 -- 172.21.15.116:0/48110168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9c0ffb00 msgr2=0x7f5e9c0fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.744+0000 7f5ea2ff4640 1 --2- 172.21.15.116:0/48110168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9c0ffb00 0x7f5e9c0fff20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5e90009930 tx=0x7f5e9002bc30 comp rx=0 tx=0).stop 2024-03-19T01:25:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.744+0000 7f5ea2ff4640 1 -- 172.21.15.116:0/48110168 shutdown_connections 2024-03-19T01:25:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.744+0000 7f5ea2ff4640 1 --2- 172.21.15.116:0/48110168 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9c103490 0x7f5e9c109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.744+0000 7f5ea2ff4640 1 --2- 172.21.15.116:0/48110168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e9c102ad0 0x7f5e9c102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.744+0000 7f5ea2ff4640 1 --2- 172.21.15.116:0/48110168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9c0ffb00 0x7f5e9c0fff20 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.744+0000 7f5ea2ff4640 1 -- 172.21.15.116:0/48110168 >> 172.21.15.116:0/48110168 conn(0x7f5e9c0fb470 msgr2=0x7f5e9c0fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.745+0000 7f5ea2ff4640 1 -- 172.21.15.116:0/48110168 shutdown_connections 2024-03-19T01:25:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.745+0000 7f5ea2ff4640 1 -- 172.21.15.116:0/48110168 wait complete. 2024-03-19T01:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.745+0000 7f5ea2ff4640 1 Processor -- start 2024-03-19T01:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.745+0000 7f5ea2ff4640 1 -- start start 2024-03-19T01:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5ea2ff4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e9c0ffb00 0x7f5e9c19e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5ea2ff4640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9c102ad0 0x7f5e9c19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5ea2ff4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9c103490 0x7f5e9c1a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5ea2ff4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c10d4d0 con 0x7f5e9c103490 2024-03-19T01:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5ea2ff4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c10d350 con 0x7f5e9c0ffb00 2024-03-19T01:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5ea2ff4640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c10d650 con 0x7f5e9c102ad0 2024-03-19T01:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5ea0d90640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e9c0ffb00 0x7f5e9c19e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5e9bfff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9c102ad0 0x7f5e9c19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5ea0d90640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e9c0ffb00 0x7f5e9c19e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:35714/0 (socket says 172.21.15.116:35714) 2024-03-19T01:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5ea1591640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9c103490 0x7f5e9c1a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5ea0d90640 1 -- 172.21.15.116:0/460572041 learned_addr learned my addr 172.21.15.116:0/460572041 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5e9bfff640 1 -- 172.21.15.116:0/460572041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e9c0ffb00 msgr2=0x7f5e9c19e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5e9bfff640 1 --2- 172.21.15.116:0/460572041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e9c0ffb00 0x7f5e9c19e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5e9bfff640 1 -- 172.21.15.116:0/460572041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9c103490 msgr2=0x7f5e9c1a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5e9bfff640 1 --2- 172.21.15.116:0/460572041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9c103490 0x7f5e9c1a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5e9bfff640 1 -- 172.21.15.116:0/460572041 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9c1a5d40 con 0x7f5e9c102ad0 2024-03-19T01:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5e9bfff640 1 --2- 172.21.15.116:0/460572041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9c102ad0 0x7f5e9c19e660 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5e88005fd0 tx=0x7f5e8800dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.746+0000 7f5e99ffb640 1 -- 172.21.15.116:0/460572041 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e88009c30 con 0x7f5e9c102ad0 2024-03-19T01:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.747+0000 7f5e99ffb640 1 -- 172.21.15.116:0/460572041 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e88009dd0 con 0x7f5e9c102ad0 2024-03-19T01:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.747+0000 7f5ea2ff4640 1 -- 172.21.15.116:0/460572041 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9c1a5fd0 con 0x7f5e9c102ad0 2024-03-19T01:25:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.747+0000 7f5ea2ff4640 1 -- 172.21.15.116:0/460572041 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e9c1a6540 con 0x7f5e9c102ad0 2024-03-19T01:25:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.748+0000 7f5e99ffb640 1 -- 172.21.15.116:0/460572041 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e88010460 con 0x7f5e9c102ad0 2024-03-19T01:25:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.749+0000 7f5e99ffb640 1 -- 172.21.15.116:0/460572041 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f5e88004030 con 0x7f5e9c102ad0 2024-03-19T01:25:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.750+0000 7f5e99ffb640 1 --2- 172.21.15.116:0/460572041 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5e8c0a2790 0x7f5e8c0a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.750+0000 7f5e99ffb640 1 -- 172.21.15.116:0/460572041 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f5e880ce500 con 0x7f5e9c102ad0 2024-03-19T01:25:05.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.751+0000 7f5ea0d90640 1 --2- 172.21.15.116:0/460572041 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5e8c0a2790 0x7f5e8c0a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.751+0000 7f5ea5be8740 1 Processor -- start 2024-03-19T01:25:05.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.751+0000 7f5ea5be8740 1 -- start start 2024-03-19T01:25:05.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.752+0000 7f5ea5be8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21100 0x561bacb21520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.752+0000 7f5ea5be8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21af0 0x561bacb265f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.752+0000 7f5ea5be8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb26c60 0x561bacb290f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.752+0000 7f5ea5be8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561baca85be0 con 0x561bacb26c60 2024-03-19T01:25:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.752+0000 7f5ea5be8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561baca2e960 con 0x561bacb21100 2024-03-19T01:25:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.752+0000 7f5ea5be8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561baca81c80 con 0x561bacb21af0 2024-03-19T01:25:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.752+0000 7f5ea0d90640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21100 0x561bacb21520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.752+0000 7f5ea1591640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb26c60 0x561bacb290f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.752+0000 7f5ea1591640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb26c60 0x561bacb290f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57098/0 (socket says 172.21.15.116:57098) 2024-03-19T01:25:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.752+0000 7f5ea1591640 1 -- 172.21.15.116:0/686879991 learned_addr learned my addr 172.21.15.116:0/686879991 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.753+0000 7f5ea0d90640 1 -- 172.21.15.116:0/686879991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21af0 msgr2=0x561bacb265f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-19T01:25:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.753+0000 7f5ea0d90640 1 --2- 172.21.15.116:0/686879991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21af0 0x561bacb265f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.753+0000 7f5ea0d90640 1 -- 172.21.15.116:0/686879991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb26c60 msgr2=0x561bacb290f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.753+0000 7f5ea0d90640 1 --2- 172.21.15.116:0/686879991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb26c60 0x561bacb290f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.753+0000 7f5ea0d90640 1 -- 172.21.15.116:0/686879991 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561baca25d20 con 0x561bacb21100 2024-03-19T01:25:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.753+0000 7f5ea0d90640 1 --2- 172.21.15.116:0/460572041 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5e8c0a2790 0x7f5e8c0a4c50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5e900096c0 tx=0x7f5e90000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.753+0000 7f5ea0d90640 1 --2- 172.21.15.116:0/686879991 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21100 0x561bacb21520 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5e90009900 tx=0x7f5e90046000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6d6bf733e0f4138 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.753+0000 7f5e86ffd640 1 -- 172.21.15.116:0/686879991 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e90004440 con 0x561bacb21100 2024-03-19T01:25:05.755 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.754+0000 7f5e86ffd640 1 -- 172.21.15.116:0/686879991 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e90005520 con 0x561bacb21100 2024-03-19T01:25:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.754+0000 7f5e86ffd640 1 -- 172.21.15.116:0/686879991 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e90003830 con 0x561bacb21100 2024-03-19T01:25:05.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.754+0000 7f5ea5be8740 1 -- 172.21.15.116:0/686879991 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21100 msgr2=0x561bacb21520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.755+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/686879991 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21100 0x561bacb21520 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5e90009900 tx=0x7f5e90046000 comp rx=0 tx=0).stop 2024-03-19T01:25:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.755+0000 7f5ea5be8740 1 -- 172.21.15.116:0/686879991 shutdown_connections 2024-03-19T01:25:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.755+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/686879991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb26c60 0x561bacb290f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.755+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/686879991 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21af0 0x561bacb265f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.755+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/686879991 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21100 0x561bacb21520 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.755+0000 7f5ea5be8740 1 -- 172.21.15.116:0/686879991 >> 172.21.15.116:0/686879991 conn(0x561bacb1c670 msgr2=0x561bacb1ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.755+0000 7f5ea5be8740 1 -- 172.21.15.116:0/686879991 shutdown_connections 2024-03-19T01:25:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.755+0000 7f5ea5be8740 1 -- 172.21.15.116:0/686879991 wait complete. 2024-03-19T01:25:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.755+0000 7f5ea5be8740 1 -- start start 2024-03-19T01:25:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.755+0000 7f5ea5be8740 1 Processor -- start 2024-03-19T01:25:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5ea5be8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb21100 0x561bacb3c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5ea5be8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21af0 0x561bacb3c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5ea5be8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb26c60 0x561bacb43680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5ea5be8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561bacb3ac10 con 0x561bacb21100 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5ea5be8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561baca41db0 con 0x561bacb21af0 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5ea5be8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561bacb3ad90 con 0x561bacb26c60 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5ea5be8740 10 client.0 _ll_register_callbacks cb 0x561baca46e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5e9bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21af0 0x561bacb3c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5ea0d90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb21100 0x561bacb3c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5ea0d90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb21100 0x561bacb3c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57106/0 (socket says 172.21.15.116:57106) 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5ea0d90640 1 -- 172.21.15.116:0/184699346 learned_addr learned my addr 172.21.15.116:0/184699346 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5e9bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21af0 0x561bacb3c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:35728/0 (socket says 172.21.15.116:35728) 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5ea1591640 1 --2- 172.21.15.116:0/184699346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb26c60 0x561bacb43680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5e9bfff640 1 -- 172.21.15.116:0/184699346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb26c60 msgr2=0x561bacb43680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5e9bfff640 1 --2- 172.21.15.116:0/184699346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb26c60 0x561bacb43680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5e9bfff640 1 -- 172.21.15.116:0/184699346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb21100 msgr2=0x561bacb3c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5e9bfff640 1 --2- 172.21.15.116:0/184699346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb21100 0x561bacb3c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5e9bfff640 1 -- 172.21.15.116:0/184699346 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561baca40c60 con 0x561bacb21af0 2024-03-19T01:25:05.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5ea0d90640 1 --2- 172.21.15.116:0/184699346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb21100 0x561bacb3c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:25:05.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5e9bfff640 1 --2- 172.21.15.116:0/184699346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21af0 0x561bacb3c640 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x561bacb22560 tx=0x7f5e8809af30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3f0a14ca61b8b60 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:05.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.756+0000 7f5e7bfff640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5e7bfff640 1 -- 172.21.15.116:0/184699346 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e88020040 con 0x561bacb21af0 2024-03-19T01:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5e7bfff640 1 -- 172.21.15.116:0/184699346 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f5e880975a0 con 0x561bacb21af0 2024-03-19T01:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5ea5be8740 1 -- 172.21.15.116:0/184699346 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e90005520 con 0x561bacb21af0 2024-03-19T01:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5e7bfff640 1 -- 172.21.15.116:0/184699346 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5e880ce0d0 con 0x561bacb21af0 2024-03-19T01:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5ea5be8740 10 client.26090 fetch_fsmap learned FSMap version 47 2024-03-19T01:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5ea5be8740 1 -- 172.21.15.116:0/184699346 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x561bac79a210 con 0x561bacb21af0 2024-03-19T01:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5e7bfff640 1 -- 172.21.15.116:0/184699346 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e90005520 con 0x561bacb21af0 2024-03-19T01:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5e7bfff640 1 -- 172.21.15.116:0/184699346 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f5e88096c80 con 0x561bacb21af0 2024-03-19T01:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5ea5be8740 10 client.26090 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5ea5be8740 10 client.26090 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5ea5be8740 20 client.26090 populate_metadata read hostname 'smithi116' 2024-03-19T01:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/184699346 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561bacb47f70 0x561bacb4a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5ea5be8740 4 client.26090 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.757+0000 7f5ea5be8740 1 -- 172.21.15.116:0/184699346 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561baca93120 con 0x561bacb47f70 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.758+0000 7f5ea0d90640 1 --2- 172.21.15.116:0/184699346 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561bacb47f70 0x561bacb4a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.758+0000 7f5e7bfff640 10 client.26090 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.758+0000 7f5ea0d90640 1 --2- 172.21.15.116:0/184699346 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561bacb47f70 0x561bacb4a360 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5b4d25bededa05a server_cookie=7900cebfe409c7dd in_seq=0 out_seq=0 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5e7bfff640 1 -- 172.21.15.116:0/184699346 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561baca93120 con 0x561bacb47f70 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5e7bfff640 10 client.26090 handle_command_reply: tid=0 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5ea5be8740 1 client.26090 shutdown 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5ea5be8740 1 -- 172.21.15.116:0/184699346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21af0 msgr2=0x561bacb3c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/184699346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21af0 0x561bacb3c640 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x561bacb22560 tx=0x7f5e8809af30 comp rx=0 tx=0).stop 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5ea5be8740 1 -- 172.21.15.116:0/184699346 shutdown_connections 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/184699346 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561bacb47f70 0x561bacb4a360 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/184699346 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb26c60 0x561bacb43680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/184699346 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21af0 0x561bacb3c640 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5e7bfff640 0 client.26090 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/184699346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb21100 0x561bacb3c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5ea5be8740 1 -- 172.21.15.116:0/184699346 >> 172.21.15.116:0/184699346 conn(0x561bacb1c670 msgr2=0x561bacb1d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5ea5be8740 1 -- 172.21.15.116:0/184699346 shutdown_connections 2024-03-19T01:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5ea5be8740 1 -- 172.21.15.116:0/184699346 wait complete. 2024-03-19T01:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.759+0000 7f5ea5be8740 20 client.26090 trim_cache size 0 max 16384 2024-03-19T01:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.779+0000 7f5ea5be8740 1 Processor -- start 2024-03-19T01:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.779+0000 7f5ea5be8740 1 -- start start 2024-03-19T01:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.779+0000 7f5ea5be8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21100 0x561bacb1ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.779+0000 7f5ea5be8740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21af0 0x561bacb43640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.779+0000 7f5ea5be8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb26c60 0x561bacb4ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.779+0000 7f5ea5be8740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c1a6a40 con 0x561bacb26c60 2024-03-19T01:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.779+0000 7f5ea5be8740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c1a8640 con 0x561bacb21af0 2024-03-19T01:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.779+0000 7f5ea5be8740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c1a7110 con 0x561bacb21100 2024-03-19T01:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.779+0000 7f5ea1591640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb26c60 0x561bacb4ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.779+0000 7f5e9bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21af0 0x561bacb43640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.779+0000 7f5ea1591640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb26c60 0x561bacb4ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57110/0 (socket says 172.21.15.116:57110) 2024-03-19T01:25:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.779+0000 7f5ea1591640 1 -- 172.21.15.116:0/188842415 learned_addr learned my addr 172.21.15.116:0/188842415 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.779+0000 7f5ea0d90640 1 --2- 172.21.15.116:0/188842415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21100 0x561bacb1ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.780+0000 7f5ea1591640 1 -- 172.21.15.116:0/188842415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21100 msgr2=0x561bacb1ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.780+0000 7f5ea1591640 1 --2- 172.21.15.116:0/188842415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21100 0x561bacb1ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.780+0000 7f5ea1591640 1 -- 172.21.15.116:0/188842415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21af0 msgr2=0x561bacb43640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.780+0000 7f5ea1591640 1 --2- 172.21.15.116:0/188842415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21af0 0x561bacb43640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.780+0000 7f5ea1591640 1 -- 172.21.15.116:0/188842415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bac866620 con 0x561bacb26c60 2024-03-19T01:25:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.780+0000 7f5e9bfff640 1 --2- 172.21.15.116:0/188842415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21af0 0x561bacb43640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:25:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.780+0000 7f5ea1591640 1 --2- 172.21.15.116:0/188842415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb26c60 0x561bacb4ebf0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5e94003b60 tx=0x7f5e94004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=782dbc41a48ff499 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.780+0000 7f5e867fc640 1 -- 172.21.15.116:0/188842415 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e94020070 con 0x561bacb26c60 2024-03-19T01:25:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.780+0000 7f5e867fc640 1 -- 172.21.15.116:0/188842415 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e9401b910 con 0x561bacb26c60 2024-03-19T01:25:05.783 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.780+0000 7f5e867fc640 1 -- 172.21.15.116:0/188842415 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e94010a30 con 0x561bacb26c60 2024-03-19T01:25:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.781+0000 7f5ea5be8740 1 -- 172.21.15.116:0/188842415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb26c60 msgr2=0x561bacb4ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.781+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/188842415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb26c60 0x561bacb4ebf0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5e94003b60 tx=0x7f5e94004410 comp rx=0 tx=0).stop 2024-03-19T01:25:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.781+0000 7f5ea5be8740 1 -- 172.21.15.116:0/188842415 shutdown_connections 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.781+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/188842415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb26c60 0x561bacb4ebf0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.781+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/188842415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb21af0 0x561bacb43640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.781+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/188842415 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21100 0x561bacb1ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.781+0000 7f5ea5be8740 1 -- 172.21.15.116:0/188842415 >> 172.21.15.116:0/188842415 conn(0x561bacb1c670 msgr2=0x561bacb1f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.781+0000 7f5ea5be8740 1 -- 172.21.15.116:0/188842415 shutdown_connections 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.781+0000 7f5ea5be8740 1 -- 172.21.15.116:0/188842415 wait complete. 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.782+0000 7f5ea5be8740 1 -- start start 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.782+0000 7f5ea5be8740 1 Processor -- start 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.782+0000 7f5ea5be8740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21100 0x561bacb41030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.782+0000 7f5ea5be8740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb21af0 0x561bacb41570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.782+0000 7f5ea0d90640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21100 0x561bacb41030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.782+0000 7f5e9bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb21af0 0x561bacb41570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.782+0000 7f5e9bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb21af0 0x561bacb41570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57118/0 (socket says 172.21.15.116:57118) 2024-03-19T01:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.782+0000 7f5e9bfff640 1 -- 172.21.15.116:0/636796007 learned_addr learned my addr 172.21.15.116:0/636796007 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.782+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/636796007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb26c60 0x561bacb42580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.782+0000 7f5ea5be8740 1 -- 172.21.15.116:0/636796007 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c1a7f70 con 0x561bacb21af0 2024-03-19T01:25:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.782+0000 7f5ea5be8740 1 -- 172.21.15.116:0/636796007 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c1a78a0 con 0x561bacb26c60 2024-03-19T01:25:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.782+0000 7f5ea5be8740 1 -- 172.21.15.116:0/636796007 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9c071d00 con 0x561bacb21100 2024-03-19T01:25:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.782+0000 7f5ea1591640 1 --2- 172.21.15.116:0/636796007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb26c60 0x561bacb42580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.783+0000 7f5ea5be8740 10 client.0 _ll_register_callbacks cb 0x7f5e8800bef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:25:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.783+0000 7f5ea0d90640 1 -- 172.21.15.116:0/636796007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb26c60 msgr2=0x561bacb42580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.783+0000 7f5ea0d90640 1 --2- 172.21.15.116:0/636796007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb26c60 0x561bacb42580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.783+0000 7f5ea0d90640 1 -- 172.21.15.116:0/636796007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb21af0 msgr2=0x561bacb41570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.783+0000 7f5ea0d90640 1 --2- 172.21.15.116:0/636796007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb21af0 0x561bacb41570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.783+0000 7f5ea0d90640 1 -- 172.21.15.116:0/636796007 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bacb42b70 con 0x561bacb21100 2024-03-19T01:25:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.783+0000 7f5ea0d90640 1 --2- 172.21.15.116:0/636796007 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21100 0x561bacb41030 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x561bacb3ce00 tx=0x7f5e900092d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60895d7432890acc server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:05.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.784+0000 7f5e7bfff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:25:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.785+0000 7f5e7bfff640 1 -- 172.21.15.116:0/636796007 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9005e0e0 con 0x561bacb21100 2024-03-19T01:25:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.785+0000 7f5e7bfff640 1 -- 172.21.15.116:0/636796007 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f5e9005e280 con 0x561bacb21100 2024-03-19T01:25:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.785+0000 7f5ea5be8740 1 -- 172.21.15.116:0/636796007 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e9401b910 con 0x561bacb21100 2024-03-19T01:25:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.785+0000 7f5e7bfff640 1 -- 172.21.15.116:0/636796007 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5e9006f050 con 0x561bacb21100 2024-03-19T01:25:05.789 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.785+0000 7f5ea5be8740 10 client.26089 fetch_fsmap learned FSMap version 47 2024-03-19T01:25:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.785+0000 7f5ea5be8740 1 -- 172.21.15.116:0/636796007 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x561bacbae580 con 0x561bacb21100 2024-03-19T01:25:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.786+0000 7f5e7bfff640 1 -- 172.21.15.116:0/636796007 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e90041040 con 0x561bacb21100 2024-03-19T01:25:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.786+0000 7f5e7bfff640 1 -- 172.21.15.116:0/636796007 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f5e90034440 con 0x561bacb21100 2024-03-19T01:25:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.786+0000 7f5ea5be8740 10 client.26089 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:25:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.786+0000 7f5ea5be8740 10 client.26089 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:25:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.786+0000 7f5ea5be8740 20 client.26089 populate_metadata read hostname 'smithi116' 2024-03-19T01:25:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.786+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/636796007 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561bacbaeb60 0x561bacbb0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.786+0000 7f5ea5be8740 4 client.26089 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:25:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.786+0000 7f5ea5be8740 1 -- 172.21.15.116:0/636796007 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f5e9401b910 con 0x561bacbaeb60 2024-03-19T01:25:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.786+0000 7f5e9bfff640 1 --2- 172.21.15.116:0/636796007 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561bacbaeb60 0x561bacbb0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.787+0000 7f5e9bfff640 1 --2- 172.21.15.116:0/636796007 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561bacbaeb60 0x561bacbb0f80 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3aa5da02d2814d53 server_cookie=f511ae776367628f in_seq=0 out_seq=0 2024-03-19T01:25:05.790 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.787+0000 7f5e7bfff640 10 client.26089 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:05.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.794+0000 7f5e7bfff640 1 -- 172.21.15.116:0/636796007 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5e9401b910 con 0x561bacbaeb60 2024-03-19T01:25:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.794+0000 7f5e7bfff640 10 client.26089 handle_command_reply: tid=0 2024-03-19T01:25:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.794+0000 7f5ea5be8740 1 client.26089 shutdown 2024-03-19T01:25:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.795+0000 7f5ea5be8740 1 -- 172.21.15.116:0/636796007 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21100 msgr2=0x561bacb41030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:05.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.795+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/636796007 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21100 0x561bacb41030 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x561bacb3ce00 tx=0x7f5e900092d0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.795+0000 7f5ea5be8740 1 -- 172.21.15.116:0/636796007 shutdown_connections 2024-03-19T01:25:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.795+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/636796007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561bacb26c60 0x561bacb42580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.795+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/636796007 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561bacb21af0 0x561bacb41570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.795+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/636796007 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561bacbaeb60 0x561bacbb0f80 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.795+0000 7f5ea5be8740 1 --2- 172.21.15.116:0/636796007 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561bacb21100 0x561bacb41030 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.795+0000 7f5ea5be8740 1 -- 172.21.15.116:0/636796007 >> 172.21.15.116:0/636796007 conn(0x561bacb1c670 msgr2=0x561bacb20160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.796+0000 7f5e7bfff640 0 client.26089 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.796+0000 7f5ea5be8740 1 -- 172.21.15.116:0/636796007 shutdown_connections 2024-03-19T01:25:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.796+0000 7f5ea5be8740 1 -- 172.21.15.116:0/636796007 wait complete. 2024-03-19T01:25:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.796+0000 7f5ea5be8740 20 client.26089 trim_cache size 0 max 16384 2024-03-19T01:25:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:25:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:25:05.798 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:25:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:25:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.797+0000 7f5ea2ff4640 1 -- 172.21.15.116:0/460572041 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5e8c0a2790 msgr2=0x7f5e8c0a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.797+0000 7f5ea2ff4640 1 --2- 172.21.15.116:0/460572041 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5e8c0a2790 0x7f5e8c0a4c50 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5e900096c0 tx=0x7f5e90000bc0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.797+0000 7f5ea2ff4640 1 -- 172.21.15.116:0/460572041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9c102ad0 msgr2=0x7f5e9c19e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.797+0000 7f5ea2ff4640 1 --2- 172.21.15.116:0/460572041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9c102ad0 0x7f5e9c19e660 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5e88005fd0 tx=0x7f5e8800dde0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.797+0000 7f5ea2ff4640 1 -- 172.21.15.116:0/460572041 shutdown_connections 2024-03-19T01:25:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.797+0000 7f5ea2ff4640 1 --2- 172.21.15.116:0/460572041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5e9c103490 0x7f5e9c1a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.799 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.797+0000 7f5ea2ff4640 1 --2- 172.21.15.116:0/460572041 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5e9c102ad0 0x7f5e9c19e660 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.797+0000 7f5ea2ff4640 1 --2- 172.21.15.116:0/460572041 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5e8c0a2790 0x7f5e8c0a4c50 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.797+0000 7f5ea2ff4640 1 --2- 172.21.15.116:0/460572041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e9c0ffb00 0x7f5e9c19e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.797+0000 7f5ea2ff4640 1 -- 172.21.15.116:0/460572041 >> 172.21.15.116:0/460572041 conn(0x7f5e9c0fb470 msgr2=0x7f5e9c1008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.797+0000 7f5ea2ff4640 1 -- 172.21.15.116:0/460572041 shutdown_connections 2024-03-19T01:25:05.800 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:05.797+0000 7f5ea2ff4640 1 -- 172.21.15.116:0/460572041 wait complete. 2024-03-19T01:25:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 740.0 s, 1198.4 tps, lat 83.115 ms stddev 101.608 2024-03-19T01:25:07.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:07 smithi156 ceph-mon[66799]: pgmap v726: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.1 MiB/s wr, 655 op/s 2024-03-19T01:25:07.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:07 smithi162 ceph-mon[67178]: pgmap v726: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.1 MiB/s wr, 655 op/s 2024-03-19T01:25:07.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:07 smithi116 ceph-mon[62701]: pgmap v726: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.1 MiB/s wr, 655 op/s 2024-03-19T01:25:09.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:09 smithi156 ceph-mon[66799]: pgmap v727: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.9 MiB/s wr, 622 op/s 2024-03-19T01:25:09.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:09 smithi162 ceph-mon[67178]: pgmap v727: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.9 MiB/s wr, 622 op/s 2024-03-19T01:25:09.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:09 smithi116 ceph-mon[62701]: pgmap v727: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 5.9 MiB/s wr, 622 op/s 2024-03-19T01:25:11.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:11 smithi156 ceph-mon[66799]: pgmap v728: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 MiB/s wr, 632 op/s 2024-03-19T01:25:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 745.0 s, 1368.6 tps, lat 73.882 ms stddev 77.741 2024-03-19T01:25:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:11 smithi162 ceph-mon[67178]: pgmap v728: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 MiB/s wr, 632 op/s 2024-03-19T01:25:11.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:11 smithi116 ceph-mon[62701]: pgmap v728: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 MiB/s wr, 632 op/s 2024-03-19T01:25:13.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:13 smithi156 ceph-mon[66799]: pgmap v729: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 MiB/s wr, 653 op/s 2024-03-19T01:25:13.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:13 smithi162 ceph-mon[67178]: pgmap v729: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 MiB/s wr, 653 op/s 2024-03-19T01:25:13.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:13 smithi116 ceph-mon[62701]: pgmap v729: 97 pgs: 97 active+clean; 1.6 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 MiB/s wr, 653 op/s 2024-03-19T01:25:14.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:14 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:25:14.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:14 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:25:14.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:14 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:25:15.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:15 smithi156 ceph-mon[66799]: pgmap v730: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.5 MiB/s wr, 586 op/s 2024-03-19T01:25:15.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:15 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:25:15.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:15 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:25:15.550 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:15 smithi162 ceph-mon[67178]: pgmap v730: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.5 MiB/s wr, 586 op/s 2024-03-19T01:25:15.550 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:15 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:25:15.550 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:15 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:25:15.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:15 smithi116 ceph-mon[62701]: pgmap v730: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.5 MiB/s wr, 586 op/s 2024-03-19T01:25:15.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:15 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:25:15.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:15 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:25:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 750.0 s, 1126.0 tps, lat 88.225 ms stddev 95.209 2024-03-19T01:25:16.563 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:16 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:25:16.563 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:16 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:25:16.563 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:16 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:25:16.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:16 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:25:16.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:16 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:25:16.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:16 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:25:16.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:16 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:25:16.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:16 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:25:16.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:16 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:25:17.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:17 smithi162 ceph-mon[67178]: pgmap v731: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.1 MiB/s wr, 653 op/s 2024-03-19T01:25:17.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:17 smithi116 ceph-mon[62701]: pgmap v731: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.1 MiB/s wr, 653 op/s 2024-03-19T01:25:17.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:17 smithi156 ceph-mon[66799]: pgmap v731: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.1 MiB/s wr, 653 op/s 2024-03-19T01:25:18.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.937+0000 7f0474fb2640 1 -- 172.21.15.116:0/1860515220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0470108820 msgr2=0x7f047010ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:18.939 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.937+0000 7f0474fb2640 1 --2- 172.21.15.116:0/1860515220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0470108820 0x7f047010ac50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f0468009ca0 tx=0x7f046802c140 comp rx=0 tx=0).stop 2024-03-19T01:25:18.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.940+0000 7f0474fb2640 1 -- 172.21.15.116:0/1860515220 shutdown_connections 2024-03-19T01:25:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.940+0000 7f0474fb2640 1 --2- 172.21.15.116:0/1860515220 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f047010db40 0x7f047010dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.940+0000 7f0474fb2640 1 --2- 172.21.15.116:0/1860515220 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f047010b190 0x7f047010d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.940+0000 7f0474fb2640 1 --2- 172.21.15.116:0/1860515220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0470108820 0x7f047010ac50 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.940+0000 7f0474fb2640 1 -- 172.21.15.116:0/1860515220 >> 172.21.15.116:0/1860515220 conn(0x7f047006e860 msgr2=0x7f0470070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.945+0000 7f0474fb2640 1 -- 172.21.15.116:0/1860515220 shutdown_connections 2024-03-19T01:25:18.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.949+0000 7f0474fb2640 1 -- 172.21.15.116:0/1860515220 wait complete. 2024-03-19T01:25:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.949+0000 7f0474fb2640 1 Processor -- start 2024-03-19T01:25:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.950+0000 7f0474fb2640 1 -- start start 2024-03-19T01:25:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.951+0000 7f0474fb2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0470108820 0x7f0470119aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.951+0000 7f0474fb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047010b190 0x7f0470119fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.951+0000 7f0474fb2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f047010db40 0x7f0470118120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.951+0000 7f0474fb2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f04701109a0 con 0x7f047010b190 2024-03-19T01:25:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.951+0000 7f0474fb2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0470110820 con 0x7f0470108820 2024-03-19T01:25:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.951+0000 7f0474fb2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f0470110b20 con 0x7f047010db40 2024-03-19T01:25:18.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.951+0000 7f046ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f047010db40 0x7f0470118120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.951+0000 7f046ed9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f047010db40 0x7f0470118120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38822/0 (socket says 172.21.15.116:38822) 2024-03-19T01:25:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.951+0000 7f046ed9d640 1 -- 172.21.15.116:0/1559268185 learned_addr learned my addr 172.21.15.116:0/1559268185 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.951+0000 7f046e59c640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0470108820 0x7f0470119aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:18.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.952+0000 7f046dd9b640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047010b190 0x7f0470119fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.954+0000 7f046ed9d640 1 -- 172.21.15.116:0/1559268185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0470108820 msgr2=0x7f0470119aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.954+0000 7f046ed9d640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0470108820 0x7f0470119aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.954+0000 7f046ed9d640 1 -- 172.21.15.116:0/1559268185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047010b190 msgr2=0x7f0470119fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.954+0000 7f046ed9d640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047010b190 0x7f0470119fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.955 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.954+0000 7f046ed9d640 1 -- 172.21.15.116:0/1559268185 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04701188f0 con 0x7f047010db40 2024-03-19T01:25:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.955+0000 7f046ed9d640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f047010db40 0x7f0470118120 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f046400b3f0 tx=0x7f046400b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.955+0000 7f045f7fe640 1 -- 172.21.15.116:0/1559268185 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0464015020 con 0x7f047010db40 2024-03-19T01:25:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.955+0000 7f045f7fe640 1 -- 172.21.15.116:0/1559268185 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0464011d10 con 0x7f047010db40 2024-03-19T01:25:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.956+0000 7f0474fb2640 1 -- 172.21.15.116:0/1559268185 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0470118be0 con 0x7f047010db40 2024-03-19T01:25:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.956+0000 7f045f7fe640 1 -- 172.21.15.116:0/1559268185 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046400f2e0 con 0x7f047010db40 2024-03-19T01:25:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.956+0000 7f0474fb2640 1 -- 172.21.15.116:0/1559268185 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0470079b00 con 0x7f047010db40 2024-03-19T01:25:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.959+0000 7f0477b9e740 1 Processor -- start 2024-03-19T01:25:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.961+0000 7f045f7fe640 1 -- 172.21.15.116:0/1559268185 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f0464018020 con 0x7f047010db40 2024-03-19T01:25:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.961+0000 7f0477b9e740 1 -- start start 2024-03-19T01:25:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.961+0000 7f0477b9e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda2100 0x55da0cda2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:18.961 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.961+0000 7f0477b9e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2af0 0x55da0cda75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.961+0000 7f0477b9e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda7c60 0x55da0cdaa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.961+0000 7f0477b9e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55da0cd06be0 con 0x55da0cda2af0 2024-03-19T01:25:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.961+0000 7f0477b9e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55da0ccaf960 con 0x55da0cda2100 2024-03-19T01:25:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.961+0000 7f0477b9e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55da0cd02c80 con 0x55da0cda7c60 2024-03-19T01:25:18.962 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.962+0000 7f045f7fe640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f04580a2ae0 0x7f04580a4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.962+0000 7f046dd9b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda2100 0x55da0cda2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.962+0000 7f046e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2af0 0x55da0cda75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.962+0000 7f046e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2af0 0x55da0cda75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56762/0 (socket says 172.21.15.116:56762) 2024-03-19T01:25:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.962+0000 7f046e59c640 1 -- 172.21.15.116:0/1174279586 learned_addr learned my addr 172.21.15.116:0/1174279586 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:18.963 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.962+0000 7f046dd9b640 1 --2- 172.21.15.116:0/1174279586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda7c60 0x55da0cdaa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.962+0000 7f045f7fe640 1 -- 172.21.15.116:0/1559268185 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f04640ce060 con 0x7f047010db40 2024-03-19T01:25:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.963+0000 7f046ed9d640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f04580a2ae0 0x7f04580a4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.963+0000 7f046dd9b640 1 -- 172.21.15.116:0/1174279586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda7c60 msgr2=0x55da0cdaa0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.963+0000 7f046dd9b640 1 --2- 172.21.15.116:0/1174279586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda7c60 0x55da0cdaa0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.963+0000 7f046dd9b640 1 -- 172.21.15.116:0/1174279586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2af0 msgr2=0x55da0cda75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.963+0000 7f046dd9b640 1 --2- 172.21.15.116:0/1174279586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2af0 0x55da0cda75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.964 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.963+0000 7f046dd9b640 1 -- 172.21.15.116:0/1174279586 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da0cca6d20 con 0x55da0cda2100 2024-03-19T01:25:18.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.963+0000 7f046ed9d640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f04580a2ae0 0x7f04580a4fa0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f046400b3c0 tx=0x7f0464011f30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:18.965 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.964+0000 7f046dd9b640 1 --2- 172.21.15.116:0/1174279586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda2100 0x55da0cda2520 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0460006fd0 tx=0x7f04600090e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be845f4acf09bd4b server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.966+0000 7f045cff9640 1 -- 172.21.15.116:0/1174279586 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460015580 con 0x55da0cda2100 2024-03-19T01:25:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.966+0000 7f045cff9640 1 -- 172.21.15.116:0/1174279586 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f046001d070 con 0x55da0cda2100 2024-03-19T01:25:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.966+0000 7f045cff9640 1 -- 172.21.15.116:0/1174279586 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460018650 con 0x55da0cda2100 2024-03-19T01:25:18.967 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.967+0000 7f0477b9e740 1 -- 172.21.15.116:0/1174279586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda2100 msgr2=0x55da0cda2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.967+0000 7f0477b9e740 1 --2- 172.21.15.116:0/1174279586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda2100 0x55da0cda2520 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0460006fd0 tx=0x7f04600090e0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.967+0000 7f0477b9e740 1 -- 172.21.15.116:0/1174279586 shutdown_connections 2024-03-19T01:25:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.967+0000 7f0477b9e740 1 --2- 172.21.15.116:0/1174279586 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda7c60 0x55da0cdaa0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.967+0000 7f0477b9e740 1 --2- 172.21.15.116:0/1174279586 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2af0 0x55da0cda75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.967+0000 7f0477b9e740 1 --2- 172.21.15.116:0/1174279586 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda2100 0x55da0cda2520 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.967+0000 7f0477b9e740 1 -- 172.21.15.116:0/1174279586 >> 172.21.15.116:0/1174279586 conn(0x55da0cd9d670 msgr2=0x55da0cd9da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:18.968 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.968+0000 7f0477b9e740 1 -- 172.21.15.116:0/1174279586 shutdown_connections 2024-03-19T01:25:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.968+0000 7f0477b9e740 1 -- 172.21.15.116:0/1174279586 wait complete. 2024-03-19T01:25:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f0477b9e740 1 -- start start 2024-03-19T01:25:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f0477b9e740 1 Processor -- start 2024-03-19T01:25:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f0477b9e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 0x55da0cdac130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f0477b9e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda2af0 0x55da0cdac670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:18.969 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f0477b9e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda7c60 0x55da0cdb36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f0477b9e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55da0cdaabb0 con 0x55da0cda7c60 2024-03-19T01:25:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f0477b9e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55da0ccc2db0 con 0x55da0cda2af0 2024-03-19T01:25:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f0477b9e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55da0cdaad30 con 0x55da0cda2100 2024-03-19T01:25:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f0477b9e740 10 client.0 _ll_register_callbacks cb 0x55da0ccc7e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:25:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f046dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 0x55da0cdac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f046dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 0x55da0cdac130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38836/0 (socket says 172.21.15.116:38836) 2024-03-19T01:25:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f046dd9b640 1 -- 172.21.15.116:0/795700691 learned_addr learned my addr 172.21.15.116:0/795700691 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:18.970 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f046dd9b640 1 --2- 172.21.15.116:0/795700691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda7c60 0x55da0cdb36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.969+0000 7f046e59c640 1 --2- 172.21.15.116:0/795700691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda2af0 0x55da0cdac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.970+0000 7f046dd9b640 1 -- 172.21.15.116:0/795700691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda2af0 msgr2=0x55da0cdac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.970+0000 7f046dd9b640 1 --2- 172.21.15.116:0/795700691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda2af0 0x55da0cdac670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.970+0000 7f046dd9b640 1 -- 172.21.15.116:0/795700691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda7c60 msgr2=0x55da0cdb36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.970+0000 7f046dd9b640 1 --2- 172.21.15.116:0/795700691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda7c60 0x55da0cdb36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.971 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.970+0000 7f046dd9b640 1 -- 172.21.15.116:0/795700691 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da0ccc1c60 con 0x55da0cda2100 2024-03-19T01:25:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.970+0000 7f046dd9b640 1 --2- 172.21.15.116:0/795700691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 0x55da0cdac130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0460009d10 tx=0x7f046000f040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5009cd4226df73c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.970+0000 7f0455ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:25:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.970+0000 7f0455ffb640 1 -- 172.21.15.116:0/795700691 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460026070 con 0x55da0cda2100 2024-03-19T01:25:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.970+0000 7f0477b9e740 1 -- 172.21.15.116:0/795700691 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f046001d070 con 0x55da0cda2100 2024-03-19T01:25:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.970+0000 7f0455ffb640 1 -- 172.21.15.116:0/795700691 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f0460004020 con 0x55da0cda2100 2024-03-19T01:25:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.971+0000 7f0455ffb640 1 -- 172.21.15.116:0/795700691 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f0460004220 con 0x55da0cda2100 2024-03-19T01:25:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.971+0000 7f0477b9e740 10 client.26098 fetch_fsmap learned FSMap version 47 2024-03-19T01:25:18.972 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.971+0000 7f0477b9e740 1 -- 172.21.15.116:0/795700691 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55da0ca1b210 con 0x55da0cda2100 2024-03-19T01:25:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.972+0000 7f0455ffb640 1 -- 172.21.15.116:0/795700691 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f046001d070 con 0x55da0cda2100 2024-03-19T01:25:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.972+0000 7f0455ffb640 1 -- 172.21.15.116:0/795700691 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f0460015050 con 0x55da0cda2100 2024-03-19T01:25:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.972+0000 7f0477b9e740 10 client.26098 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:25:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.972+0000 7f0477b9e740 10 client.26098 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:25:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.972+0000 7f0477b9e740 20 client.26098 populate_metadata read hostname 'smithi116' 2024-03-19T01:25:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.972+0000 7f0477b9e740 1 --2- 172.21.15.116:0/795700691 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55da0cdb7dd0 0x55da0cdba1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.972+0000 7f0477b9e740 4 client.26098 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:25:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.972+0000 7f0477b9e740 1 -- 172.21.15.116:0/795700691 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da0cd14120 con 0x55da0cdb7dd0 2024-03-19T01:25:18.973 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.973+0000 7f046e59c640 1 --2- 172.21.15.116:0/795700691 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55da0cdb7dd0 0x55da0cdba1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.974+0000 7f0455ffb640 10 client.26098 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:18.975 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.974+0000 7f046e59c640 1 --2- 172.21.15.116:0/795700691 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55da0cdb7dd0 0x55da0cdba1f0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48a06e5816d83e88 server_cookie=31c0d1098cb4ff94 in_seq=0 out_seq=0 2024-03-19T01:25:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.977+0000 7f0455ffb640 1 -- 172.21.15.116:0/795700691 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55da0cd14120 con 0x55da0cdb7dd0 2024-03-19T01:25:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.977+0000 7f0455ffb640 10 client.26098 handle_command_reply: tid=0 2024-03-19T01:25:18.977 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.977+0000 7f0477b9e740 1 client.26098 shutdown 2024-03-19T01:25:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.977+0000 7f0477b9e740 1 -- 172.21.15.116:0/795700691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 msgr2=0x55da0cdac130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.977+0000 7f0477b9e740 1 --2- 172.21.15.116:0/795700691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 0x55da0cdac130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0460009d10 tx=0x7f046000f040 comp rx=0 tx=0).stop 2024-03-19T01:25:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.977+0000 7f0477b9e740 1 -- 172.21.15.116:0/795700691 shutdown_connections 2024-03-19T01:25:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.977+0000 7f0477b9e740 1 --2- 172.21.15.116:0/795700691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda7c60 0x55da0cdb36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.978 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.977+0000 7f0477b9e740 1 --2- 172.21.15.116:0/795700691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda2af0 0x55da0cdac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.977+0000 7f0477b9e740 1 --2- 172.21.15.116:0/795700691 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55da0cdb7dd0 0x55da0cdba1f0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.978+0000 7f0477b9e740 1 --2- 172.21.15.116:0/795700691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 0x55da0cdac130 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.978+0000 7f0477b9e740 1 -- 172.21.15.116:0/795700691 >> 172.21.15.116:0/795700691 conn(0x55da0cd9d670 msgr2=0x55da0cd9e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.978+0000 7f0455ffb640 0 client.26098 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.978+0000 7f0477b9e740 1 -- 172.21.15.116:0/795700691 shutdown_connections 2024-03-19T01:25:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.978+0000 7f0477b9e740 1 -- 172.21.15.116:0/795700691 wait complete. 2024-03-19T01:25:18.979 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.978+0000 7f0477b9e740 20 client.26098 trim_cache size 0 max 16384 2024-03-19T01:25:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.998+0000 7f0477b9e740 1 Processor -- start 2024-03-19T01:25:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:18.999+0000 7f0477b9e740 1 -- start start 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f0477b9e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2100 0x55da0cdac450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f0477b9e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2af0 0x55da0cdb3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f0477b9e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda7c60 0x55da0cda3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f0477b9e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f045000e520 con 0x55da0cda2100 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f0477b9e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0450010120 con 0x55da0cda7c60 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f0477b9e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f045000ebf0 con 0x55da0cda2af0 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f046dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2100 0x55da0cdac450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f046e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2af0 0x55da0cdb3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f046e59c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2af0 0x55da0cdb3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38842/0 (socket says 172.21.15.116:38842) 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f046e59c640 1 -- 172.21.15.116:0/3183977235 learned_addr learned my addr 172.21.15.116:0/3183977235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f046dd9b640 1 --2- 172.21.15.116:0/3183977235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda7c60 0x55da0cda3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f046e59c640 1 -- 172.21.15.116:0/3183977235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda7c60 msgr2=0x55da0cda3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f046e59c640 1 --2- 172.21.15.116:0/3183977235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda7c60 0x55da0cda3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f046e59c640 1 -- 172.21.15.116:0/3183977235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2100 msgr2=0x55da0cdac450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f046e59c640 1 --2- 172.21.15.116:0/3183977235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2100 0x55da0cdac450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f046e59c640 1 -- 172.21.15.116:0/3183977235 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da0cae7620 con 0x55da0cda2af0 2024-03-19T01:25:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f046dd9b640 1 --2- 172.21.15.116:0/3183977235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda7c60 0x55da0cda3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:25:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.000+0000 7f046e59c640 1 --2- 172.21.15.116:0/3183977235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2af0 0x55da0cdb3670 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0468006010 tx=0x7f046802c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16de6620ea8cc1a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.001+0000 7f0457fff640 1 -- 172.21.15.116:0/3183977235 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0468039040 con 0x55da0cda2af0 2024-03-19T01:25:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.001+0000 7f0457fff640 1 -- 172.21.15.116:0/3183977235 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0468032df0 con 0x55da0cda2af0 2024-03-19T01:25:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.001+0000 7f0457fff640 1 -- 172.21.15.116:0/3183977235 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0468004460 con 0x55da0cda2af0 2024-03-19T01:25:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.002+0000 7f0477b9e740 1 -- 172.21.15.116:0/3183977235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2af0 msgr2=0x55da0cdb3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.002+0000 7f0477b9e740 1 --2- 172.21.15.116:0/3183977235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2af0 0x55da0cdb3670 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0468006010 tx=0x7f046802c710 comp rx=0 tx=0).stop 2024-03-19T01:25:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.002+0000 7f0477b9e740 1 -- 172.21.15.116:0/3183977235 shutdown_connections 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.002+0000 7f0477b9e740 1 --2- 172.21.15.116:0/3183977235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda7c60 0x55da0cda3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.002+0000 7f0477b9e740 1 --2- 172.21.15.116:0/3183977235 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2af0 0x55da0cdb3670 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.002+0000 7f0477b9e740 1 --2- 172.21.15.116:0/3183977235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2100 0x55da0cdac450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.002+0000 7f0477b9e740 1 -- 172.21.15.116:0/3183977235 >> 172.21.15.116:0/3183977235 conn(0x55da0cd9d670 msgr2=0x55da0cdabd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.002+0000 7f0477b9e740 1 -- 172.21.15.116:0/3183977235 shutdown_connections 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.002+0000 7f0477b9e740 1 -- 172.21.15.116:0/3183977235 wait complete. 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.002+0000 7f0477b9e740 1 -- start start 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f0477b9e740 1 Processor -- start 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f0477b9e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 0x55da0cdb10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f0477b9e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2af0 0x55da0cdb1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f0477b9e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda7c60 0x55da0cdb2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f0477b9e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f045000fa50 con 0x55da0cda2af0 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f0477b9e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f045000f380 con 0x55da0cda7c60 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f0477b9e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f045000c500 con 0x55da0cda2100 2024-03-19T01:25:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f046dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 0x55da0cdb10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f0477b9e740 10 client.0 _ll_register_callbacks cb 0x7f0460009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:25:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f046dd9b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 0x55da0cdb10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38854/0 (socket says 172.21.15.116:38854) 2024-03-19T01:25:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f046dd9b640 1 -- 172.21.15.116:0/3058319242 learned_addr learned my addr 172.21.15.116:0/3058319242 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f046e59c640 1 --2- 172.21.15.116:0/3058319242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2af0 0x55da0cdb1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f046dd9b640 1 --2- 172.21.15.116:0/3058319242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda7c60 0x55da0cdb2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f046dd9b640 1 -- 172.21.15.116:0/3058319242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda7c60 msgr2=0x55da0cdb2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f046dd9b640 1 --2- 172.21.15.116:0/3058319242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda7c60 0x55da0cdb2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f046dd9b640 1 -- 172.21.15.116:0/3058319242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2af0 msgr2=0x55da0cdb1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f046dd9b640 1 --2- 172.21.15.116:0/3058319242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2af0 0x55da0cdb1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.003+0000 7f046dd9b640 1 -- 172.21.15.116:0/3058319242 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da0cdb2b60 con 0x55da0cda2100 2024-03-19T01:25:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.004+0000 7f046dd9b640 1 --2- 172.21.15.116:0/3058319242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 0x55da0cdb10c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0460004790 tx=0x7f0460006d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bddfd49e6c836145 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.004+0000 7f0455ffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:25:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.005+0000 7f0455ffb640 1 -- 172.21.15.116:0/3058319242 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460026040 con 0x55da0cda2100 2024-03-19T01:25:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.005+0000 7f0455ffb640 1 -- 172.21.15.116:0/3058319242 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f046002fc60 con 0x55da0cda2100 2024-03-19T01:25:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.005+0000 7f0477b9e740 1 -- 172.21.15.116:0/3058319242 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0468032df0 con 0x55da0cda2100 2024-03-19T01:25:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.005+0000 7f0455ffb640 1 -- 172.21.15.116:0/3058319242 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f046002f7f0 con 0x55da0cda2100 2024-03-19T01:25:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.005+0000 7f0477b9e740 10 client.26104 fetch_fsmap learned FSMap version 47 2024-03-19T01:25:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.005+0000 7f0477b9e740 1 -- 172.21.15.116:0/3058319242 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55da0ce1e360 con 0x55da0cda2100 2024-03-19T01:25:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.006+0000 7f0455ffb640 1 -- 172.21.15.116:0/3058319242 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0468032df0 con 0x55da0cda2100 2024-03-19T01:25:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.006+0000 7f0455ffb640 1 -- 172.21.15.116:0/3058319242 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f046003e440 con 0x55da0cda2100 2024-03-19T01:25:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.006+0000 7f0477b9e740 10 client.26104 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:25:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.006+0000 7f0477b9e740 10 client.26104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:25:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.006+0000 7f0477b9e740 20 client.26104 populate_metadata read hostname 'smithi116' 2024-03-19T01:25:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.006+0000 7f0477b9e740 1 --2- 172.21.15.116:0/3058319242 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55da0ce1ec80 0x55da0ce210a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.006+0000 7f0477b9e740 4 client.26104 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:25:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.006+0000 7f0477b9e740 1 -- 172.21.15.116:0/3058319242 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55da0cd0e470 con 0x55da0ce1ec80 2024-03-19T01:25:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.006+0000 7f046e59c640 1 --2- 172.21.15.116:0/3058319242 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55da0ce1ec80 0x55da0ce210a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.007+0000 7f0455ffb640 10 client.26104 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.007+0000 7f046e59c640 1 --2- 172.21.15.116:0/3058319242 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55da0ce1ec80 0x55da0ce210a0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5483310410029471 server_cookie=c52fe6b002e819a1 in_seq=0 out_seq=0 2024-03-19T01:25:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.015+0000 7f0455ffb640 1 -- 172.21.15.116:0/3058319242 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55da0cd0e470 con 0x55da0ce1ec80 2024-03-19T01:25:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.015+0000 7f0455ffb640 10 client.26104 handle_command_reply: tid=0 2024-03-19T01:25:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.015+0000 7f0477b9e740 1 client.26104 shutdown 2024-03-19T01:25:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.015+0000 7f0477b9e740 1 -- 172.21.15.116:0/3058319242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 msgr2=0x55da0cdb10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.015+0000 7f0477b9e740 1 --2- 172.21.15.116:0/3058319242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 0x55da0cdb10c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0460004790 tx=0x7f0460006d10 comp rx=0 tx=0).stop 2024-03-19T01:25:19.016 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.015+0000 7f0477b9e740 1 -- 172.21.15.116:0/3058319242 shutdown_connections 2024-03-19T01:25:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.015+0000 7f0477b9e740 1 --2- 172.21.15.116:0/3058319242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55da0cda7c60 0x55da0cdb2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.015+0000 7f0477b9e740 1 --2- 172.21.15.116:0/3058319242 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55da0cda2af0 0x55da0cdb1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.015+0000 7f0477b9e740 1 --2- 172.21.15.116:0/3058319242 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55da0ce1ec80 0x55da0ce210a0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.015+0000 7f0477b9e740 1 --2- 172.21.15.116:0/3058319242 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55da0cda2100 0x55da0cdb10c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.015+0000 7f0455ffb640 0 client.26104 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.015+0000 7f0477b9e740 1 -- 172.21.15.116:0/3058319242 >> 172.21.15.116:0/3058319242 conn(0x55da0cd9d670 msgr2=0x55da0cda08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.015+0000 7f0477b9e740 1 -- 172.21.15.116:0/3058319242 shutdown_connections 2024-03-19T01:25:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0477b9e740 1 -- 172.21.15.116:0/3058319242 wait complete. 2024-03-19T01:25:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0477b9e740 20 client.26104 trim_cache size 0 max 16384 2024-03-19T01:25:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:25:19.017 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:25:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:25:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:25:19.018 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0474fb2640 1 -- 172.21.15.116:0/1559268185 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f04580a2ae0 msgr2=0x7f04580a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0474fb2640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f04580a2ae0 0x7f04580a4fa0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f046400b3c0 tx=0x7f0464011f30 comp rx=0 tx=0).stop 2024-03-19T01:25:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0474fb2640 1 -- 172.21.15.116:0/1559268185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f047010db40 msgr2=0x7f0470118120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0474fb2640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f047010db40 0x7f0470118120 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f046400b3f0 tx=0x7f046400b8b0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0474fb2640 1 -- 172.21.15.116:0/1559268185 shutdown_connections 2024-03-19T01:25:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0474fb2640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f047010db40 0x7f0470118120 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0474fb2640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f047010b190 0x7f0470119fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0474fb2640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f04580a2ae0 0x7f04580a4fa0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0474fb2640 1 --2- 172.21.15.116:0/1559268185 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0470108820 0x7f0470119aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0474fb2640 1 -- 172.21.15.116:0/1559268185 >> 172.21.15.116:0/1559268185 conn(0x7f047006e860 msgr2=0x7f0470108ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0474fb2640 1 -- 172.21.15.116:0/1559268185 shutdown_connections 2024-03-19T01:25:19.019 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:19.016+0000 7f0474fb2640 1 -- 172.21.15.116:0/1559268185 wait complete. 2024-03-19T01:25:19.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:19 smithi162 ceph-mon[67178]: pgmap v732: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 5.3 MiB/s wr, 590 op/s 2024-03-19T01:25:19.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:19 smithi116 ceph-mon[62701]: pgmap v732: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 5.3 MiB/s wr, 590 op/s 2024-03-19T01:25:19.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:19 smithi156 ceph-mon[66799]: pgmap v732: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 5.3 MiB/s wr, 590 op/s 2024-03-19T01:25:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 755.0 s, 2079.6 tps, lat 48.196 ms stddev 56.150 2024-03-19T01:25:21.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:21 smithi162 ceph-mon[67178]: pgmap v733: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 MiB/s wr, 599 op/s 2024-03-19T01:25:21.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:21 smithi116 ceph-mon[62701]: pgmap v733: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 MiB/s wr, 599 op/s 2024-03-19T01:25:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:21 smithi156 ceph-mon[66799]: pgmap v733: 97 pgs: 97 active+clean; 1.6 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.7 MiB/s wr, 599 op/s 2024-03-19T01:25:23.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:23 smithi162 ceph-mon[67178]: pgmap v734: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.7 MiB/s wr, 622 op/s 2024-03-19T01:25:23.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:23 smithi116 ceph-mon[62701]: pgmap v734: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.7 MiB/s wr, 622 op/s 2024-03-19T01:25:23.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:23 smithi156 ceph-mon[66799]: pgmap v734: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.7 MiB/s wr, 622 op/s 2024-03-19T01:25:25.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:25 smithi162 ceph-mon[67178]: pgmap v735: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.2 MiB/s wr, 558 op/s 2024-03-19T01:25:25.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:25 smithi116 ceph-mon[62701]: pgmap v735: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.2 MiB/s wr, 558 op/s 2024-03-19T01:25:25.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:25 smithi156 ceph-mon[66799]: pgmap v735: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.2 MiB/s wr, 558 op/s 2024-03-19T01:25:26.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 760.0 s, 1847.2 tps, lat 54.222 ms stddev 63.593 2024-03-19T01:25:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:27 smithi162 ceph-mon[67178]: pgmap v736: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 660 op/s 2024-03-19T01:25:27.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:27 smithi116 ceph-mon[62701]: pgmap v736: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 660 op/s 2024-03-19T01:25:27.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:27 smithi156 ceph-mon[66799]: pgmap v736: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.3 MiB/s wr, 660 op/s 2024-03-19T01:25:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:29 smithi162 ceph-mon[67178]: pgmap v737: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 MiB/s wr, 646 op/s 2024-03-19T01:25:29.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:29 smithi116 ceph-mon[62701]: pgmap v737: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 MiB/s wr, 646 op/s 2024-03-19T01:25:29.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:29 smithi156 ceph-mon[66799]: pgmap v737: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 MiB/s wr, 646 op/s 2024-03-19T01:25:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 765.0 s, 2062.2 tps, lat 48.524 ms stddev 57.239 2024-03-19T01:25:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:31 smithi162 ceph-mon[67178]: pgmap v738: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.9 MiB/s wr, 604 op/s 2024-03-19T01:25:31.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:31 smithi116 ceph-mon[62701]: pgmap v738: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.9 MiB/s wr, 604 op/s 2024-03-19T01:25:31.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:31 smithi156 ceph-mon[66799]: pgmap v738: 97 pgs: 97 active+clean; 1.6 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.9 MiB/s wr, 604 op/s 2024-03-19T01:25:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.156+0000 7f2c52140640 1 -- 172.21.15.116:0/1237915117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c4c101220 msgr2=0x7f2c4c105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:32.157 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.156+0000 7f2c52140640 1 --2- 172.21.15.116:0/1237915117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c4c101220 0x7f2c4c105ac0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2c4401acb0 tx=0x7f2c4403d170 comp rx=0 tx=0).stop 2024-03-19T01:25:32.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.158+0000 7f2c52140640 1 -- 172.21.15.116:0/1237915117 shutdown_connections 2024-03-19T01:25:32.161 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.158+0000 7f2c52140640 1 --2- 172.21.15.116:0/1237915117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c4c101220 0x7f2c4c105ac0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.158+0000 7f2c52140640 1 --2- 172.21.15.116:0/1237915117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c4c100860 0x7f2c4c100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.158+0000 7f2c52140640 1 --2- 172.21.15.116:0/1237915117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c4c0ffb00 0x7f2c4c0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.158+0000 7f2c52140640 1 -- 172.21.15.116:0/1237915117 >> 172.21.15.116:0/1237915117 conn(0x7f2c4c0fb470 msgr2=0x7f2c4c0fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:32.162 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.162+0000 7f2c52140640 1 -- 172.21.15.116:0/1237915117 shutdown_connections 2024-03-19T01:25:32.163 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.163+0000 7f2c52140640 1 -- 172.21.15.116:0/1237915117 wait complete. 2024-03-19T01:25:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.163+0000 7f2c52140640 1 Processor -- start 2024-03-19T01:25:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c52140640 1 -- start start 2024-03-19T01:25:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c52140640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c4c0ffb00 0x7f2c4c06d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c52140640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c4c100860 0x7f2c4c06f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c52140640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c4c101220 0x7f2c4c071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c52140640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1093d0 con 0x7f2c4c100860 2024-03-19T01:25:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c52140640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c109250 con 0x7f2c4c101220 2024-03-19T01:25:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c52140640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c109550 con 0x7f2c4c0ffb00 2024-03-19T01:25:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c4affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c4c100860 0x7f2c4c06f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c4c101220 0x7f2c4c071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.166 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c4affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c4c100860 0x7f2c4c06f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51564/0 (socket says 172.21.15.116:51564) 2024-03-19T01:25:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c4b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c4c0ffb00 0x7f2c4c06d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c4c101220 0x7f2c4c071cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:53166/0 (socket says 172.21.15.116:53166) 2024-03-19T01:25:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c4affd640 1 -- 172.21.15.116:0/1066885828 learned_addr learned my addr 172.21.15.116:0/1066885828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c4bfff640 1 -- 172.21.15.116:0/1066885828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c4c0ffb00 msgr2=0x7f2c4c06d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c4bfff640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c4c0ffb00 0x7f2c4c06d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c4bfff640 1 -- 172.21.15.116:0/1066885828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c4c100860 msgr2=0x7f2c4c06f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c4bfff640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c4c100860 0x7f2c4c06f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.164+0000 7f2c4bfff640 1 -- 172.21.15.116:0/1066885828 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4c0723d0 con 0x7f2c4c101220 2024-03-19T01:25:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.165+0000 7f2c4b7fe640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c4c0ffb00 0x7f2c4c06d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:25:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.165+0000 7f2c4affd640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c4c100860 0x7f2c4c06f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:25:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.165+0000 7f2c4bfff640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c4c101220 0x7f2c4c071cd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2c4401ac00 tx=0x7f2c44003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:32.167 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.165+0000 7f2c48ff9640 1 -- 172.21.15.116:0/1066885828 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c44052020 con 0x7f2c4c101220 2024-03-19T01:25:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.165+0000 7f2c48ff9640 1 -- 172.21.15.116:0/1066885828 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c44003d90 con 0x7f2c4c101220 2024-03-19T01:25:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.165+0000 7f2c48ff9640 1 -- 172.21.15.116:0/1066885828 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4404a040 con 0x7f2c4c101220 2024-03-19T01:25:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.165+0000 7f2c52140640 1 -- 172.21.15.116:0/1066885828 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c4c072660 con 0x7f2c4c101220 2024-03-19T01:25:32.168 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.165+0000 7f2c52140640 1 -- 172.21.15.116:0/1066885828 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c4c072b50 con 0x7f2c4c101220 2024-03-19T01:25:32.169 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.169+0000 7f2c54dfd740 1 Processor -- start 2024-03-19T01:25:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.170+0000 7f2c48ff9640 1 -- 172.21.15.116:0/1066885828 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f2c44055020 con 0x7f2c4c101220 2024-03-19T01:25:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.170+0000 7f2c54dfd740 1 -- start start 2024-03-19T01:25:32.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.170+0000 7f2c48ff9640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2c340a28a0 0x7f2c340a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.170+0000 7f2c54dfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96a100 0x56424d96a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c54dfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96aaf0 0x56424d96f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c48ff9640 1 -- 172.21.15.116:0/1066885828 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2c44101550 con 0x7f2c4c101220 2024-03-19T01:25:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c54dfd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96fc60 0x56424d9720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c54dfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56424d8cebe0 con 0x56424d96a100 2024-03-19T01:25:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c54dfd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56424d877960 con 0x56424d96fc60 2024-03-19T01:25:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c54dfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56424d8cac80 con 0x56424d96aaf0 2024-03-19T01:25:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4affd640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2c340a28a0 0x7f2c340a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.172 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96aaf0 0x56424d96f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96fc60 0x56424d9720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96a100 0x56424d96a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96fc60 0x56424d9720f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:53178/0 (socket says 172.21.15.116:53178) 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4bfff640 1 -- 172.21.15.116:0/332400992 learned_addr learned my addr 172.21.15.116:0/332400992 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96aaf0 0x56424d96f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48372/0 (socket says 172.21.15.116:48372) 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4bfff640 1 -- 172.21.15.116:0/332400992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96aaf0 msgr2=0x56424d96f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4affd640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2c340a28a0 0x7f2c340a4d60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c111190 tx=0x7f2c400159a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4bfff640 1 --2- 172.21.15.116:0/332400992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96aaf0 0x56424d96f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4bfff640 1 -- 172.21.15.116:0/332400992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96a100 msgr2=0x56424d96a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4bfff640 1 --2- 172.21.15.116:0/332400992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96a100 0x56424d96a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4bfff640 1 -- 172.21.15.116:0/332400992 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56424d86ed20 con 0x56424d96fc60 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4b7fe640 1 --2- 172.21.15.116:0/332400992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96a100 0x56424d96a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.171+0000 7f2c4bfff640 1 --2- 172.21.15.116:0/332400992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96fc60 0x56424d9720f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2c440ca3e0 tx=0x7f2c44045440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f643d5e1e6124812 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:32.173 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.172+0000 7f2c39ffb640 1 -- 172.21.15.116:0/332400992 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c440d2020 con 0x56424d96fc60 2024-03-19T01:25:32.174 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.172+0000 7f2c39ffb640 1 -- 172.21.15.116:0/332400992 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c440d6670 con 0x56424d96fc60 2024-03-19T01:25:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.172+0000 7f2c39ffb640 1 -- 172.21.15.116:0/332400992 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c440d6440 con 0x56424d96fc60 2024-03-19T01:25:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.173+0000 7f2c54dfd740 1 -- 172.21.15.116:0/332400992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96fc60 msgr2=0x56424d9720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.173+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/332400992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96fc60 0x56424d9720f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2c440ca3e0 tx=0x7f2c44045440 comp rx=0 tx=0).stop 2024-03-19T01:25:32.176 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.173+0000 7f2c54dfd740 1 -- 172.21.15.116:0/332400992 shutdown_connections 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.173+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/332400992 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96fc60 0x56424d9720f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.173+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/332400992 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96aaf0 0x56424d96f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.173+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/332400992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96a100 0x56424d96a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.173+0000 7f2c54dfd740 1 -- 172.21.15.116:0/332400992 >> 172.21.15.116:0/332400992 conn(0x56424d965670 msgr2=0x56424d965a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.173+0000 7f2c54dfd740 1 -- 172.21.15.116:0/332400992 shutdown_connections 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.173+0000 7f2c54dfd740 1 -- 172.21.15.116:0/332400992 wait complete. 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.174+0000 7f2c54dfd740 1 -- start start 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.174+0000 7f2c54dfd740 1 Processor -- start 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.175+0000 7f2c54dfd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96a100 0x56424d973f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.175+0000 7f2c54dfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96aaf0 0x56424d9744c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.175+0000 7f2c54dfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 0x56424d97b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.175+0000 7f2c4b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96a100 0x56424d973f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.175+0000 7f2c4b7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96a100 0x56424d973f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:53194/0 (socket says 172.21.15.116:53194) 2024-03-19T01:25:32.177 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.175+0000 7f2c4affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96aaf0 0x56424d9744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.175+0000 7f2c4affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96aaf0 0x56424d9744c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48382/0 (socket says 172.21.15.116:48382) 2024-03-19T01:25:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.175+0000 7f2c4b7fe640 1 -- 172.21.15.116:0/315461092 learned_addr learned my addr 172.21.15.116:0/315461092 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.176+0000 7f2c54dfd740 1 -- 172.21.15.116:0/315461092 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56424d972ae0 con 0x56424d96fc60 2024-03-19T01:25:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.176+0000 7f2c54dfd740 1 -- 172.21.15.116:0/315461092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56424d88adb0 con 0x56424d96a100 2024-03-19T01:25:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.176+0000 7f2c54dfd740 1 -- 172.21.15.116:0/315461092 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56424d972c60 con 0x56424d96aaf0 2024-03-19T01:25:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.176+0000 7f2c4bfff640 1 --2- 172.21.15.116:0/315461092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 0x56424d97b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.178 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.176+0000 7f2c54dfd740 10 client.0 _ll_register_callbacks cb 0x56424d88fe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:25:32.180 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.177+0000 7f2c4b7fe640 1 -- 172.21.15.116:0/315461092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96aaf0 msgr2=0x56424d9744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.177+0000 7f2c4b7fe640 1 --2- 172.21.15.116:0/315461092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96aaf0 0x56424d9744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.177+0000 7f2c4b7fe640 1 -- 172.21.15.116:0/315461092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 msgr2=0x56424d97b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.177+0000 7f2c4b7fe640 1 --2- 172.21.15.116:0/315461092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 0x56424d97b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.177+0000 7f2c4b7fe640 1 -- 172.21.15.116:0/315461092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56424d889c60 con 0x56424d96a100 2024-03-19T01:25:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.177+0000 7f2c4bfff640 1 --2- 172.21.15.116:0/315461092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 0x56424d97b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:25:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.177+0000 7f2c4b7fe640 1 --2- 172.21.15.116:0/315461092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96a100 0x56424d973f80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c011570 tx=0x7f2c3c011a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e12c0b84a0a3dae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.177+0000 7f2c2effd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:25:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.177+0000 7f2c2effd640 1 -- 172.21.15.116:0/315461092 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c015870 con 0x56424d96a100 2024-03-19T01:25:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.177+0000 7f2c54dfd740 1 -- 172.21.15.116:0/315461092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c440d6670 con 0x56424d96a100 2024-03-19T01:25:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.178+0000 7f2c2effd640 1 -- 172.21.15.116:0/315461092 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2c3c015a10 con 0x56424d96a100 2024-03-19T01:25:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.178+0000 7f2c2effd640 1 -- 172.21.15.116:0/315461092 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f2c3c015c10 con 0x56424d96a100 2024-03-19T01:25:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.178+0000 7f2c54dfd740 10 client.26123 fetch_fsmap learned FSMap version 47 2024-03-19T01:25:32.181 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.178+0000 7f2c54dfd740 1 -- 172.21.15.116:0/315461092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x56424d5e3210 con 0x56424d96a100 2024-03-19T01:25:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.178+0000 7f2c2effd640 1 -- 172.21.15.116:0/315461092 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c440d6670 con 0x56424d96a100 2024-03-19T01:25:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.178+0000 7f2c2effd640 1 -- 172.21.15.116:0/315461092 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f2c3c00f580 con 0x56424d96a100 2024-03-19T01:25:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.178+0000 7f2c54dfd740 10 client.26123 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:25:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.178+0000 7f2c54dfd740 10 client.26123 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:25:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.178+0000 7f2c54dfd740 20 client.26123 populate_metadata read hostname 'smithi116' 2024-03-19T01:25:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.178+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/315461092 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56424d97fde0 0x56424d982250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.178+0000 7f2c54dfd740 4 client.26123 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:25:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.178+0000 7f2c54dfd740 1 -- 172.21.15.116:0/315461092 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56424d8dc120 con 0x56424d97fde0 2024-03-19T01:25:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.179+0000 7f2c4affd640 1 --2- 172.21.15.116:0/315461092 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56424d97fde0 0x56424d982250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.179+0000 7f2c4affd640 1 --2- 172.21.15.116:0/315461092 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56424d97fde0 0x56424d982250 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e117bb9035fcbbf server_cookie=34fe0d36d233b08e in_seq=0 out_seq=0 2024-03-19T01:25:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.179+0000 7f2c2effd640 10 client.26123 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c2effd640 1 -- 172.21.15.116:0/315461092 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56424d8dc120 con 0x56424d97fde0 2024-03-19T01:25:32.182 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c2effd640 10 client.26123 handle_command_reply: tid=0 2024-03-19T01:25:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c54dfd740 1 client.26123 shutdown 2024-03-19T01:25:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c54dfd740 1 -- 172.21.15.116:0/315461092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96a100 msgr2=0x56424d973f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/315461092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96a100 0x56424d973f80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c011570 tx=0x7f2c3c011a30 comp rx=0 tx=0).stop 2024-03-19T01:25:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c54dfd740 1 -- 172.21.15.116:0/315461092 shutdown_connections 2024-03-19T01:25:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/315461092 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 0x56424d97b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/315461092 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56424d97fde0 0x56424d982250 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/315461092 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96aaf0 0x56424d9744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c2effd640 0 client.26123 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/315461092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96a100 0x56424d973f80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c54dfd740 1 -- 172.21.15.116:0/315461092 >> 172.21.15.116:0/315461092 conn(0x56424d965670 msgr2=0x56424d9668c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c54dfd740 1 -- 172.21.15.116:0/315461092 shutdown_connections 2024-03-19T01:25:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c54dfd740 1 -- 172.21.15.116:0/315461092 wait complete. 2024-03-19T01:25:32.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.180+0000 7f2c54dfd740 20 client.26123 trim_cache size 0 max 16384 2024-03-19T01:25:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.202+0000 7f2c54dfd740 1 Processor -- start 2024-03-19T01:25:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.203+0000 7f2c54dfd740 1 -- start start 2024-03-19T01:25:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.203+0000 7f2c54dfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96a100 0x56424d9735a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.203+0000 7f2c54dfd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96aaf0 0x56424d97b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.203+0000 7f2c54dfd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 0x56424d973ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.203+0000 7f2c54dfd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1af410 con 0x56424d96fc60 2024-03-19T01:25:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.203+0000 7f2c54dfd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1b1010 con 0x56424d96aaf0 2024-03-19T01:25:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.203+0000 7f2c54dfd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1afae0 con 0x56424d96a100 2024-03-19T01:25:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.203+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 0x56424d973ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.203+0000 7f2c4bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 0x56424d973ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51570/0 (socket says 172.21.15.116:51570) 2024-03-19T01:25:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.203+0000 7f2c4bfff640 1 -- 172.21.15.116:0/2802577187 learned_addr learned my addr 172.21.15.116:0/2802577187 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.203+0000 7f2c4affd640 1 --2- 172.21.15.116:0/2802577187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96aaf0 0x56424d97b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.204+0000 7f2c4b7fe640 1 --2- 172.21.15.116:0/2802577187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96a100 0x56424d9735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.204+0000 7f2c4affd640 1 -- 172.21.15.116:0/2802577187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96a100 msgr2=0x56424d9735a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.204+0000 7f2c4affd640 1 --2- 172.21.15.116:0/2802577187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96a100 0x56424d9735a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.204+0000 7f2c4affd640 1 -- 172.21.15.116:0/2802577187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 msgr2=0x56424d973ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.204+0000 7f2c4affd640 1 --2- 172.21.15.116:0/2802577187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 0x56424d973ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.204+0000 7f2c4affd640 1 -- 172.21.15.116:0/2802577187 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56424d6af620 con 0x56424d96aaf0 2024-03-19T01:25:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.204+0000 7f2c4affd640 1 --2- 172.21.15.116:0/2802577187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96aaf0 0x56424d97b420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x56424d96aef0 tx=0x7f2c4001c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5c2db014068e129 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.204+0000 7f2c397fa640 1 -- 172.21.15.116:0/2802577187 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40032070 con 0x56424d96aaf0 2024-03-19T01:25:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.204+0000 7f2c397fa640 1 -- 172.21.15.116:0/2802577187 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c40018ce0 con 0x56424d96aaf0 2024-03-19T01:25:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.204+0000 7f2c397fa640 1 -- 172.21.15.116:0/2802577187 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40025cb0 con 0x56424d96aaf0 2024-03-19T01:25:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.205+0000 7f2c54dfd740 1 -- 172.21.15.116:0/2802577187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96aaf0 msgr2=0x56424d97b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.205+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/2802577187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96aaf0 0x56424d97b420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x56424d96aef0 tx=0x7f2c4001c000 comp rx=0 tx=0).stop 2024-03-19T01:25:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.206+0000 7f2c54dfd740 1 -- 172.21.15.116:0/2802577187 shutdown_connections 2024-03-19T01:25:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.206+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/2802577187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 0x56424d973ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.206+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/2802577187 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96aaf0 0x56424d97b420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.206+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/2802577187 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96a100 0x56424d9735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.206+0000 7f2c54dfd740 1 -- 172.21.15.116:0/2802577187 >> 172.21.15.116:0/2802577187 conn(0x56424d965670 msgr2=0x56424d96b590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.206+0000 7f2c54dfd740 1 -- 172.21.15.116:0/2802577187 shutdown_connections 2024-03-19T01:25:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.206+0000 7f2c54dfd740 1 -- 172.21.15.116:0/2802577187 wait complete. 2024-03-19T01:25:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.207+0000 7f2c54dfd740 1 -- start start 2024-03-19T01:25:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.207+0000 7f2c54dfd740 1 Processor -- start 2024-03-19T01:25:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.208+0000 7f2c54dfd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96a100 0x56424d978d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.208+0000 7f2c4b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96a100 0x56424d978d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.208+0000 7f2c4b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96a100 0x56424d978d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48404/0 (socket says 172.21.15.116:48404) 2024-03-19T01:25:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.208+0000 7f2c54dfd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96aaf0 0x56424d979250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.208+0000 7f2c4b7fe640 1 -- 172.21.15.116:0/3873054766 learned_addr learned my addr 172.21.15.116:0/3873054766 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.209+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/3873054766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 0x56424d97a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.209+0000 7f2c54dfd740 1 -- 172.21.15.116:0/3873054766 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1b0940 con 0x56424d96fc60 2024-03-19T01:25:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.209+0000 7f2c4affd640 1 --2- 172.21.15.116:0/3873054766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96aaf0 0x56424d979250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.209+0000 7f2c54dfd740 1 -- 172.21.15.116:0/3873054766 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1b0270 con 0x56424d96aaf0 2024-03-19T01:25:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.209+0000 7f2c54dfd740 1 -- 172.21.15.116:0/3873054766 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c1ad3f0 con 0x56424d96a100 2024-03-19T01:25:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.209+0000 7f2c54dfd740 10 client.0 _ll_register_callbacks cb 0x7f2c3c009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:25:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.209+0000 7f2c4affd640 1 -- 172.21.15.116:0/3873054766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96a100 msgr2=0x56424d978d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.209+0000 7f2c4affd640 1 --2- 172.21.15.116:0/3873054766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96a100 0x56424d978d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.209+0000 7f2c4affd640 1 -- 172.21.15.116:0/3873054766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 msgr2=0x56424d97a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.209+0000 7f2c4affd640 1 --2- 172.21.15.116:0/3873054766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 0x56424d97a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.209+0000 7f2c4affd640 1 -- 172.21.15.116:0/3873054766 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56424d97a850 con 0x56424d96aaf0 2024-03-19T01:25:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.210+0000 7f2c4affd640 1 --2- 172.21.15.116:0/3873054766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96aaf0 0x56424d979250 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x56424d966110 tx=0x7f2c40025640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ab8dc8fb24f89d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.210+0000 7f2c2effd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:25:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.210+0000 7f2c2effd640 1 -- 172.21.15.116:0/3873054766 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c40032040 con 0x56424d96aaf0 2024-03-19T01:25:32.212 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.210+0000 7f2c2effd640 1 -- 172.21.15.116:0/3873054766 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2c40025c30 con 0x56424d96aaf0 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.210+0000 7f2c54dfd740 1 -- 172.21.15.116:0/3873054766 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2c40018ce0 con 0x56424d96aaf0 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.210+0000 7f2c2effd640 1 -- 172.21.15.116:0/3873054766 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f2c40025e30 con 0x56424d96aaf0 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.210+0000 7f2c54dfd740 10 client.26129 fetch_fsmap learned FSMap version 47 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.210+0000 7f2c54dfd740 1 -- 172.21.15.116:0/3873054766 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x56424d9e6360 con 0x56424d96aaf0 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.211+0000 7f2c2effd640 1 -- 172.21.15.116:0/3873054766 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c40018ae0 con 0x56424d96aaf0 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.211+0000 7f2c2effd640 1 -- 172.21.15.116:0/3873054766 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 47) ==== 8370+0+0 (secure 0 0 0) 0x7f2c4003e030 con 0x56424d96aaf0 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.211+0000 7f2c54dfd740 10 client.26129 fetch_fsmap finished waiting for FSMap version 47 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.211+0000 7f2c54dfd740 10 client.26129 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.211+0000 7f2c54dfd740 20 client.26129 populate_metadata read hostname 'smithi116' 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.211+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/3873054766 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56424d97fde0 0x56424d9e8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.211+0000 7f2c54dfd740 4 client.26129 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.211+0000 7f2c54dfd740 1 -- 172.21.15.116:0/3873054766 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f2c40018ce0 con 0x56424d97fde0 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.211+0000 7f2c4b7fe640 1 --2- 172.21.15.116:0/3873054766 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56424d97fde0 0x56424d9e8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:32.213 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.212+0000 7f2c2effd640 10 client.26129 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:32.214 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.212+0000 7f2c4b7fe640 1 --2- 172.21.15.116:0/3873054766 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56424d97fde0 0x56424d9e8c80 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fac6393e6c9b5aef server_cookie=374d69271930164c in_seq=0 out_seq=0 2024-03-19T01:25:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c2effd640 1 -- 172.21.15.116:0/3873054766 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f2c40018ce0 con 0x56424d97fde0 2024-03-19T01:25:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c2effd640 10 client.26129 handle_command_reply: tid=0 2024-03-19T01:25:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c54dfd740 1 client.26129 shutdown 2024-03-19T01:25:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c54dfd740 1 -- 172.21.15.116:0/3873054766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96aaf0 msgr2=0x56424d979250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/3873054766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96aaf0 0x56424d979250 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x56424d966110 tx=0x7f2c40025640 comp rx=0 tx=0).stop 2024-03-19T01:25:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c54dfd740 1 -- 172.21.15.116:0/3873054766 shutdown_connections 2024-03-19T01:25:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/3873054766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56424d96fc60 0x56424d97a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/3873054766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56424d96aaf0 0x56424d979250 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.225 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/3873054766 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56424d97fde0 0x56424d9e8c80 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c54dfd740 1 --2- 172.21.15.116:0/3873054766 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56424d96a100 0x56424d978d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c2effd640 0 client.26129 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c54dfd740 1 -- 172.21.15.116:0/3873054766 >> 172.21.15.116:0/3873054766 conn(0x56424d965670 msgr2=0x56424d9688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c54dfd740 1 -- 172.21.15.116:0/3873054766 shutdown_connections 2024-03-19T01:25:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.224+0000 7f2c54dfd740 1 -- 172.21.15.116:0/3873054766 wait complete. 2024-03-19T01:25:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.225+0000 7f2c54dfd740 20 client.26129 trim_cache size 0 max 16384 2024-03-19T01:25:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:25:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:25:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:25:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:25:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.225+0000 7f2c52140640 1 -- 172.21.15.116:0/1066885828 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2c340a28a0 msgr2=0x7f2c340a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.225+0000 7f2c52140640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2c340a28a0 0x7f2c340a4d60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c111190 tx=0x7f2c400159a0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.225+0000 7f2c52140640 1 -- 172.21.15.116:0/1066885828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c4c101220 msgr2=0x7f2c4c071cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.225+0000 7f2c52140640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c4c101220 0x7f2c4c071cd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2c4401ac00 tx=0x7f2c44003ba0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.225+0000 7f2c52140640 1 -- 172.21.15.116:0/1066885828 shutdown_connections 2024-03-19T01:25:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.225+0000 7f2c52140640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2c340a28a0 0x7f2c340a4d60 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.225+0000 7f2c52140640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2c4c101220 0x7f2c4c071cd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.225+0000 7f2c52140640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c4c100860 0x7f2c4c06f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.225+0000 7f2c52140640 1 --2- 172.21.15.116:0/1066885828 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2c4c0ffb00 0x7f2c4c06d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.225+0000 7f2c52140640 1 -- 172.21.15.116:0/1066885828 >> 172.21.15.116:0/1066885828 conn(0x7f2c4c0fb470 msgr2=0x7f2c4c103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.225+0000 7f2c52140640 1 -- 172.21.15.116:0/1066885828 shutdown_connections 2024-03-19T01:25:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:32.225+0000 7f2c52140640 1 -- 172.21.15.116:0/1066885828 wait complete. 2024-03-19T01:25:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:33 smithi162 ceph-mon[67178]: pgmap v739: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 633 op/s 2024-03-19T01:25:33.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:33 smithi116 ceph-mon[62701]: pgmap v739: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 633 op/s 2024-03-19T01:25:33.726 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:33 smithi156 ceph-mon[66799]: pgmap v739: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 633 op/s 2024-03-19T01:25:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:35 smithi162 ceph-mon[67178]: pgmap v740: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.1 MiB/s wr, 555 op/s 2024-03-19T01:25:35.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:35 smithi116 ceph-mon[62701]: pgmap v740: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.1 MiB/s wr, 555 op/s 2024-03-19T01:25:35.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:35 smithi156 ceph-mon[66799]: pgmap v740: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.1 MiB/s wr, 555 op/s 2024-03-19T01:25:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 770.0 s, 2018.4 tps, lat 49.763 ms stddev 57.146 2024-03-19T01:25:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:37 smithi162 ceph-mon[67178]: pgmap v741: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.5 MiB/s wr, 747 op/s 2024-03-19T01:25:37.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:37 smithi116 ceph-mon[62701]: pgmap v741: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.5 MiB/s wr, 747 op/s 2024-03-19T01:25:37.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:37 smithi156 ceph-mon[66799]: pgmap v741: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 7.5 MiB/s wr, 747 op/s 2024-03-19T01:25:39.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:39 smithi162 ceph-mon[67178]: pgmap v742: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.5 MiB/s wr, 663 op/s 2024-03-19T01:25:39.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:39 smithi116 ceph-mon[62701]: pgmap v742: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.5 MiB/s wr, 663 op/s 2024-03-19T01:25:39.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:39 smithi156 ceph-mon[66799]: pgmap v742: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.5 MiB/s wr, 663 op/s 2024-03-19T01:25:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 775.0 s, 1477.4 tps, lat 67.030 ms stddev 74.048 2024-03-19T01:25:41.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:41 smithi162 ceph-mon[67178]: pgmap v743: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.8 MiB/s wr, 604 op/s 2024-03-19T01:25:41.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:41 smithi116 ceph-mon[62701]: pgmap v743: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.8 MiB/s wr, 604 op/s 2024-03-19T01:25:41.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:41 smithi156 ceph-mon[66799]: pgmap v743: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.8 MiB/s wr, 604 op/s 2024-03-19T01:25:42.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:42 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:25:42.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:42 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:25:42.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:42 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:25:43.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:43 smithi162 ceph-mon[67178]: pgmap v744: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.8 MiB/s wr, 702 op/s 2024-03-19T01:25:43.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:43 smithi116 ceph-mon[62701]: pgmap v744: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.8 MiB/s wr, 702 op/s 2024-03-19T01:25:43.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:43 smithi156 ceph-mon[66799]: pgmap v744: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.8 MiB/s wr, 702 op/s 2024-03-19T01:25:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.357+0000 7f02ca275640 1 -- 172.21.15.116:0/3686829768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c40ffb20 msgr2=0x7f02c40fff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:45.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.357+0000 7f02ca275640 1 --2- 172.21.15.116:0/3686829768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c40ffb20 0x7f02c40fff40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f02b8009930 tx=0x7f02b802bc30 comp rx=0 tx=0).stop 2024-03-19T01:25:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.357+0000 7f02ca275640 1 -- 172.21.15.116:0/3686829768 shutdown_connections 2024-03-19T01:25:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.357+0000 7f02ca275640 1 --2- 172.21.15.116:0/3686829768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c4103550 0x7f02c4109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.357+0000 7f02ca275640 1 --2- 172.21.15.116:0/3686829768 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02c4102b90 0x7f02c4103010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.357+0000 7f02ca275640 1 --2- 172.21.15.116:0/3686829768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c40ffb20 0x7f02c40fff40 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.357+0000 7f02ca275640 1 -- 172.21.15.116:0/3686829768 >> 172.21.15.116:0/3686829768 conn(0x7f02c40fb470 msgr2=0x7f02c40fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.357+0000 7f02ca275640 1 -- 172.21.15.116:0/3686829768 shutdown_connections 2024-03-19T01:25:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.357+0000 7f02ca275640 1 -- 172.21.15.116:0/3686829768 wait complete. 2024-03-19T01:25:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.358+0000 7f02ca275640 1 Processor -- start 2024-03-19T01:25:45.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.358+0000 7f02ca275640 1 -- start start 2024-03-19T01:25:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.358+0000 7f02ca275640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c40ffb20 0x7f02c4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.360 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.359+0000 7f02ca275640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c4102b90 0x7f02c4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.359+0000 7f02c3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c40ffb20 0x7f02c4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.359+0000 7f02c3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c40ffb20 0x7f02c4197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:50204/0 (socket says 172.21.15.116:50204) 2024-03-19T01:25:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.359+0000 7f02c3fff640 1 -- 172.21.15.116:0/3566145146 learned_addr learned my addr 172.21.15.116:0/3566145146 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.359+0000 7f02ca275640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02c4103550 0x7f02c419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.360+0000 7f02ca275640 1 -- 172.21.15.116:0/3566145146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02c410d4d0 con 0x7f02c4102b90 2024-03-19T01:25:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.360+0000 7f02c8812640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02c4103550 0x7f02c419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.360+0000 7f02ca275640 1 -- 172.21.15.116:0/3566145146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f02c410d350 con 0x7f02c40ffb20 2024-03-19T01:25:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.360+0000 7f02ca275640 1 -- 172.21.15.116:0/3566145146 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02c410d650 con 0x7f02c4103550 2024-03-19T01:25:45.361 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.360+0000 7f02c37fe640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c4102b90 0x7f02c4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.360+0000 7f02c3fff640 1 -- 172.21.15.116:0/3566145146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02c4103550 msgr2=0x7f02c419f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.360+0000 7f02c3fff640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02c4103550 0x7f02c419f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.360+0000 7f02c3fff640 1 -- 172.21.15.116:0/3566145146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c4102b90 msgr2=0x7f02c4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.360+0000 7f02c3fff640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c4102b90 0x7f02c4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.360+0000 7f02c3fff640 1 -- 172.21.15.116:0/3566145146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c419f770 con 0x7f02c40ffb20 2024-03-19T01:25:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.360+0000 7f02c8812640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02c4103550 0x7f02c419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:25:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.361+0000 7f02c3fff640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c40ffb20 0x7f02c4197b50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f02b802c7b0 tx=0x7f02b802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:45.362 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.361+0000 7f02c17fa640 1 -- 172.21.15.116:0/3566145146 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b8041070 con 0x7f02c40ffb20 2024-03-19T01:25:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.361+0000 7f02c17fa640 1 -- 172.21.15.116:0/3566145146 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02b8004450 con 0x7f02c40ffb20 2024-03-19T01:25:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.361+0000 7f02c17fa640 1 -- 172.21.15.116:0/3566145146 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b8034440 con 0x7f02c40ffb20 2024-03-19T01:25:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.361+0000 7f02ca275640 1 -- 172.21.15.116:0/3566145146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c419fa00 con 0x7f02c40ffb20 2024-03-19T01:25:45.363 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.361+0000 7f02ca275640 1 -- 172.21.15.116:0/3566145146 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c419fea0 con 0x7f02c40ffb20 2024-03-19T01:25:45.364 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.364+0000 7f02c17fa640 1 -- 172.21.15.116:0/3566145146 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f02b8044070 con 0x7f02c40ffb20 2024-03-19T01:25:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02cce59740 1 Processor -- start 2024-03-19T01:25:45.367 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02cce59740 1 -- start start 2024-03-19T01:25:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02c17fa640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02ac0a2a10 0x7f02ac0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02cce59740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c57520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02cce59740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 0x561aa2c5c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02cce59740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c5f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02cce59740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561aa2bbbbe0 con 0x561aa2c5cc60 2024-03-19T01:25:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02cce59740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561aa2b64960 con 0x561aa2c57af0 2024-03-19T01:25:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02cce59740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561aa2bb7c80 con 0x561aa2c57100 2024-03-19T01:25:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02c3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c57520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02c8812640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02ac0a2a10 0x7f02ac0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02c37fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 0x561aa2c5c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02c3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c57520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:56082/0 (socket says 172.21.15.116:56082) 2024-03-19T01:25:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02c3fff640 1 -- 172.21.15.116:0/1075594005 learned_addr learned my addr 172.21.15.116:0/1075594005 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:45.368 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02c17fa640 1 -- 172.21.15.116:0/3566145146 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f02b80f04c0 con 0x7f02c40ffb20 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.365+0000 7f02c8812640 1 --2- 172.21.15.116:0/1075594005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c5f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.366+0000 7f02c3fff640 1 -- 172.21.15.116:0/1075594005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 msgr2=0x561aa2c5c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.366+0000 7f02c3fff640 1 --2- 172.21.15.116:0/1075594005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 0x561aa2c5c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.366+0000 7f02c3fff640 1 -- 172.21.15.116:0/1075594005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 msgr2=0x561aa2c5f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.366+0000 7f02c3fff640 1 --2- 172.21.15.116:0/1075594005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c5f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.366+0000 7f02c3fff640 1 -- 172.21.15.116:0/1075594005 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aa2b5bd20 con 0x561aa2c57100 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.366+0000 7f02c3fff640 1 --2- 172.21.15.116:0/1075594005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c57520 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f02b802cfd0 tx=0x7f02b8009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a981c659e8061f14 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.366+0000 7f02c8812640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02ac0a2a10 0x7f02ac0a4ed0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f02bc005fd0 tx=0x7f02bc011720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.366+0000 7f02aa7fc640 1 -- 172.21.15.116:0/1075594005 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b80bf020 con 0x561aa2c57100 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.366+0000 7f02aa7fc640 1 -- 172.21.15.116:0/1075594005 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02b80b9140 con 0x561aa2c57100 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.366+0000 7f02aa7fc640 1 -- 172.21.15.116:0/1075594005 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b80bce40 con 0x561aa2c57100 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.367+0000 7f02cce59740 1 -- 172.21.15.116:0/1075594005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 msgr2=0x561aa2c57520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.367+0000 7f02cce59740 1 --2- 172.21.15.116:0/1075594005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c57520 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f02b802cfd0 tx=0x7f02b8009260 comp rx=0 tx=0).stop 2024-03-19T01:25:45.369 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.367+0000 7f02cce59740 1 -- 172.21.15.116:0/1075594005 shutdown_connections 2024-03-19T01:25:45.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.367+0000 7f02cce59740 1 --2- 172.21.15.116:0/1075594005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c5f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.367+0000 7f02cce59740 1 --2- 172.21.15.116:0/1075594005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 0x561aa2c5c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.367+0000 7f02cce59740 1 --2- 172.21.15.116:0/1075594005 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c57520 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.367+0000 7f02cce59740 1 -- 172.21.15.116:0/1075594005 >> 172.21.15.116:0/1075594005 conn(0x561aa2c52670 msgr2=0x561aa2c52a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:45.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.367+0000 7f02cce59740 1 -- 172.21.15.116:0/1075594005 shutdown_connections 2024-03-19T01:25:45.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.367+0000 7f02cce59740 1 -- 172.21.15.116:0/1075594005 wait complete. 2024-03-19T01:25:45.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.367+0000 7f02cce59740 1 -- start start 2024-03-19T01:25:45.370 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.367+0000 7f02cce59740 1 Processor -- start 2024-03-19T01:25:45.373 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02cce59740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c60d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.373 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02cce59740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c57af0 0x561aa2c61290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.373 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02cce59740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c5cc60 0x561aa2c682d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02cce59740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x561aa2bb7c80 con 0x561aa2c57af0 2024-03-19T01:25:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02cce59740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561aa2b77db0 con 0x561aa2c5cc60 2024-03-19T01:25:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02cce59740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x561aa2c5f9d0 con 0x561aa2c57100 2024-03-19T01:25:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02c37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c57af0 0x561aa2c61290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02cce59740 10 client.0 _ll_register_callbacks cb 0x561aa2b7ce70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:25:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02c3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c60d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02c37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c57af0 0x561aa2c61290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40598/0 (socket says 172.21.15.116:40598) 2024-03-19T01:25:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02c37fe640 1 -- 172.21.15.116:0/2454612375 learned_addr learned my addr 172.21.15.116:0/2454612375 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02c8812640 1 --2- 172.21.15.116:0/2454612375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c5cc60 0x561aa2c682d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02c3fff640 1 -- 172.21.15.116:0/2454612375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c5cc60 msgr2=0x561aa2c682d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02c3fff640 1 --2- 172.21.15.116:0/2454612375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c5cc60 0x561aa2c682d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02c3fff640 1 -- 172.21.15.116:0/2454612375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c57af0 msgr2=0x561aa2c61290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:45.374 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02c3fff640 1 --2- 172.21.15.116:0/2454612375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c57af0 0x561aa2c61290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02c3fff640 1 -- 172.21.15.116:0/2454612375 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aa2b76c60 con 0x561aa2c57100 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02c3fff640 1 --2- 172.21.15.116:0/2454612375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c60d50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f02b8036040 tx=0x7f02b8037190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83a07c9fac2bf72a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02a37fe640 1 -- 172.21.15.116:0/2454612375 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b80da060 con 0x561aa2c57100 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02a37fe640 1 -- 172.21.15.116:0/2454612375 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f02b80c5150 con 0x561aa2c57100 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.368+0000 7f02cce59740 1 -- 172.21.15.116:0/2454612375 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02b80b9140 con 0x561aa2c57100 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.369+0000 7f02a37fe640 1 -- 172.21.15.116:0/2454612375 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f02b80bf050 con 0x561aa2c57100 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.369+0000 7f02cce59740 10 client.26131 fetch_fsmap learned FSMap version 48 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.369+0000 7f02cce59740 1 -- 172.21.15.116:0/2454612375 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x561aa28d0210 con 0x561aa2c57100 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.369+0000 7f02a37fe640 1 -- 172.21.15.116:0/2454612375 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02b80b9140 con 0x561aa2c57100 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.369+0000 7f02a37fe640 1 -- 172.21.15.116:0/2454612375 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 48) ==== 8358+0+0 (secure 0 0 0) 0x7f02b80bc290 con 0x561aa2c57100 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.370+0000 7f02cce59740 10 client.26131 fetch_fsmap finished waiting for FSMap version 48 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.370+0000 7f02cce59740 10 client.26131 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:25:45.375 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.370+0000 7f02cce59740 20 client.26131 populate_metadata read hostname 'smithi116' 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.370+0000 7f02cce59740 1 --2- 172.21.15.116:0/2454612375 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561aa2c6c600 0x561aa2c6ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.370+0000 7f02cce59740 4 client.26131 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.370+0000 7f02cce59740 1 -- 172.21.15.116:0/2454612375 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561aa2bc9120 con 0x561aa2c6c600 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.370+0000 7f02c37fe640 1 --2- 172.21.15.116:0/2454612375 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561aa2c6c600 0x561aa2c6ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.371+0000 7f02c37fe640 1 --2- 172.21.15.116:0/2454612375 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561aa2c6c600 0x561aa2c6ea30 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74f868ed6d819d61 server_cookie=38a66d1f09f3255e in_seq=0 out_seq=0 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.371+0000 7f02a37fe640 10 client.26131 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.372+0000 7f02a37fe640 1 -- 172.21.15.116:0/2454612375 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561aa2bc9120 con 0x561aa2c6c600 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.372+0000 7f02a37fe640 10 client.26131 handle_command_reply: tid=0 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.372+0000 7f02cce59740 1 client.26131 shutdown 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.372+0000 7f02cce59740 1 -- 172.21.15.116:0/2454612375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 msgr2=0x561aa2c60d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.372+0000 7f02cce59740 1 --2- 172.21.15.116:0/2454612375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c60d50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f02b8036040 tx=0x7f02b8037190 comp rx=0 tx=0).stop 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.373+0000 7f02cce59740 1 -- 172.21.15.116:0/2454612375 shutdown_connections 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.373+0000 7f02cce59740 1 --2- 172.21.15.116:0/2454612375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c5cc60 0x561aa2c682d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.373+0000 7f02cce59740 1 --2- 172.21.15.116:0/2454612375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c57af0 0x561aa2c61290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.376 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.373+0000 7f02cce59740 1 --2- 172.21.15.116:0/2454612375 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561aa2c6c600 0x561aa2c6ea30 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.373+0000 7f02cce59740 1 --2- 172.21.15.116:0/2454612375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c60d50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.373+0000 7f02a37fe640 0 client.26131 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.373+0000 7f02cce59740 1 -- 172.21.15.116:0/2454612375 >> 172.21.15.116:0/2454612375 conn(0x561aa2c52670 msgr2=0x561aa2c538c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.373+0000 7f02cce59740 1 -- 172.21.15.116:0/2454612375 shutdown_connections 2024-03-19T01:25:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.373+0000 7f02cce59740 1 -- 172.21.15.116:0/2454612375 wait complete. 2024-03-19T01:25:45.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.373+0000 7f02cce59740 20 client.26131 trim_cache size 0 max 16384 2024-03-19T01:25:45.402 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.402+0000 7f02cce59740 1 Processor -- start 2024-03-19T01:25:45.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.403+0000 7f02cce59740 1 -- start start 2024-03-19T01:25:45.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.403+0000 7f02cce59740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c61070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.403+0000 7f02cce59740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 0x561aa2c68290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.403+0000 7f02cce59740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c58510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.403+0000 7f02cce59740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02b000e440 con 0x561aa2c5cc60 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.403+0000 7f02cce59740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f02b000ffb0 con 0x561aa2c57af0 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.403+0000 7f02cce59740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02b000eb10 con 0x561aa2c57100 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.403+0000 7f02c8812640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c58510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.403+0000 7f02c3fff640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c61070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.403+0000 7f02c8812640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c58510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40610/0 (socket says 172.21.15.116:40610) 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.403+0000 7f02c8812640 1 -- 172.21.15.116:0/2071419948 learned_addr learned my addr 172.21.15.116:0/2071419948 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.403+0000 7f02c37fe640 1 --2- 172.21.15.116:0/2071419948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 0x561aa2c68290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.404+0000 7f02c3fff640 1 -- 172.21.15.116:0/2071419948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 msgr2=0x561aa2c68290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.404+0000 7f02c3fff640 1 --2- 172.21.15.116:0/2071419948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 0x561aa2c68290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.404+0000 7f02c3fff640 1 -- 172.21.15.116:0/2071419948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 msgr2=0x561aa2c58510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.404+0000 7f02c3fff640 1 --2- 172.21.15.116:0/2071419948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c58510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.404+0000 7f02c3fff640 1 -- 172.21.15.116:0/2071419948 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561aa299c620 con 0x561aa2c57100 2024-03-19T01:25:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.404+0000 7f02c3fff640 1 --2- 172.21.15.116:0/2071419948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c61070 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f02b80bcb40 tx=0x7f02b80c1110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96c7bfd8ccebe5ef server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.404+0000 7f02a9ffb640 1 -- 172.21.15.116:0/2071419948 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b80da070 con 0x561aa2c57100 2024-03-19T01:25:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.404+0000 7f02a9ffb640 1 -- 172.21.15.116:0/2071419948 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02b80b9140 con 0x561aa2c57100 2024-03-19T01:25:45.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.404+0000 7f02a9ffb640 1 -- 172.21.15.116:0/2071419948 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b80f00d0 con 0x561aa2c57100 2024-03-19T01:25:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.405+0000 7f02cce59740 1 -- 172.21.15.116:0/2071419948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 msgr2=0x561aa2c61070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.405+0000 7f02cce59740 1 --2- 172.21.15.116:0/2071419948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c61070 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f02b80bcb40 tx=0x7f02b80c1110 comp rx=0 tx=0).stop 2024-03-19T01:25:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.405+0000 7f02cce59740 1 -- 172.21.15.116:0/2071419948 shutdown_connections 2024-03-19T01:25:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.405+0000 7f02cce59740 1 --2- 172.21.15.116:0/2071419948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c58510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.405+0000 7f02cce59740 1 --2- 172.21.15.116:0/2071419948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 0x561aa2c68290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.405+0000 7f02cce59740 1 --2- 172.21.15.116:0/2071419948 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c61070 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.405+0000 7f02cce59740 1 -- 172.21.15.116:0/2071419948 >> 172.21.15.116:0/2071419948 conn(0x561aa2c52670 msgr2=0x561aa2c60960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.405+0000 7f02cce59740 1 -- 172.21.15.116:0/2071419948 shutdown_connections 2024-03-19T01:25:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.405+0000 7f02cce59740 1 -- 172.21.15.116:0/2071419948 wait complete. 2024-03-19T01:25:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.405+0000 7f02cce59740 1 -- start start 2024-03-19T01:25:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02cce59740 1 Processor -- start 2024-03-19T01:25:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02cce59740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c65980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02cce59740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 0x561aa2c65ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02cce59740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c66e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02cce59740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02b000f8e0 con 0x561aa2c5cc60 2024-03-19T01:25:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02cce59740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f02b000f210 con 0x561aa2c57af0 2024-03-19T01:25:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02cce59740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f02b000c420 con 0x561aa2c57100 2024-03-19T01:25:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02cce59740 10 client.0 _ll_register_callbacks cb 0x7f02b400b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:25:45.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02c8812640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c66e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02c8812640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c66e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40612/0 (socket says 172.21.15.116:40612) 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02c8812640 1 -- 172.21.15.116:0/3474880893 learned_addr learned my addr 172.21.15.116:0/3474880893 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02c37fe640 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 0x561aa2c65ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02c3fff640 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c65980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02c8812640 1 -- 172.21.15.116:0/3474880893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 msgr2=0x561aa2c65980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02c8812640 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c65980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02c8812640 1 -- 172.21.15.116:0/3474880893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 msgr2=0x561aa2c65ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02c8812640 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 0x561aa2c65ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02c8812640 1 -- 172.21.15.116:0/3474880893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561aa2c67450 con 0x561aa2c5cc60 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.406+0000 7f02c3fff640 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c65980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02c8812640 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c66e60 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0112a0 tx=0x7f02bc009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c6e4819a6c89185 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02a37fe640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02a37fe640 1 -- 172.21.15.116:0/3474880893 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02bc011450 con 0x561aa2c5cc60 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02a37fe640 1 -- 172.21.15.116:0/3474880893 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f02bc01a040 con 0x561aa2c5cc60 2024-03-19T01:25:45.411 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02cce59740 1 -- 172.21.15.116:0/3474880893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02b80b9140 con 0x561aa2c5cc60 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02a37fe640 1 -- 172.21.15.116:0/3474880893 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f02bc004b20 con 0x561aa2c5cc60 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02cce59740 10 client.16299 fetch_fsmap learned FSMap version 48 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02cce59740 1 -- 172.21.15.116:0/3474880893 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x561aa2cd2970 con 0x561aa2c5cc60 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02a37fe640 1 -- 172.21.15.116:0/3474880893 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02b80b9140 con 0x561aa2c5cc60 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02a37fe640 1 -- 172.21.15.116:0/3474880893 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 48) ==== 8358+0+0 (secure 0 0 0) 0x7f02bc01e050 con 0x561aa2c5cc60 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02cce59740 10 client.16299 fetch_fsmap finished waiting for FSMap version 48 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02cce59740 10 client.16299 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02cce59740 20 client.16299 populate_metadata read hostname 'smithi116' 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02cce59740 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561aa2cd3470 0x561aa2cd5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02cce59740 4 client.16299 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.407+0000 7f02cce59740 1 -- 172.21.15.116:0/3474880893 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x561aa2bc3470 con 0x561aa2cd3470 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.408+0000 7f02c3fff640 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561aa2cd3470 0x561aa2cd5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.408+0000 7f02a37fe640 10 client.16299 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:45.412 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.408+0000 7f02c3fff640 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561aa2cd3470 0x561aa2cd5860 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3769b9b71e991009 server_cookie=3121c67a7c57138 in_seq=0 out_seq=0 2024-03-19T01:25:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02a37fe640 1 -- 172.21.15.116:0/3474880893 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x561aa2bc3470 con 0x561aa2cd3470 2024-03-19T01:25:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02a37fe640 10 client.16299 handle_command_reply: tid=0 2024-03-19T01:25:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02cce59740 1 client.16299 shutdown 2024-03-19T01:25:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02cce59740 1 -- 172.21.15.116:0/3474880893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 msgr2=0x561aa2c66e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02cce59740 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c66e60 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0112a0 tx=0x7f02bc009500 comp rx=0 tx=0).stop 2024-03-19T01:25:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02cce59740 1 -- 172.21.15.116:0/3474880893 shutdown_connections 2024-03-19T01:25:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02cce59740 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x561aa2c5cc60 0x561aa2c66e60 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.419 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02cce59740 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561aa2c57af0 0x561aa2c65ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02cce59740 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x561aa2cd3470 0x561aa2cd5860 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02a37fe640 0 client.16299 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:45.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02cce59740 1 --2- 172.21.15.116:0/3474880893 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x561aa2c57100 0x561aa2c65980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02cce59740 1 -- 172.21.15.116:0/3474880893 >> 172.21.15.116:0/3474880893 conn(0x561aa2c52670 msgr2=0x561aa2c558d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:45.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02cce59740 1 -- 172.21.15.116:0/3474880893 shutdown_connections 2024-03-19T01:25:45.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02cce59740 1 -- 172.21.15.116:0/3474880893 wait complete. 2024-03-19T01:25:45.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.418+0000 7f02cce59740 20 client.16299 trim_cache size 0 max 16384 2024-03-19T01:25:45.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:25:45.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:25:45.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:25:45.420 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:25:45.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.419+0000 7f02ca275640 1 -- 172.21.15.116:0/3566145146 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02ac0a2a10 msgr2=0x7f02ac0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:45.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.419+0000 7f02ca275640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02ac0a2a10 0x7f02ac0a4ed0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f02bc005fd0 tx=0x7f02bc011720 comp rx=0 tx=0).stop 2024-03-19T01:25:45.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.419+0000 7f02ca275640 1 -- 172.21.15.116:0/3566145146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c40ffb20 msgr2=0x7f02c4197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:45.421 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.419+0000 7f02ca275640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c40ffb20 0x7f02c4197b50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f02b802c7b0 tx=0x7f02b802c6c0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.419+0000 7f02ca275640 1 -- 172.21.15.116:0/3566145146 shutdown_connections 2024-03-19T01:25:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.419+0000 7f02ca275640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f02c4103550 0x7f02c419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.419+0000 7f02ca275640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02c4102b90 0x7f02c4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.419+0000 7f02ca275640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f02ac0a2a10 0x7f02ac0a4ed0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.419+0000 7f02ca275640 1 --2- 172.21.15.116:0/3566145146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02c40ffb20 0x7f02c4197b50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.419+0000 7f02ca275640 1 -- 172.21.15.116:0/3566145146 >> 172.21.15.116:0/3566145146 conn(0x7f02c40fb470 msgr2=0x7f02c4105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.419+0000 7f02ca275640 1 -- 172.21.15.116:0/3566145146 shutdown_connections 2024-03-19T01:25:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:45.419+0000 7f02ca275640 1 -- 172.21.15.116:0/3566145146 wait complete. 2024-03-19T01:25:45.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:45 smithi162 ceph-mon[67178]: pgmap v745: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.2 MiB/s wr, 621 op/s 2024-03-19T01:25:45.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:45 smithi116 ceph-mon[62701]: pgmap v745: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.2 MiB/s wr, 621 op/s 2024-03-19T01:25:45.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:45 smithi156 ceph-mon[66799]: pgmap v745: 97 pgs: 97 active+clean; 1.6 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.2 MiB/s wr, 621 op/s 2024-03-19T01:25:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 780.0 s, 1435.8 tps, lat 70.228 ms stddev 76.521 2024-03-19T01:25:47.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:47 smithi162 ceph-mon[67178]: pgmap v746: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.3 MiB/s wr, 742 op/s 2024-03-19T01:25:47.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:47 smithi116 ceph-mon[62701]: pgmap v746: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.3 MiB/s wr, 742 op/s 2024-03-19T01:25:47.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:47 smithi156 ceph-mon[66799]: pgmap v746: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 7.3 MiB/s wr, 742 op/s 2024-03-19T01:25:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:49 smithi162 ceph-mon[67178]: pgmap v747: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 580 op/s 2024-03-19T01:25:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:49 smithi116 ceph-mon[62701]: pgmap v747: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 580 op/s 2024-03-19T01:25:49.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:49 smithi156 ceph-mon[66799]: pgmap v747: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 580 op/s 2024-03-19T01:25:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 785.0 s, 1653.6 tps, lat 59.604 ms stddev 71.609 2024-03-19T01:25:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:51 smithi162 ceph-mon[67178]: pgmap v748: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.0 MiB/s wr, 561 op/s 2024-03-19T01:25:51.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:51 smithi116 ceph-mon[62701]: pgmap v748: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.0 MiB/s wr, 561 op/s 2024-03-19T01:25:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:51 smithi156 ceph-mon[66799]: pgmap v748: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.0 MiB/s wr, 561 op/s 2024-03-19T01:25:52.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:52 smithi162 ceph-mon[67178]: pgmap v749: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.2 MiB/s wr, 665 op/s 2024-03-19T01:25:52.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:52 smithi116 ceph-mon[62701]: pgmap v749: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.2 MiB/s wr, 665 op/s 2024-03-19T01:25:52.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:52 smithi156 ceph-mon[66799]: pgmap v749: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 6.2 MiB/s wr, 665 op/s 2024-03-19T01:25:54.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:54 smithi156 ceph-mon[66799]: pgmap v750: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 4.8 MiB/s wr, 529 op/s 2024-03-19T01:25:54.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:54 smithi162 ceph-mon[67178]: pgmap v750: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 4.8 MiB/s wr, 529 op/s 2024-03-19T01:25:54.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:54 smithi116 ceph-mon[62701]: pgmap v750: 97 pgs: 97 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 4.8 MiB/s wr, 529 op/s 2024-03-19T01:25:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 790.0 s, 1443.0 tps, lat 69.794 ms stddev 75.053 2024-03-19T01:25:56.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:56 smithi156 ceph-mon[66799]: pgmap v751: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.3 MiB/s wr, 656 op/s 2024-03-19T01:25:56.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:56 smithi162 ceph-mon[67178]: pgmap v751: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.3 MiB/s wr, 656 op/s 2024-03-19T01:25:56.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:56 smithi116 ceph-mon[62701]: pgmap v751: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 6.3 MiB/s wr, 656 op/s 2024-03-19T01:25:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.532+0000 7f2ab57b2640 1 -- 172.21.15.116:0/884862079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab0104c30 msgr2=0x7f2ab0107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.532+0000 7f2ab57b2640 1 --2- 172.21.15.116:0/884862079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab0104c30 0x7f2ab0107020 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009900 tx=0x7f2aa402bc30 comp rx=0 tx=0).stop 2024-03-19T01:25:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.533+0000 7f2ab57b2640 1 -- 172.21.15.116:0/884862079 shutdown_connections 2024-03-19T01:25:58.534 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.533+0000 7f2ab57b2640 1 --2- 172.21.15.116:0/884862079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab0107560 0x7f2ab01099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.533+0000 7f2ab57b2640 1 --2- 172.21.15.116:0/884862079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab0104c30 0x7f2ab0107020 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.533+0000 7f2ab57b2640 1 --2- 172.21.15.116:0/884862079 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ab0102300 0x7f2ab01046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.533+0000 7f2ab57b2640 1 -- 172.21.15.116:0/884862079 >> 172.21.15.116:0/884862079 conn(0x7f2ab00fbcf0 msgr2=0x7f2ab00fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.533+0000 7f2ab57b2640 1 -- 172.21.15.116:0/884862079 shutdown_connections 2024-03-19T01:25:58.535 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.533+0000 7f2ab57b2640 1 -- 172.21.15.116:0/884862079 wait complete. 2024-03-19T01:25:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.533+0000 7f2ab57b2640 1 Processor -- start 2024-03-19T01:25:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2ab57b2640 1 -- start start 2024-03-19T01:25:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2ab57b2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab0102300 0x7f2ab0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2ab57b2640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ab0104c30 0x7f2ab0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2ab57b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab0107560 0x7f2ab019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2ab57b2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab010d510 con 0x7f2ab0107560 2024-03-19T01:25:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2ab57b2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab010d390 con 0x7f2ab0102300 2024-03-19T01:25:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2ab57b2640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab010d690 con 0x7f2ab0104c30 2024-03-19T01:25:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2aaf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab0107560 0x7f2ab019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2aaeffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab0102300 0x7f2ab0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.537 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2aae7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ab0104c30 0x7f2ab0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2aaeffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab0102300 0x7f2ab0197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:47012/0 (socket says 172.21.15.116:47012) 2024-03-19T01:25:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2aaeffd640 1 -- 172.21.15.116:0/1551529959 learned_addr learned my addr 172.21.15.116:0/1551529959 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2aaeffd640 1 -- 172.21.15.116:0/1551529959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ab0104c30 msgr2=0x7f2ab0198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2aaeffd640 1 --2- 172.21.15.116:0/1551529959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ab0104c30 0x7f2ab0198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2aaeffd640 1 -- 172.21.15.116:0/1551529959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab0107560 msgr2=0x7f2ab019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2aaeffd640 1 --2- 172.21.15.116:0/1551529959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab0107560 0x7f2ab019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2aaeffd640 1 -- 172.21.15.116:0/1551529959 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab019f6e0 con 0x7f2ab0102300 2024-03-19T01:25:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.534+0000 7f2aaf7fe640 1 --2- 172.21.15.116:0/1551529959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab0107560 0x7f2ab019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:25:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.535+0000 7f2aaeffd640 1 --2- 172.21.15.116:0/1551529959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab0102300 0x7f2ab0197ac0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000e9f0 tx=0x7f2aa000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.535+0000 7f2a97fff640 1 -- 172.21.15.116:0/1551529959 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa000cd80 con 0x7f2ab0102300 2024-03-19T01:25:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.535+0000 7f2a97fff640 1 -- 172.21.15.116:0/1551529959 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa0005500 con 0x7f2ab0102300 2024-03-19T01:25:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.535+0000 7f2ab57b2640 1 -- 172.21.15.116:0/1551529959 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab019f9d0 con 0x7f2ab0102300 2024-03-19T01:25:58.538 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.535+0000 7f2a97fff640 1 -- 172.21.15.116:0/1551529959 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0010690 con 0x7f2ab0102300 2024-03-19T01:25:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.535+0000 7f2ab57b2640 1 -- 172.21.15.116:0/1551529959 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab019fea0 con 0x7f2ab0102300 2024-03-19T01:25:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.538+0000 7f2a97fff640 1 -- 172.21.15.116:0/1551529959 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f2aa00046c0 con 0x7f2ab0102300 2024-03-19T01:25:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.538+0000 7f2ab839e740 1 Processor -- start 2024-03-19T01:25:58.539 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.538+0000 7f2ab839e740 1 -- start start 2024-03-19T01:25:58.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.538+0000 7f2ab839e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43510 0x56434cb43930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.539+0000 7f2ab839e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 0x56434cb48a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.539+0000 7f2ab839e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb49070 0x56434cb4b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.539+0000 7f2ab839e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56434caa3be0 con 0x56434cb43f00 2024-03-19T01:25:58.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.539+0000 7f2ab839e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56434ca4c960 con 0x56434cb43510 2024-03-19T01:25:58.540 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.539+0000 7f2ab839e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56434ca9fc80 con 0x56434cb49070 2024-03-19T01:25:58.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.539+0000 7f2aaeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 0x56434cb48a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.539+0000 7f2aaeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 0x56434cb48a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57948/0 (socket says 172.21.15.116:57948) 2024-03-19T01:25:58.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.539+0000 7f2aaeffd640 1 -- 172.21.15.116:0/4115717052 learned_addr learned my addr 172.21.15.116:0/4115717052 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:58.541 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.539+0000 7f2a97fff640 1 --2- 172.21.15.116:0/1551529959 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2a980a28a0 0x7f2a980a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.539+0000 7f2a97fff640 1 -- 172.21.15.116:0/1551529959 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2aa00cee60 con 0x7f2ab0102300 2024-03-19T01:25:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.539+0000 7f2aaf7fe640 1 --2- 172.21.15.116:0/4115717052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43510 0x56434cb43930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.540+0000 7f2aae7fc640 1 --2- 172.21.15.116:0/4115717052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb49070 0x56434cb4b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.540+0000 7f2aaeffd640 1 -- 172.21.15.116:0/4115717052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb49070 msgr2=0x56434cb4b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.540+0000 7f2aaeffd640 1 --2- 172.21.15.116:0/4115717052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb49070 0x56434cb4b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.540+0000 7f2aaeffd640 1 -- 172.21.15.116:0/4115717052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43510 msgr2=0x56434cb43930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.540+0000 7f2aaeffd640 1 --2- 172.21.15.116:0/4115717052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43510 0x56434cb43930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.540+0000 7f2aaeffd640 1 -- 172.21.15.116:0/4115717052 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56434ca43d20 con 0x56434cb43f00 2024-03-19T01:25:58.542 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.540+0000 7f2aaf7fe640 1 --2- 172.21.15.116:0/1551529959 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2a980a28a0 0x7f2a980a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.541+0000 7f2aaeffd640 1 --2- 172.21.15.116:0/4115717052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 0x56434cb48a00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2aa009b770 tx=0x7f2aa0097480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f556f8a3ec639933 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:58.545 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.541+0000 7f2aaf7fe640 1 --2- 172.21.15.116:0/1551529959 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2a980a28a0 0x7f2a980a4d60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8005c40 tx=0x7f2aa8009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.541+0000 7f2a957fa640 1 -- 172.21.15.116:0/4115717052 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa009b320 con 0x56434cb43f00 2024-03-19T01:25:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.541+0000 7f2a957fa640 1 -- 172.21.15.116:0/4115717052 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa009bee0 con 0x56434cb43f00 2024-03-19T01:25:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.541+0000 7f2a957fa640 1 -- 172.21.15.116:0/4115717052 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa009b140 con 0x56434cb43f00 2024-03-19T01:25:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.542+0000 7f2ab839e740 1 -- 172.21.15.116:0/4115717052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 msgr2=0x56434cb48a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.542+0000 7f2ab839e740 1 --2- 172.21.15.116:0/4115717052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 0x56434cb48a00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2aa009b770 tx=0x7f2aa0097480 comp rx=0 tx=0).stop 2024-03-19T01:25:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.542+0000 7f2ab839e740 1 -- 172.21.15.116:0/4115717052 shutdown_connections 2024-03-19T01:25:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.542+0000 7f2ab839e740 1 --2- 172.21.15.116:0/4115717052 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb49070 0x56434cb4b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.542+0000 7f2ab839e740 1 --2- 172.21.15.116:0/4115717052 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 0x56434cb48a00 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.542+0000 7f2ab839e740 1 --2- 172.21.15.116:0/4115717052 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43510 0x56434cb43930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.542+0000 7f2ab839e740 1 -- 172.21.15.116:0/4115717052 >> 172.21.15.116:0/4115717052 conn(0x56434cb3a670 msgr2=0x56434cb3aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.542+0000 7f2ab839e740 1 -- 172.21.15.116:0/4115717052 shutdown_connections 2024-03-19T01:25:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.542+0000 7f2ab839e740 1 -- 172.21.15.116:0/4115717052 wait complete. 2024-03-19T01:25:58.546 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.542+0000 7f2ab839e740 1 -- start start 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.542+0000 7f2ab839e740 1 Processor -- start 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2ab839e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43510 0x56434cb462e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2ab839e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43f00 0x56434cb46820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2ab839e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb49070 0x56434cb47830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2ab839e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x56434cb3efc0 con 0x56434cb43510 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2ab839e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56434ca5fdb0 con 0x56434cb43f00 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2ab839e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x56434cb3f140 con 0x56434cb49070 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2ab839e740 10 client.0 _ll_register_callbacks cb 0x56434ca64e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2aaeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43510 0x56434cb462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2aaeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43510 0x56434cb462e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57962/0 (socket says 172.21.15.116:57962) 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2aaf7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb49070 0x56434cb47830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2aaeffd640 1 -- 172.21.15.116:0/796695755 learned_addr learned my addr 172.21.15.116:0/796695755 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2aae7fc640 1 --2- 172.21.15.116:0/796695755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43f00 0x56434cb46820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.547 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2aaeffd640 1 -- 172.21.15.116:0/796695755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb49070 msgr2=0x56434cb47830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2aaeffd640 1 --2- 172.21.15.116:0/796695755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb49070 0x56434cb47830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2aaeffd640 1 -- 172.21.15.116:0/796695755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43f00 msgr2=0x56434cb46820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2aaeffd640 1 --2- 172.21.15.116:0/796695755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43f00 0x56434cb46820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2aaeffd640 1 -- 172.21.15.116:0/796695755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56434ca5ec60 con 0x56434cb43510 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2aaeffd640 1 --2- 172.21.15.116:0/796695755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43510 0x56434cb462e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00cc990 tx=0x7f2aa00a4070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f63c53800c508d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2a92ff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2a92ff5640 1 -- 172.21.15.116:0/796695755 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa00b3020 con 0x56434cb43510 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2ab839e740 1 -- 172.21.15.116:0/796695755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2aa009bee0 con 0x56434cb43510 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.543+0000 7f2a92ff5640 1 -- 172.21.15.116:0/796695755 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2aa0097e50 con 0x56434cb43510 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.544+0000 7f2a92ff5640 1 -- 172.21.15.116:0/796695755 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f2aa009b320 con 0x56434cb43510 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.544+0000 7f2ab839e740 10 client.16311 fetch_fsmap learned FSMap version 48 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.544+0000 7f2ab839e740 1 -- 172.21.15.116:0/796695755 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x56434c7b8210 con 0x56434cb43510 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.544+0000 7f2a92ff5640 1 -- 172.21.15.116:0/796695755 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa009bee0 con 0x56434cb43510 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.544+0000 7f2a92ff5640 1 -- 172.21.15.116:0/796695755 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 48) ==== 8358+0+0 (secure 0 0 0) 0x7f2aa00a8080 con 0x56434cb43510 2024-03-19T01:25:58.548 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.544+0000 7f2ab839e740 10 client.16311 fetch_fsmap finished waiting for FSMap version 48 2024-03-19T01:25:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.544+0000 7f2ab839e740 10 client.16311 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:25:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.544+0000 7f2ab839e740 20 client.16311 populate_metadata read hostname 'smithi116' 2024-03-19T01:25:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.544+0000 7f2ab839e740 1 --2- 172.21.15.116:0/796695755 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56434cb5d3b0 0x56434cb5f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.544+0000 7f2ab839e740 4 client.16311 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:25:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.544+0000 7f2ab839e740 1 -- 172.21.15.116:0/796695755 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56434cab1120 con 0x56434cb5d3b0 2024-03-19T01:25:58.549 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.544+0000 7f2aae7fc640 1 --2- 172.21.15.116:0/796695755 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56434cb5d3b0 0x56434cb5f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.550 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.545+0000 7f2aae7fc640 1 --2- 172.21.15.116:0/796695755 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56434cb5d3b0 0x56434cb5f7a0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb47766294406372 server_cookie=130b953c6eb76e2d in_seq=0 out_seq=0 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.545+0000 7f2a92ff5640 10 client.16311 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2a92ff5640 1 -- 172.21.15.116:0/796695755 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56434cab1120 con 0x56434cb5d3b0 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2a92ff5640 10 client.16311 handle_command_reply: tid=0 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2ab839e740 1 client.16311 shutdown 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2ab839e740 1 -- 172.21.15.116:0/796695755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43510 msgr2=0x56434cb462e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2ab839e740 1 --2- 172.21.15.116:0/796695755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43510 0x56434cb462e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00cc990 tx=0x7f2aa00a4070 comp rx=0 tx=0).stop 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2ab839e740 1 -- 172.21.15.116:0/796695755 shutdown_connections 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2ab839e740 1 --2- 172.21.15.116:0/796695755 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56434cb5d3b0 0x56434cb5f7a0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2ab839e740 1 --2- 172.21.15.116:0/796695755 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb49070 0x56434cb47830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2a92ff5640 0 client.16311 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2ab839e740 1 --2- 172.21.15.116:0/796695755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43f00 0x56434cb46820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2ab839e740 1 --2- 172.21.15.116:0/796695755 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43510 0x56434cb462e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2ab839e740 1 -- 172.21.15.116:0/796695755 >> 172.21.15.116:0/796695755 conn(0x56434cb3a670 msgr2=0x56434cb3b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:58.551 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2ab839e740 1 -- 172.21.15.116:0/796695755 shutdown_connections 2024-03-19T01:25:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2ab839e740 1 -- 172.21.15.116:0/796695755 wait complete. 2024-03-19T01:25:58.552 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.546+0000 7f2ab839e740 20 client.16311 trim_cache size 0 max 16384 2024-03-19T01:25:58.570 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.567+0000 7f2ab839e740 1 Processor -- start 2024-03-19T01:25:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.567+0000 7f2ab839e740 1 -- start start 2024-03-19T01:25:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2ab839e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb43510 0x56434cb45020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2ab839e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 0x56434cb46600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2ab839e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb49070 0x56434cb3f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2ab839e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab01a6b70 con 0x56434cb43f00 2024-03-19T01:25:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2ab839e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab01a86e0 con 0x56434cb49070 2024-03-19T01:25:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2ab839e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab01a7240 con 0x56434cb43510 2024-03-19T01:25:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2aae7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 0x56434cb46600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2aae7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 0x56434cb46600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57978/0 (socket says 172.21.15.116:57978) 2024-03-19T01:25:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2aae7fc640 1 -- 172.21.15.116:0/2127321105 learned_addr learned my addr 172.21.15.116:0/2127321105 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2aaf7fe640 1 --2- 172.21.15.116:0/2127321105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb49070 0x56434cb3f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.571 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2aaeffd640 1 --2- 172.21.15.116:0/2127321105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb43510 0x56434cb45020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2aaf7fe640 1 -- 172.21.15.116:0/2127321105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb43510 msgr2=0x56434cb45020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2aaf7fe640 1 --2- 172.21.15.116:0/2127321105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb43510 0x56434cb45020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2aaf7fe640 1 -- 172.21.15.116:0/2127321105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 msgr2=0x56434cb46600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2aaf7fe640 1 --2- 172.21.15.116:0/2127321105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 0x56434cb46600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2aaf7fe640 1 -- 172.21.15.116:0/2127321105 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56434c884620 con 0x56434cb49070 2024-03-19T01:25:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2aae7fc640 1 --2- 172.21.15.116:0/2127321105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 0x56434cb46600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:25:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.568+0000 7f2aaf7fe640 1 --2- 172.21.15.116:0/2127321105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb49070 0x56434cb3f4e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x56434cb44760 tx=0x7f2aa8008000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3148d7af856ddd15 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.569+0000 7f2a94ff9640 1 -- 172.21.15.116:0/2127321105 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa800fe80 con 0x56434cb49070 2024-03-19T01:25:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.569+0000 7f2a94ff9640 1 -- 172.21.15.116:0/2127321105 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa8005020 con 0x56434cb49070 2024-03-19T01:25:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.569+0000 7f2a94ff9640 1 -- 172.21.15.116:0/2127321105 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa801d6c0 con 0x56434cb49070 2024-03-19T01:25:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.569+0000 7f2ab839e740 1 -- 172.21.15.116:0/2127321105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb49070 msgr2=0x56434cb3f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:58.572 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.569+0000 7f2ab839e740 1 --2- 172.21.15.116:0/2127321105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb49070 0x56434cb3f4e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x56434cb44760 tx=0x7f2aa8008000 comp rx=0 tx=0).stop 2024-03-19T01:25:58.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.569+0000 7f2ab839e740 1 -- 172.21.15.116:0/2127321105 shutdown_connections 2024-03-19T01:25:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.569+0000 7f2ab839e740 1 --2- 172.21.15.116:0/2127321105 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb49070 0x56434cb3f4e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.569+0000 7f2ab839e740 1 --2- 172.21.15.116:0/2127321105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb43f00 0x56434cb46600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.569+0000 7f2ab839e740 1 --2- 172.21.15.116:0/2127321105 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb43510 0x56434cb45020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.569+0000 7f2ab839e740 1 -- 172.21.15.116:0/2127321105 >> 172.21.15.116:0/2127321105 conn(0x56434cb3a670 msgr2=0x56434cb45ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.569+0000 7f2ab839e740 1 -- 172.21.15.116:0/2127321105 shutdown_connections 2024-03-19T01:25:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.570+0000 7f2ab839e740 1 -- 172.21.15.116:0/2127321105 wait complete. 2024-03-19T01:25:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.570+0000 7f2ab839e740 1 -- start start 2024-03-19T01:25:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.570+0000 7f2ab839e740 1 Processor -- start 2024-03-19T01:25:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.570+0000 7f2ab839e740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43510 0x56434cb57ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.570+0000 7f2ab839e740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb43f00 0x56434cb57ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2ab839e740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb49070 0x56434cb59000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2ab839e740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab01a8010 con 0x56434cb49070 2024-03-19T01:25:58.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2ab839e740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab01a7940 con 0x56434cb43510 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2aaeffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43510 0x56434cb57ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2ab839e740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab01a4b50 con 0x56434cb43f00 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2aae7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb43f00 0x56434cb57ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2aae7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb43f00 0x56434cb57ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:45600/0 (socket says 172.21.15.116:45600) 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2aae7fc640 1 -- 172.21.15.116:0/3530807945 learned_addr learned my addr 172.21.15.116:0/3530807945 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2ab839e740 10 client.0 _ll_register_callbacks cb 0x7f2aa0009720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2aaf7fe640 1 --2- 172.21.15.116:0/3530807945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb49070 0x56434cb59000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2aaeffd640 1 -- 172.21.15.116:0/3530807945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb43f00 msgr2=0x56434cb57ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2aaeffd640 1 --2- 172.21.15.116:0/3530807945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb43f00 0x56434cb57ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2aaeffd640 1 -- 172.21.15.116:0/3530807945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb49070 msgr2=0x56434cb59000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2aaeffd640 1 --2- 172.21.15.116:0/3530807945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb49070 0x56434cb59000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2aaeffd640 1 -- 172.21.15.116:0/3530807945 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56434cb595f0 con 0x56434cb43510 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2aaeffd640 1 --2- 172.21.15.116:0/3530807945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43510 0x56434cb57ab0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2aa009b770 tx=0x7f2aa009ac30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef326e1cf2f072e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2a92ff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:25:58.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.571+0000 7f2a92ff5640 1 -- 172.21.15.116:0/3530807945 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa00b3040 con 0x56434cb43510 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.572+0000 7f2a92ff5640 1 -- 172.21.15.116:0/3530807945 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f2aa00a5020 con 0x56434cb43510 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.572+0000 7f2ab839e740 1 -- 172.21.15.116:0/3530807945 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2aa8005020 con 0x56434cb43510 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.572+0000 7f2a92ff5640 1 -- 172.21.15.116:0/3530807945 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f2aa008b050 con 0x56434cb43510 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.572+0000 7f2ab839e740 10 client.26165 fetch_fsmap learned FSMap version 48 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.572+0000 7f2ab839e740 1 -- 172.21.15.116:0/3530807945 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x56434cbc3820 con 0x56434cb43510 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.572+0000 7f2a92ff5640 1 -- 172.21.15.116:0/3530807945 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa8005020 con 0x56434cb43510 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.573+0000 7f2a92ff5640 1 -- 172.21.15.116:0/3530807945 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 48) ==== 8358+0+0 (secure 0 0 0) 0x7f2aa00a3af0 con 0x56434cb43510 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.573+0000 7f2ab839e740 10 client.26165 fetch_fsmap finished waiting for FSMap version 48 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.573+0000 7f2ab839e740 10 client.26165 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.573+0000 7f2ab839e740 20 client.26165 populate_metadata read hostname 'smithi116' 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.573+0000 7f2ab839e740 1 --2- 172.21.15.116:0/3530807945 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56434cbc3fd0 0x56434cbc63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.573+0000 7f2ab839e740 4 client.26165 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.573+0000 7f2ab839e740 1 -- 172.21.15.116:0/3530807945 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56434caab470 con 0x56434cbc3fd0 2024-03-19T01:25:58.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.573+0000 7f2aae7fc640 1 --2- 172.21.15.116:0/3530807945 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56434cbc3fd0 0x56434cbc63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:25:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.573+0000 7f2a92ff5640 10 client.26165 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.573+0000 7f2aae7fc640 1 --2- 172.21.15.116:0/3530807945 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56434cbc3fd0 0x56434cbc63f0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6fc194a45d3ec75 server_cookie=8a1580e56b2f0c06 in_seq=0 out_seq=0 2024-03-19T01:25:58.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.582+0000 7f2a92ff5640 1 -- 172.21.15.116:0/3530807945 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56434caab470 con 0x56434cbc3fd0 2024-03-19T01:25:58.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.582+0000 7f2a92ff5640 10 client.26165 handle_command_reply: tid=0 2024-03-19T01:25:58.584 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.583+0000 7f2ab839e740 1 client.26165 shutdown 2024-03-19T01:25:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.583+0000 7f2ab839e740 1 -- 172.21.15.116:0/3530807945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43510 msgr2=0x56434cb57ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.583+0000 7f2ab839e740 1 --2- 172.21.15.116:0/3530807945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43510 0x56434cb57ab0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2aa009b770 tx=0x7f2aa009ac30 comp rx=0 tx=0).stop 2024-03-19T01:25:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.583+0000 7f2ab839e740 1 -- 172.21.15.116:0/3530807945 shutdown_connections 2024-03-19T01:25:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.583+0000 7f2ab839e740 1 --2- 172.21.15.116:0/3530807945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x56434cb49070 0x56434cb59000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.583+0000 7f2ab839e740 1 --2- 172.21.15.116:0/3530807945 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x56434cbc3fd0 0x56434cbc63f0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.585 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.583+0000 7f2a92ff5640 0 client.26165 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:25:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.583+0000 7f2ab839e740 1 --2- 172.21.15.116:0/3530807945 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x56434cb43f00 0x56434cb57ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.583+0000 7f2ab839e740 1 --2- 172.21.15.116:0/3530807945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56434cb43510 0x56434cb57ab0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.583+0000 7f2ab839e740 1 -- 172.21.15.116:0/3530807945 >> 172.21.15.116:0/3530807945 conn(0x56434cb3a670 msgr2=0x56434cb3d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.584+0000 7f2ab839e740 1 -- 172.21.15.116:0/3530807945 shutdown_connections 2024-03-19T01:25:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.584+0000 7f2ab839e740 1 -- 172.21.15.116:0/3530807945 wait complete. 2024-03-19T01:25:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.584+0000 7f2ab839e740 20 client.26165 trim_cache size 0 max 16384 2024-03-19T01:25:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:25:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:25:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:25:58.586 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:25:58.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.584+0000 7f2ab57b2640 1 -- 172.21.15.116:0/1551529959 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2a980a28a0 msgr2=0x7f2a980a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:58.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.584+0000 7f2ab57b2640 1 --2- 172.21.15.116:0/1551529959 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2a980a28a0 0x7f2a980a4d60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8005c40 tx=0x7f2aa8009500 comp rx=0 tx=0).stop 2024-03-19T01:25:58.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.584+0000 7f2ab57b2640 1 -- 172.21.15.116:0/1551529959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab0102300 msgr2=0x7f2ab0197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:25:58.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.584+0000 7f2ab57b2640 1 --2- 172.21.15.116:0/1551529959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab0102300 0x7f2ab0197ac0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2aa000e9f0 tx=0x7f2aa000edf0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.584+0000 7f2ab57b2640 1 -- 172.21.15.116:0/1551529959 shutdown_connections 2024-03-19T01:25:58.587 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.584+0000 7f2ab57b2640 1 --2- 172.21.15.116:0/1551529959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ab0107560 0x7f2ab019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.584+0000 7f2ab57b2640 1 --2- 172.21.15.116:0/1551529959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f2ab0104c30 0x7f2ab0198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.584+0000 7f2ab57b2640 1 --2- 172.21.15.116:0/1551529959 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f2a980a28a0 0x7f2a980a4d60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.584+0000 7f2ab57b2640 1 --2- 172.21.15.116:0/1551529959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab0102300 0x7f2ab0197ac0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:25:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.584+0000 7f2ab57b2640 1 -- 172.21.15.116:0/1551529959 >> 172.21.15.116:0/1551529959 conn(0x7f2ab00fbcf0 msgr2=0x7f2ab0102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:25:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.585+0000 7f2ab57b2640 1 -- 172.21.15.116:0/1551529959 shutdown_connections 2024-03-19T01:25:58.588 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:25:58.585+0000 7f2ab57b2640 1 -- 172.21.15.116:0/1551529959 wait complete. 2024-03-19T01:25:58.706 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:25:58 smithi116 ceph-mon[62701]: pgmap v752: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 598 op/s 2024-03-19T01:25:58.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:25:58 smithi156 ceph-mon[66799]: pgmap v752: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 598 op/s 2024-03-19T01:25:58.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:25:58 smithi162 ceph-mon[67178]: pgmap v752: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.4 MiB/s wr, 598 op/s 2024-03-19T01:26:00.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:00 smithi156 ceph-mon[66799]: pgmap v753: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.1 MiB/s wr, 580 op/s 2024-03-19T01:26:00.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:00 smithi162 ceph-mon[67178]: pgmap v753: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.1 MiB/s wr, 580 op/s 2024-03-19T01:26:00.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:00 smithi116 ceph-mon[62701]: pgmap v753: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.1 MiB/s wr, 580 op/s 2024-03-19T01:26:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 795.0 s, 1254.6 tps, lat 79.490 ms stddev 84.549 2024-03-19T01:26:02.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:02 smithi156 ceph-mon[66799]: pgmap v754: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.6 MiB/s wr, 635 op/s 2024-03-19T01:26:02.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:02 smithi162 ceph-mon[67178]: pgmap v754: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.6 MiB/s wr, 635 op/s 2024-03-19T01:26:02.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:02 smithi116 ceph-mon[62701]: pgmap v754: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.6 MiB/s wr, 635 op/s 2024-03-19T01:26:04.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:04 smithi156 ceph-mon[66799]: pgmap v755: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.9 MiB/s wr, 549 op/s 2024-03-19T01:26:04.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:04 smithi162 ceph-mon[67178]: pgmap v755: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.9 MiB/s wr, 549 op/s 2024-03-19T01:26:04.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:04 smithi116 ceph-mon[62701]: pgmap v755: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 5.9 MiB/s wr, 549 op/s 2024-03-19T01:26:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 800.0 s, 1231.2 tps, lat 81.154 ms stddev 85.980 2024-03-19T01:26:06.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:06 smithi156 ceph-mon[66799]: pgmap v756: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.2 MiB/s wr, 677 op/s 2024-03-19T01:26:06.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:06 smithi162 ceph-mon[67178]: pgmap v756: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.2 MiB/s wr, 677 op/s 2024-03-19T01:26:06.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:06 smithi116 ceph-mon[62701]: pgmap v756: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.2 MiB/s wr, 677 op/s 2024-03-19T01:26:07.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:07 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:26:07.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:07 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:26:07.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:07 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:26:08.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:08 smithi156 ceph-mon[66799]: pgmap v757: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 MiB/s wr, 621 op/s 2024-03-19T01:26:08.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:08 smithi162 ceph-mon[67178]: pgmap v757: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 MiB/s wr, 621 op/s 2024-03-19T01:26:08.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:08 smithi116 ceph-mon[62701]: pgmap v757: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 MiB/s wr, 621 op/s 2024-03-19T01:26:10.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:10 smithi156 ceph-mon[66799]: pgmap v758: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.4 MiB/s wr, 581 op/s 2024-03-19T01:26:10.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:10 smithi162 ceph-mon[67178]: pgmap v758: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.4 MiB/s wr, 581 op/s 2024-03-19T01:26:10.907 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:10 smithi116 ceph-mon[62701]: pgmap v758: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 5.4 MiB/s wr, 581 op/s 2024-03-19T01:26:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 805.0 s, 1313.2 tps, lat 76.926 ms stddev 82.372 2024-03-19T01:26:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.696+0000 7f7712d3b640 1 -- 172.21.15.116:0/1767631339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c100740 msgr2=0x7f770c100b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.696+0000 7f7712d3b640 1 --2- 172.21.15.116:0/1767631339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c100740 0x7f770c100b60 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f7700009900 tx=0x7f770002bc30 comp rx=0 tx=0).stop 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.696+0000 7f7712d3b640 1 -- 172.21.15.116:0/1767631339 shutdown_connections 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.696+0000 7f7712d3b640 1 --2- 172.21.15.116:0/1767631339 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f770c1022f0 0x7f770c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.696+0000 7f7712d3b640 1 --2- 172.21.15.116:0/1767631339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f770c101930 0x7f770c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.696+0000 7f7712d3b640 1 --2- 172.21.15.116:0/1767631339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c100740 0x7f770c100b60 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.696+0000 7f7712d3b640 1 -- 172.21.15.116:0/1767631339 >> 172.21.15.116:0/1767631339 conn(0x7f770c0fbcb0 msgr2=0x7f770c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.696+0000 7f7712d3b640 1 -- 172.21.15.116:0/1767631339 shutdown_connections 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.696+0000 7f7712d3b640 1 -- 172.21.15.116:0/1767631339 wait complete. 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7712d3b640 1 Processor -- start 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7712d3b640 1 -- start start 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7712d3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c100740 0x7f770c199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7712d3b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f770c101930 0x7f770c19a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7712d3b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f770c1022f0 0x7f770c1a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7712d3b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f770c10f6d0 con 0x7f770c100740 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7712d3b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f770c10f550 con 0x7f770c101930 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7712d3b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f770c10f850 con 0x7f770c1022f0 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7710ad7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c100740 0x7f770c199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7710ad7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c100740 0x7f770c199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42338/0 (socket says 172.21.15.116:42338) 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7710ad7640 1 -- 172.21.15.116:0/3445130310 learned_addr learned my addr 172.21.15.116:0/3445130310 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f77112d8640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f770c1022f0 0x7f770c1a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f770bfff640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f770c101930 0x7f770c19a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7710ad7640 1 -- 172.21.15.116:0/3445130310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f770c1022f0 msgr2=0x7f770c1a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7710ad7640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f770c1022f0 0x7f770c1a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7710ad7640 1 -- 172.21.15.116:0/3445130310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f770c101930 msgr2=0x7f770c19a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7710ad7640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f770c101930 0x7f770c19a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.697+0000 7f7710ad7640 1 -- 172.21.15.116:0/3445130310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770c1a18e0 con 0x7f770c100740 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.698+0000 7f7710ad7640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c100740 0x7f770c199ca0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7700000c00 tx=0x7f7700005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:11.700 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.698+0000 7f7709ffb640 1 -- 172.21.15.116:0/3445130310 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7700041070 con 0x7f770c100740 2024-03-19T01:26:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.698+0000 7f7712d3b640 1 -- 172.21.15.116:0/3445130310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f770c1a1b70 con 0x7f770c100740 2024-03-19T01:26:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.698+0000 7f7709ffb640 1 -- 172.21.15.116:0/3445130310 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77000044d0 con 0x7f770c100740 2024-03-19T01:26:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.698+0000 7f7709ffb640 1 -- 172.21.15.116:0/3445130310 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7700034440 con 0x7f770c100740 2024-03-19T01:26:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.698+0000 7f7712d3b640 1 -- 172.21.15.116:0/3445130310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f770c1a2060 con 0x7f770c100740 2024-03-19T01:26:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.700+0000 7f7709ffb640 1 -- 172.21.15.116:0/3445130310 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7700044070 con 0x7f770c100740 2024-03-19T01:26:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.701+0000 7f7709ffb640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f76fc0a2920 0x7f76fc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.701+0000 7f7709ffb640 1 -- 172.21.15.116:0/3445130310 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f77000f05d0 con 0x7f770c100740 2024-03-19T01:26:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.702+0000 7f770bfff640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f76fc0a2920 0x7f76fc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f770bfff640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f76fc0a2920 0x7f76fc0a4de0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f770c19b1a0 tx=0x7f76f800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f7715a07740 1 Processor -- start 2024-03-19T01:26:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f7715a07740 1 -- start start 2024-03-19T01:26:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f7715a07740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e590520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.704 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f7715a07740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e590af0 0x55680e5955f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f7715a07740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e595c60 0x55680e5980f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f7715a07740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55680e4f4be0 con 0x55680e590af0 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f7715a07740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55680e49d960 con 0x55680e595c60 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f7715a07740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55680e4f0c80 con 0x55680e590100 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f770bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e590af0 0x55680e5955f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f770bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e590af0 0x55680e5955f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42354/0 (socket says 172.21.15.116:42354) 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f7710ad7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e590520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f770bfff640 1 -- 172.21.15.116:0/2170660215 learned_addr learned my addr 172.21.15.116:0/2170660215 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f77112d8640 1 --2- 172.21.15.116:0/2170660215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e595c60 0x55680e5980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f770bfff640 1 -- 172.21.15.116:0/2170660215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 msgr2=0x55680e590520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f770bfff640 1 --2- 172.21.15.116:0/2170660215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e590520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f770bfff640 1 -- 172.21.15.116:0/2170660215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e595c60 msgr2=0x55680e5980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f770bfff640 1 --2- 172.21.15.116:0/2170660215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e595c60 0x55680e5980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.703+0000 7f770bfff640 1 -- 172.21.15.116:0/2170660215 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55680e494d20 con 0x55680e590af0 2024-03-19T01:26:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.704+0000 7f770bfff640 1 --2- 172.21.15.116:0/2170660215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e590af0 0x55680e5955f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f76f800bfd0 tx=0x7f76f8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f577b559366b064c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.704+0000 7f76f6ffd640 1 -- 172.21.15.116:0/2170660215 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f801e620 con 0x55680e590af0 2024-03-19T01:26:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.704+0000 7f76f6ffd640 1 -- 172.21.15.116:0/2170660215 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76f801eca0 con 0x55680e590af0 2024-03-19T01:26:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.704+0000 7f76f6ffd640 1 -- 172.21.15.116:0/2170660215 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f801cc50 con 0x55680e590af0 2024-03-19T01:26:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.705+0000 7f7715a07740 1 -- 172.21.15.116:0/2170660215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e590af0 msgr2=0x55680e5955f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.705+0000 7f7715a07740 1 --2- 172.21.15.116:0/2170660215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e590af0 0x55680e5955f0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f76f800bfd0 tx=0x7f76f8013000 comp rx=0 tx=0).stop 2024-03-19T01:26:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.705+0000 7f7715a07740 1 -- 172.21.15.116:0/2170660215 shutdown_connections 2024-03-19T01:26:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.705+0000 7f7715a07740 1 --2- 172.21.15.116:0/2170660215 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e595c60 0x55680e5980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.705+0000 7f7715a07740 1 --2- 172.21.15.116:0/2170660215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e590af0 0x55680e5955f0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.705+0000 7f7715a07740 1 --2- 172.21.15.116:0/2170660215 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e590520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.710 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.705+0000 7f7715a07740 1 -- 172.21.15.116:0/2170660215 >> 172.21.15.116:0/2170660215 conn(0x55680e58b670 msgr2=0x55680e58ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.705+0000 7f7715a07740 1 -- 172.21.15.116:0/2170660215 shutdown_connections 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.705+0000 7f7715a07740 1 -- 172.21.15.116:0/2170660215 wait complete. 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.705+0000 7f7715a07740 1 -- start start 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.706+0000 7f7715a07740 1 Processor -- start 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.706+0000 7f7715a07740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e59a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.706+0000 7f7715a07740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e59a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.706+0000 7f7715a07740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e595c60 0x55680e5a1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.706+0000 7f7715a07740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55680e598c60 con 0x55680e595c60 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.706+0000 7f7710ad7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e59a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.706+0000 7f7715a07740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55680e4b0db0 con 0x55680e590af0 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.706+0000 7f7715a07740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55680e598de0 con 0x55680e590100 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.706+0000 7f770bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e59a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.706+0000 7f770bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e59a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:60604/0 (socket says 172.21.15.116:60604) 2024-03-19T01:26:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.706+0000 7f7710ad7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e59a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:40322/0 (socket says 172.21.15.116:40322) 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.706+0000 7f770bfff640 1 -- 172.21.15.116:0/3334055524 learned_addr learned my addr 172.21.15.116:0/3334055524 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.706+0000 7f77112d8640 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e595c60 0x55680e5a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f7715a07740 10 client.0 _ll_register_callbacks cb 0x55680e4b5e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f77112d8640 1 -- 172.21.15.116:0/3334055524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 msgr2=0x55680e59a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f77112d8640 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e59a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f77112d8640 1 -- 172.21.15.116:0/3334055524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 msgr2=0x55680e59a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f77112d8640 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e59a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f77112d8640 1 -- 172.21.15.116:0/3334055524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55680e4afc60 con 0x55680e595c60 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f7710ad7640 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e59a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f770bfff640 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e59a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f77112d8640 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e595c60 0x55680e5a1700 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7704005fd0 tx=0x7f770400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59188cfa513e51e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f76ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f76ebfff640 1 -- 172.21.15.116:0/3334055524 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704011e60 con 0x55680e595c60 2024-03-19T01:26:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f7715a07740 1 -- 172.21.15.116:0/3334055524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76f801eca0 con 0x55680e595c60 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f76ebfff640 1 -- 172.21.15.116:0/3334055524 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7704003970 con 0x55680e595c60 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f76ebfff640 1 -- 172.21.15.116:0/3334055524 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f7704009e30 con 0x55680e595c60 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f7715a07740 10 client.16329 fetch_fsmap learned FSMap version 49 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.707+0000 7f7715a07740 1 -- 172.21.15.116:0/3334055524 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55680e209210 con 0x55680e595c60 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.708+0000 7f76ebfff640 1 -- 172.21.15.116:0/3334055524 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76f801eca0 con 0x55680e595c60 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.708+0000 7f76ebfff640 1 -- 172.21.15.116:0/3334055524 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f7704025420 con 0x55680e595c60 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.708+0000 7f7715a07740 10 client.16329 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.708+0000 7f7715a07740 10 client.16329 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.708+0000 7f7715a07740 20 client.16329 populate_metadata read hostname 'smithi116' 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.708+0000 7f7715a07740 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55680e5a62d0 0x55680e5a8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.708+0000 7f7715a07740 4 client.16329 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.708+0000 7f7715a07740 1 -- 172.21.15.116:0/3334055524 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55680e502120 con 0x55680e5a62d0 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.708+0000 7f7710ad7640 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55680e5a62d0 0x55680e5a8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.708+0000 7f7710ad7640 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55680e5a62d0 0x55680e5a8770 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc7f01db7aacec3 server_cookie=2dbc17647857e2c3 in_seq=0 out_seq=0 2024-03-19T01:26:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.708+0000 7f76ebfff640 10 client.16329 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.710+0000 7f76ebfff640 1 -- 172.21.15.116:0/3334055524 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55680e502120 con 0x55680e5a62d0 2024-03-19T01:26:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.710+0000 7f76ebfff640 10 client.16329 handle_command_reply: tid=0 2024-03-19T01:26:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.710+0000 7f7715a07740 1 client.16329 shutdown 2024-03-19T01:26:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.711+0000 7f7715a07740 1 -- 172.21.15.116:0/3334055524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e595c60 msgr2=0x55680e5a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.711+0000 7f7715a07740 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e595c60 0x55680e5a1700 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7704005fd0 tx=0x7f770400f790 comp rx=0 tx=0).stop 2024-03-19T01:26:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.711+0000 7f7715a07740 1 -- 172.21.15.116:0/3334055524 shutdown_connections 2024-03-19T01:26:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.711+0000 7f7715a07740 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e595c60 0x55680e5a1700 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.711+0000 7f7715a07740 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e59a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.711+0000 7f7715a07740 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55680e5a62d0 0x55680e5a8770 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.711+0000 7f7715a07740 1 --2- 172.21.15.116:0/3334055524 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e59a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.711+0000 7f7715a07740 1 -- 172.21.15.116:0/3334055524 >> 172.21.15.116:0/3334055524 conn(0x55680e58b670 msgr2=0x55680e58c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:11.715 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.711+0000 7f76ebfff640 0 client.16329 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.711+0000 7f7715a07740 1 -- 172.21.15.116:0/3334055524 shutdown_connections 2024-03-19T01:26:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.712+0000 7f7715a07740 1 -- 172.21.15.116:0/3334055524 wait complete. 2024-03-19T01:26:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.712+0000 7f7715a07740 20 client.16329 trim_cache size 0 max 16384 2024-03-19T01:26:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.729+0000 7f7715a07740 1 Processor -- start 2024-03-19T01:26:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.729+0000 7f7715a07740 1 -- start start 2024-03-19T01:26:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.729+0000 7f7715a07740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e59a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.729+0000 7f7715a07740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e5a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.729+0000 7f7715a07740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e595c60 0x55680e59a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.729+0000 7f7715a07740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1aaf20 con 0x55680e595c60 2024-03-19T01:26:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.729+0000 7f7715a07740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1aca60 con 0x55680e590af0 2024-03-19T01:26:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.729+0000 7f7715a07740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1ab5f0 con 0x55680e590100 2024-03-19T01:26:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.729+0000 7f77112d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e595c60 0x55680e59a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.729+0000 7f770bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e5a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.731 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.729+0000 7f77112d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e595c60 0x55680e59a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42384/0 (socket says 172.21.15.116:42384) 2024-03-19T01:26:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.729+0000 7f77112d8640 1 -- 172.21.15.116:0/871296510 learned_addr learned my addr 172.21.15.116:0/871296510 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.729+0000 7f7710ad7640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e59a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.730+0000 7f77112d8640 1 -- 172.21.15.116:0/871296510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 msgr2=0x55680e59a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.730+0000 7f77112d8640 1 --2- 172.21.15.116:0/871296510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e59a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.730+0000 7f77112d8640 1 -- 172.21.15.116:0/871296510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 msgr2=0x55680e5a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.730+0000 7f77112d8640 1 --2- 172.21.15.116:0/871296510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e5a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.730+0000 7f77112d8640 1 -- 172.21.15.116:0/871296510 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55680e2d5620 con 0x55680e595c60 2024-03-19T01:26:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.730+0000 7f770bfff640 1 --2- 172.21.15.116:0/871296510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e5a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:26:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.730+0000 7f7710ad7640 1 --2- 172.21.15.116:0/871296510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e59a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:26:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.730+0000 7f77112d8640 1 --2- 172.21.15.116:0/871296510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e595c60 0x55680e59a6c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7704000f80 tx=0x7f7704013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa4e8aeb6bd7eb server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.730+0000 7f76f67fc640 1 -- 172.21.15.116:0/871296510 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704025b00 con 0x55680e595c60 2024-03-19T01:26:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.730+0000 7f76f67fc640 1 -- 172.21.15.116:0/871296510 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7704003e50 con 0x55680e595c60 2024-03-19T01:26:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.730+0000 7f76f67fc640 1 -- 172.21.15.116:0/871296510 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77040053b0 con 0x55680e595c60 2024-03-19T01:26:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.731+0000 7f7715a07740 1 -- 172.21.15.116:0/871296510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e595c60 msgr2=0x55680e59a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.731+0000 7f7715a07740 1 --2- 172.21.15.116:0/871296510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e595c60 0x55680e59a6c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7704000f80 tx=0x7f7704013000 comp rx=0 tx=0).stop 2024-03-19T01:26:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.732+0000 7f7715a07740 1 -- 172.21.15.116:0/871296510 shutdown_connections 2024-03-19T01:26:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.732+0000 7f7715a07740 1 --2- 172.21.15.116:0/871296510 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e595c60 0x55680e59a6c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.732+0000 7f7715a07740 1 --2- 172.21.15.116:0/871296510 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e5a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.732+0000 7f7715a07740 1 --2- 172.21.15.116:0/871296510 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e590100 0x55680e59a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.732+0000 7f7715a07740 1 -- 172.21.15.116:0/871296510 >> 172.21.15.116:0/871296510 conn(0x55680e58b670 msgr2=0x55680e595160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.732+0000 7f7715a07740 1 -- 172.21.15.116:0/871296510 shutdown_connections 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.732+0000 7f7715a07740 1 -- 172.21.15.116:0/871296510 wait complete. 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.732+0000 7f7715a07740 1 -- start start 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.732+0000 7f7715a07740 1 Processor -- start 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f7715a07740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e590100 0x55680e59e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f7715a07740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e59ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f7715a07740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e595c60 0x55680e59fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f7715a07740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1ac390 con 0x55680e590100 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f7715a07740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1abcc0 con 0x55680e590af0 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f7715a07740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f770c1a8f00 con 0x55680e595c60 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f7710ad7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e590100 0x55680e59e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f7715a07740 10 client.0 _ll_register_callbacks cb 0x7f7704009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f770bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e59ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f77112d8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e595c60 0x55680e59fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f770bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e59ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:60622/0 (socket says 172.21.15.116:60622) 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f770bfff640 1 -- 172.21.15.116:0/1305790143 learned_addr learned my addr 172.21.15.116:0/1305790143 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f77112d8640 1 -- 172.21.15.116:0/1305790143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 msgr2=0x55680e59ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f77112d8640 1 --2- 172.21.15.116:0/1305790143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e59ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f77112d8640 1 -- 172.21.15.116:0/1305790143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e590100 msgr2=0x55680e59e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f77112d8640 1 --2- 172.21.15.116:0/1305790143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e590100 0x55680e59e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.733+0000 7f77112d8640 1 -- 172.21.15.116:0/1305790143 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55680e5a0330 con 0x55680e595c60 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f77112d8640 1 --2- 172.21.15.116:0/1305790143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e595c60 0x55680e59fd40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7704005010 tx=0x7f7704025df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67c7204ce5186ff4 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f76ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f76ebfff640 1 -- 172.21.15.116:0/1305790143 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704011e60 con 0x55680e595c60 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f76ebfff640 1 -- 172.21.15.116:0/1305790143 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7704017070 con 0x55680e595c60 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f7715a07740 1 -- 172.21.15.116:0/1305790143 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7704003e50 con 0x55680e595c60 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f76ebfff640 1 -- 172.21.15.116:0/1305790143 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f77040254d0 con 0x55680e595c60 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f7715a07740 10 client.26173 fetch_fsmap learned FSMap version 49 2024-03-19T01:26:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f7715a07740 1 -- 172.21.15.116:0/1305790143 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55680e5a1640 con 0x55680e595c60 2024-03-19T01:26:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f76ebfff640 1 -- 172.21.15.116:0/1305790143 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7704003e50 con 0x55680e595c60 2024-03-19T01:26:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f76ebfff640 1 -- 172.21.15.116:0/1305790143 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f7704036020 con 0x55680e595c60 2024-03-19T01:26:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f7715a07740 10 client.26173 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:26:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f7715a07740 10 client.26173 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:26:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f7715a07740 20 client.26173 populate_metadata read hostname 'smithi116' 2024-03-19T01:26:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f7715a07740 1 --2- 172.21.15.116:0/1305790143 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55680e60ca70 0x55680e60ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f7715a07740 4 client.26173 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:26:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.734+0000 7f7715a07740 1 -- 172.21.15.116:0/1305790143 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55680e4fc470 con 0x55680e60ca70 2024-03-19T01:26:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.735+0000 7f7710ad7640 1 --2- 172.21.15.116:0/1305790143 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55680e60ca70 0x55680e60ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.735+0000 7f7710ad7640 1 --2- 172.21.15.116:0/1305790143 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55680e60ca70 0x55680e60ee90 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70e99037285190f6 server_cookie=5045393f204f2225 in_seq=0 out_seq=0 2024-03-19T01:26:11.739 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.735+0000 7f76ebfff640 10 client.26173 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.743+0000 7f76ebfff640 1 -- 172.21.15.116:0/1305790143 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55680e4fc470 con 0x55680e60ca70 2024-03-19T01:26:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.743+0000 7f76ebfff640 10 client.26173 handle_command_reply: tid=0 2024-03-19T01:26:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.743+0000 7f7715a07740 1 client.26173 shutdown 2024-03-19T01:26:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.743+0000 7f7715a07740 1 -- 172.21.15.116:0/1305790143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e595c60 msgr2=0x55680e59fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.743+0000 7f7715a07740 1 --2- 172.21.15.116:0/1305790143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e595c60 0x55680e59fd40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7704005010 tx=0x7f7704025df0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.743+0000 7f7715a07740 1 -- 172.21.15.116:0/1305790143 shutdown_connections 2024-03-19T01:26:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7715a07740 1 --2- 172.21.15.116:0/1305790143 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55680e60ca70 0x55680e60ee90 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7715a07740 1 --2- 172.21.15.116:0/1305790143 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55680e595c60 0x55680e59fd40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7715a07740 1 --2- 172.21.15.116:0/1305790143 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55680e590af0 0x55680e59ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f76ebfff640 0 client.26173 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7715a07740 1 --2- 172.21.15.116:0/1305790143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55680e590100 0x55680e59e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7715a07740 1 -- 172.21.15.116:0/1305790143 >> 172.21.15.116:0/1305790143 conn(0x55680e58b670 msgr2=0x55680e58a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7715a07740 1 -- 172.21.15.116:0/1305790143 shutdown_connections 2024-03-19T01:26:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7715a07740 1 -- 172.21.15.116:0/1305790143 wait complete. 2024-03-19T01:26:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7715a07740 20 client.26173 trim_cache size 0 max 16384 2024-03-19T01:26:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:26:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:26:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:26:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:26:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7712d3b640 1 -- 172.21.15.116:0/3445130310 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f76fc0a2920 msgr2=0x7f76fc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7712d3b640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f76fc0a2920 0x7f76fc0a4de0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f770c19b1a0 tx=0x7f76f800b410 comp rx=0 tx=0).stop 2024-03-19T01:26:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7712d3b640 1 -- 172.21.15.116:0/3445130310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c100740 msgr2=0x7f770c199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7712d3b640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c100740 0x7f770c199ca0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7700000c00 tx=0x7f7700005d50 comp rx=0 tx=0).stop 2024-03-19T01:26:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7712d3b640 1 -- 172.21.15.116:0/3445130310 shutdown_connections 2024-03-19T01:26:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7712d3b640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f770c1022f0 0x7f770c1a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7712d3b640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f76fc0a2920 0x7f76fc0a4de0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7712d3b640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f770c101930 0x7f770c19a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7712d3b640 1 --2- 172.21.15.116:0/3445130310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f770c100740 0x7f770c199ca0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.744+0000 7f7712d3b640 1 -- 172.21.15.116:0/3445130310 >> 172.21.15.116:0/3445130310 conn(0x7f770c0fbcb0 msgr2=0x7f770c104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.745+0000 7f7712d3b640 1 -- 172.21.15.116:0/3445130310 shutdown_connections 2024-03-19T01:26:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:11.745+0000 7f7712d3b640 1 -- 172.21.15.116:0/3445130310 wait complete. 2024-03-19T01:26:12.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:12 smithi156 ceph-mon[66799]: pgmap v759: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 MiB/s wr, 674 op/s 2024-03-19T01:26:12.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:12 smithi162 ceph-mon[67178]: pgmap v759: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 MiB/s wr, 674 op/s 2024-03-19T01:26:12.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:12 smithi116 ceph-mon[62701]: pgmap v759: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.4 MiB/s wr, 674 op/s 2024-03-19T01:26:14.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:14 smithi156 ceph-mon[66799]: pgmap v760: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.7 MiB/s wr, 587 op/s 2024-03-19T01:26:14.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:14 smithi162 ceph-mon[67178]: pgmap v760: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.7 MiB/s wr, 587 op/s 2024-03-19T01:26:14.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:14 smithi116 ceph-mon[62701]: pgmap v760: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.7 MiB/s wr, 587 op/s 2024-03-19T01:26:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 810.0 s, 1388.6 tps, lat 71.180 ms stddev 81.578 2024-03-19T01:26:16.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:16 smithi156 ceph-mon[66799]: pgmap v761: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.8 MiB/s wr, 641 op/s 2024-03-19T01:26:16.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:16 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:26:16.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:16 smithi162 ceph-mon[67178]: pgmap v761: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.8 MiB/s wr, 641 op/s 2024-03-19T01:26:16.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:16 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:26:16.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:16 smithi116 ceph-mon[62701]: pgmap v761: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.8 MiB/s wr, 641 op/s 2024-03-19T01:26:16.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:16 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:26:17.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:17 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:26:17.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:17 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:26:17.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:17 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:26:17.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:26:17.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:26:17.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:26:17.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:26:17.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:26:17.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:26:18.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:18 smithi156 ceph-mon[66799]: pgmap v762: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.4 MiB/s wr, 594 op/s 2024-03-19T01:26:18.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:18 smithi162 ceph-mon[67178]: pgmap v762: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.4 MiB/s wr, 594 op/s 2024-03-19T01:26:18.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:18 smithi116 ceph-mon[62701]: pgmap v762: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 5.4 MiB/s wr, 594 op/s 2024-03-19T01:26:20.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:20 smithi156 ceph-mon[66799]: pgmap v763: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 583 op/s 2024-03-19T01:26:20.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:20 smithi162 ceph-mon[67178]: pgmap v763: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 583 op/s 2024-03-19T01:26:20.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:20 smithi116 ceph-mon[62701]: pgmap v763: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.3 MiB/s wr, 583 op/s 2024-03-19T01:26:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 815.0 s, 1804.4 tps, lat 55.714 ms stddev 64.222 2024-03-19T01:26:22.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:22 smithi156 ceph-mon[66799]: pgmap v764: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.6 MiB/s wr, 634 op/s 2024-03-19T01:26:22.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:22 smithi162 ceph-mon[67178]: pgmap v764: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.6 MiB/s wr, 634 op/s 2024-03-19T01:26:22.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:22 smithi116 ceph-mon[62701]: pgmap v764: 97 pgs: 97 active+clean; 1.6 GiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.6 MiB/s wr, 634 op/s 2024-03-19T01:26:24.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.875+0000 7f47507d3640 1 -- 172.21.15.116:0/2352753974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748100710 msgr2=0x7f4748100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:24.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.875+0000 7f47507d3640 1 --2- 172.21.15.116:0/2352753974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748100710 0x7f4748100b30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4740009ca0 tx=0x7f474002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.875+0000 7f47507d3640 1 -- 172.21.15.116:0/2352753974 shutdown_connections 2024-03-19T01:26:24.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.875+0000 7f47507d3640 1 --2- 172.21.15.116:0/2352753974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f47481022c0 0x7f4748108b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.878 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.875+0000 7f47507d3640 1 --2- 172.21.15.116:0/2352753974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4748101900 0x7f4748101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.875+0000 7f47507d3640 1 --2- 172.21.15.116:0/2352753974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748100710 0x7f4748100b30 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.875+0000 7f47507d3640 1 -- 172.21.15.116:0/2352753974 >> 172.21.15.116:0/2352753974 conn(0x7f47480fbc80 msgr2=0x7f47480fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.875+0000 7f47507d3640 1 -- 172.21.15.116:0/2352753974 shutdown_connections 2024-03-19T01:26:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.875+0000 7f47507d3640 1 -- 172.21.15.116:0/2352753974 wait complete. 2024-03-19T01:26:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f47507d3640 1 Processor -- start 2024-03-19T01:26:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f47507d3640 1 -- start start 2024-03-19T01:26:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f47507d3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4748100710 0x7f4748069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f47507d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748101900 0x7f474819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f47507d3640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47481022c0 0x7f474819ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f47507d3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f474810c380 con 0x7f4748101900 2024-03-19T01:26:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f47507d3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f474810c200 con 0x7f4748100710 2024-03-19T01:26:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f47507d3640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f474810c500 con 0x7f47481022c0 2024-03-19T01:26:24.879 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f474dd6e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748101900 0x7f474819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f474dd6e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748101900 0x7f474819c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50046/0 (socket says 172.21.15.116:50046) 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f474e56f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4748100710 0x7f4748069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f474dd6e640 1 -- 172.21.15.116:0/3275067733 learned_addr learned my addr 172.21.15.116:0/3275067733 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f474ed70640 1 --2- 172.21.15.116:0/3275067733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47481022c0 0x7f474819ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f474dd6e640 1 -- 172.21.15.116:0/3275067733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47481022c0 msgr2=0x7f474819ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f474dd6e640 1 --2- 172.21.15.116:0/3275067733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47481022c0 0x7f474819ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f474dd6e640 1 -- 172.21.15.116:0/3275067733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4748100710 msgr2=0x7f4748069a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f474dd6e640 1 --2- 172.21.15.116:0/3275067733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4748100710 0x7f4748069a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.876+0000 7f474dd6e640 1 -- 172.21.15.116:0/3275067733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474819f740 con 0x7f4748101900 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.877+0000 7f474dd6e640 1 --2- 172.21.15.116:0/3275067733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748101900 0x7f474819c810 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f473800e9f0 tx=0x7f473800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.877+0000 7f473f7fe640 1 -- 172.21.15.116:0/3275067733 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473800cd80 con 0x7f4748101900 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.877+0000 7f473f7fe640 1 -- 172.21.15.116:0/3275067733 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4738005500 con 0x7f4748101900 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.877+0000 7f47507d3640 1 -- 172.21.15.116:0/3275067733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f474819f9d0 con 0x7f4748101900 2024-03-19T01:26:24.880 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.877+0000 7f473f7fe640 1 -- 172.21.15.116:0/3275067733 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4738010690 con 0x7f4748101900 2024-03-19T01:26:24.881 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.877+0000 7f47507d3640 1 -- 172.21.15.116:0/3275067733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f474819fed0 con 0x7f4748101900 2024-03-19T01:26:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.880+0000 7f473f7fe640 1 -- 172.21.15.116:0/3275067733 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f47380046c0 con 0x7f4748101900 2024-03-19T01:26:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f47533b7740 1 Processor -- start 2024-03-19T01:26:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f47533b7740 1 -- start start 2024-03-19T01:26:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f47533b7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2913510 0x55bce2913930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f47533b7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913f00 0x55bce2918a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f47533b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2919070 0x55bce291b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f47533b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bce2873be0 con 0x55bce2919070 2024-03-19T01:26:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f47533b7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55bce281c960 con 0x55bce2913510 2024-03-19T01:26:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f47533b7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bce286fc80 con 0x55bce2913f00 2024-03-19T01:26:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f474ed70640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2913510 0x55bce2913930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.886 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f473f7fe640 1 --2- 172.21.15.116:0/3275067733 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47340a28a0 0x7f47340a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f474ed70640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2913510 0x55bce2913930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:57816/0 (socket says 172.21.15.116:57816) 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f474ed70640 1 -- 172.21.15.116:0/1447195481 learned_addr learned my addr 172.21.15.116:0/1447195481 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f474e56f640 1 --2- 172.21.15.116:0/1447195481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913f00 0x55bce2918a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f474ed70640 1 --2- 172.21.15.116:0/3275067733 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47340a28a0 0x7f47340a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f473f7fe640 1 -- 172.21.15.116:0/3275067733 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f47380cee60 con 0x7f4748101900 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.881+0000 7f474dd6e640 1 --2- 172.21.15.116:0/1447195481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2919070 0x55bce291b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.882+0000 7f474ed70640 1 -- 172.21.15.116:0/1447195481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913f00 msgr2=0x55bce2918a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.882+0000 7f474ed70640 1 --2- 172.21.15.116:0/1447195481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913f00 0x55bce2918a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.882+0000 7f474ed70640 1 -- 172.21.15.116:0/1447195481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2919070 msgr2=0x55bce291b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.882+0000 7f474ed70640 1 --2- 172.21.15.116:0/1447195481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2919070 0x55bce291b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.882+0000 7f474ed70640 1 -- 172.21.15.116:0/1447195481 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bce2813d20 con 0x55bce2913510 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.882+0000 7f474ed70640 1 --2- 172.21.15.116:0/1447195481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2913510 0x55bce2913930 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f47440138b0 tx=0x7f4744013d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a7784038f1bb960 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.882+0000 7f474ed70640 1 --2- 172.21.15.116:0/3275067733 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47340a28a0 0x7f47340a4d60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f474401e000 tx=0x7f4744020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:24.887 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.882+0000 7f473cff9640 1 -- 172.21.15.116:0/1447195481 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744012280 con 0x55bce2913510 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.882+0000 7f473cff9640 1 -- 172.21.15.116:0/1447195481 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4744023070 con 0x55bce2913510 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.882+0000 7f473cff9640 1 -- 172.21.15.116:0/1447195481 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47440035f0 con 0x55bce2913510 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.883+0000 7f47533b7740 1 -- 172.21.15.116:0/1447195481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2913510 msgr2=0x55bce2913930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.883+0000 7f47533b7740 1 --2- 172.21.15.116:0/1447195481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2913510 0x55bce2913930 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f47440138b0 tx=0x7f4744013d70 comp rx=0 tx=0).stop 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.883+0000 7f47533b7740 1 -- 172.21.15.116:0/1447195481 shutdown_connections 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.883+0000 7f47533b7740 1 --2- 172.21.15.116:0/1447195481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2919070 0x55bce291b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.883+0000 7f47533b7740 1 --2- 172.21.15.116:0/1447195481 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913f00 0x55bce2918a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.883+0000 7f47533b7740 1 --2- 172.21.15.116:0/1447195481 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2913510 0x55bce2913930 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.883+0000 7f47533b7740 1 -- 172.21.15.116:0/1447195481 >> 172.21.15.116:0/1447195481 conn(0x55bce290a670 msgr2=0x55bce290aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.883+0000 7f47533b7740 1 -- 172.21.15.116:0/1447195481 shutdown_connections 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.883+0000 7f47533b7740 1 -- 172.21.15.116:0/1447195481 wait complete. 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.883+0000 7f47533b7740 1 -- start start 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.883+0000 7f47533b7740 1 Processor -- start 2024-03-19T01:26:24.888 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f47533b7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2913510 0x55bce291d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f47533b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce291da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f47533b7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2919070 0x55bce29249f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f47533b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55bce291bfc0 con 0x55bce2913f00 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f47533b7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55bce282fdb0 con 0x55bce2913510 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f47533b7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55bce291c140 con 0x55bce2919070 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f47533b7740 10 client.0 _ll_register_callbacks cb 0x55bce2834e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f474dd6e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce291da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f474ed70640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2919070 0x55bce29249f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f474e56f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2913510 0x55bce291d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f474dd6e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce291da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50074/0 (socket says 172.21.15.116:50074) 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f474dd6e640 1 -- 172.21.15.116:0/2981872411 learned_addr learned my addr 172.21.15.116:0/2981872411 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f474dd6e640 1 -- 172.21.15.116:0/2981872411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2919070 msgr2=0x55bce29249f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f474dd6e640 1 --2- 172.21.15.116:0/2981872411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2919070 0x55bce29249f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.889 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f474dd6e640 1 -- 172.21.15.116:0/2981872411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2913510 msgr2=0x55bce291d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f474dd6e640 1 --2- 172.21.15.116:0/2981872411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2913510 0x55bce291d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f474dd6e640 1 -- 172.21.15.116:0/2981872411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bce282ec60 con 0x55bce2913f00 2024-03-19T01:26:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f474ed70640 1 --2- 172.21.15.116:0/2981872411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2919070 0x55bce29249f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:26:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f474e56f640 1 --2- 172.21.15.116:0/2981872411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2913510 0x55bce291d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:26:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f474dd6e640 1 --2- 172.21.15.116:0/2981872411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce291da20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f473809b4e0 tx=0x7f473809b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401af6c154115f24 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f4731ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:26:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f4731ffb640 1 -- 172.21.15.116:0/2981872411 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f473809f930 con 0x55bce2913f00 2024-03-19T01:26:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f47533b7740 1 -- 172.21.15.116:0/2981872411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4744023070 con 0x55bce2913f00 2024-03-19T01:26:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f4731ffb640 1 -- 172.21.15.116:0/2981872411 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f47380cce00 con 0x55bce2913f00 2024-03-19T01:26:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.884+0000 7f4731ffb640 1 -- 172.21.15.116:0/2981872411 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f47380a3f80 con 0x55bce2913f00 2024-03-19T01:26:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.885+0000 7f47533b7740 10 client.16347 fetch_fsmap learned FSMap version 49 2024-03-19T01:26:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.885+0000 7f47533b7740 1 -- 172.21.15.116:0/2981872411 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55bce2588210 con 0x55bce2913f00 2024-03-19T01:26:24.890 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.885+0000 7f4731ffb640 1 -- 172.21.15.116:0/2981872411 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4744023070 con 0x55bce2913f00 2024-03-19T01:26:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.885+0000 7f4731ffb640 1 -- 172.21.15.116:0/2981872411 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f47380a7020 con 0x55bce2913f00 2024-03-19T01:26:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.885+0000 7f47533b7740 10 client.16347 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:26:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.885+0000 7f47533b7740 10 client.16347 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:26:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.885+0000 7f47533b7740 20 client.16347 populate_metadata read hostname 'smithi116' 2024-03-19T01:26:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.885+0000 7f47533b7740 1 --2- 172.21.15.116:0/2981872411 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55bce2929260 0x55bce292b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.885+0000 7f47533b7740 4 client.16347 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:26:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.885+0000 7f47533b7740 1 -- 172.21.15.116:0/2981872411 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bce2881120 con 0x55bce2929260 2024-03-19T01:26:24.891 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.885+0000 7f474e56f640 1 --2- 172.21.15.116:0/2981872411 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55bce2929260 0x55bce292b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.886+0000 7f474e56f640 1 --2- 172.21.15.116:0/2981872411 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55bce2929260 0x55bce292b650 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=274fcdee3372fc6b server_cookie=23497e004eb9137e in_seq=0 out_seq=0 2024-03-19T01:26:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.886+0000 7f4731ffb640 10 client.16347 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.886+0000 7f4731ffb640 1 -- 172.21.15.116:0/2981872411 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bce2881120 con 0x55bce2929260 2024-03-19T01:26:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.886+0000 7f4731ffb640 10 client.16347 handle_command_reply: tid=0 2024-03-19T01:26:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.886+0000 7f47533b7740 1 client.16347 shutdown 2024-03-19T01:26:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.887+0000 7f47533b7740 1 -- 172.21.15.116:0/2981872411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 msgr2=0x55bce291da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.887+0000 7f47533b7740 1 --2- 172.21.15.116:0/2981872411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce291da20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f473809b4e0 tx=0x7f473809b3e0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.887+0000 7f47533b7740 1 -- 172.21.15.116:0/2981872411 shutdown_connections 2024-03-19T01:26:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.887+0000 7f47533b7740 1 --2- 172.21.15.116:0/2981872411 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55bce2929260 0x55bce292b650 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.887+0000 7f4731ffb640 0 client.16347 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.887+0000 7f47533b7740 1 --2- 172.21.15.116:0/2981872411 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2919070 0x55bce29249f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.893 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.887+0000 7f47533b7740 1 --2- 172.21.15.116:0/2981872411 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce291da20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.887+0000 7f47533b7740 1 --2- 172.21.15.116:0/2981872411 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2913510 0x55bce291d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.887+0000 7f47533b7740 1 -- 172.21.15.116:0/2981872411 >> 172.21.15.116:0/2981872411 conn(0x55bce290a670 msgr2=0x55bce290b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.887+0000 7f47533b7740 1 -- 172.21.15.116:0/2981872411 shutdown_connections 2024-03-19T01:26:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.887+0000 7f47533b7740 1 -- 172.21.15.116:0/2981872411 wait complete. 2024-03-19T01:26:24.894 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.887+0000 7f47533b7740 20 client.16347 trim_cache size 0 max 16384 2024-03-19T01:26:24.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:24 smithi162 ceph-mon[67178]: pgmap v765: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 514 op/s 2024-03-19T01:26:24.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:24 smithi116 ceph-mon[62701]: pgmap v765: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 514 op/s 2024-03-19T01:26:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f47533b7740 1 Processor -- start 2024-03-19T01:26:24.906 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f47533b7740 1 -- start start 2024-03-19T01:26:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f47533b7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913510 0x55bce2909220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f47533b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce29249b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f47533b7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2919070 0x55bce2909760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f47533b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f47481a6a80 con 0x55bce2913f00 2024-03-19T01:26:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f47533b7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f47481a85c0 con 0x55bce2919070 2024-03-19T01:26:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f47533b7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47481a7150 con 0x55bce2913510 2024-03-19T01:26:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f474ed70640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2919070 0x55bce2909760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f474dd6e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce29249b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f474ed70640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2919070 0x55bce2909760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:57838/0 (socket says 172.21.15.116:57838) 2024-03-19T01:26:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f474ed70640 1 -- 172.21.15.116:0/3369183427 learned_addr learned my addr 172.21.15.116:0/3369183427 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f474dd6e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce29249b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50076/0 (socket says 172.21.15.116:50076) 2024-03-19T01:26:24.907 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f474e56f640 1 --2- 172.21.15.116:0/3369183427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913510 0x55bce2909220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f474ed70640 1 -- 172.21.15.116:0/3369183427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913510 msgr2=0x55bce2909220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f474ed70640 1 --2- 172.21.15.116:0/3369183427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913510 0x55bce2909220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f474ed70640 1 -- 172.21.15.116:0/3369183427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 msgr2=0x55bce29249b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.905+0000 7f474ed70640 1 --2- 172.21.15.116:0/3369183427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce29249b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.906+0000 7f474ed70640 1 -- 172.21.15.116:0/3369183427 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bce2654620 con 0x55bce2919070 2024-03-19T01:26:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.906+0000 7f474ed70640 1 --2- 172.21.15.116:0/3369183427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2919070 0x55bce2909760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55bce2914760 tx=0x7f4744005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ad552a3eaf42c95 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.906+0000 7f4733fff640 1 -- 172.21.15.116:0/3369183427 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47440097d0 con 0x55bce2919070 2024-03-19T01:26:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.906+0000 7f4733fff640 1 -- 172.21.15.116:0/3369183427 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4744037070 con 0x55bce2919070 2024-03-19T01:26:24.908 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.906+0000 7f4733fff640 1 -- 172.21.15.116:0/3369183427 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474401fc30 con 0x55bce2919070 2024-03-19T01:26:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.907+0000 7f47533b7740 1 -- 172.21.15.116:0/3369183427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2919070 msgr2=0x55bce2909760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.907+0000 7f47533b7740 1 --2- 172.21.15.116:0/3369183427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2919070 0x55bce2909760 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55bce2914760 tx=0x7f4744005f90 comp rx=0 tx=0).stop 2024-03-19T01:26:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.907+0000 7f47533b7740 1 -- 172.21.15.116:0/3369183427 shutdown_connections 2024-03-19T01:26:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.907+0000 7f47533b7740 1 --2- 172.21.15.116:0/3369183427 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2919070 0x55bce2909760 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.907+0000 7f47533b7740 1 --2- 172.21.15.116:0/3369183427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce29249b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.907+0000 7f47533b7740 1 --2- 172.21.15.116:0/3369183427 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913510 0x55bce2909220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.907+0000 7f47533b7740 1 -- 172.21.15.116:0/3369183427 >> 172.21.15.116:0/3369183427 conn(0x55bce290a670 msgr2=0x55bce2905940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.907+0000 7f47533b7740 1 -- 172.21.15.116:0/3369183427 shutdown_connections 2024-03-19T01:26:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f47533b7740 1 -- 172.21.15.116:0/3369183427 wait complete. 2024-03-19T01:26:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f47533b7740 1 -- start start 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f47533b7740 1 Processor -- start 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f47533b7740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913510 0x55bce2921c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f47533b7740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce2922160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f47533b7740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2919070 0x55bce2923100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f47533b7740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f47481a7ef0 con 0x55bce2913f00 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f47533b7740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f47481a7820 con 0x55bce2919070 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f47533b7740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f47481a4a60 con 0x55bce2913510 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f47533b7740 10 client.0 _ll_register_callbacks cb 0x7f473800cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f474e56f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913510 0x55bce2921c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f474dd6e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce2922160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f474e56f640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913510 0x55bce2921c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:60726/0 (socket says 172.21.15.116:60726) 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f474ed70640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2919070 0x55bce2923100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.908+0000 7f474e56f640 1 -- 172.21.15.116:0/2746446008 learned_addr learned my addr 172.21.15.116:0/2746446008 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:24.913 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f474ed70640 1 -- 172.21.15.116:0/2746446008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913510 msgr2=0x55bce2921c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f474ed70640 1 --2- 172.21.15.116:0/2746446008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913510 0x55bce2921c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f474ed70640 1 -- 172.21.15.116:0/2746446008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 msgr2=0x55bce2922160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f474ed70640 1 --2- 172.21.15.116:0/2746446008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce2922160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f474ed70640 1 -- 172.21.15.116:0/2746446008 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bce29236f0 con 0x55bce2919070 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f474dd6e640 1 --2- 172.21.15.116:0/2746446008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce2922160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f474ed70640 1 --2- 172.21.15.116:0/2746446008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2919070 0x55bce2923100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4744009be0 tx=0x7f474401fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1ed9ef4e70aca0e server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f4731ffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f4731ffb640 1 -- 172.21.15.116:0/2746446008 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474403f070 con 0x55bce2919070 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f47533b7740 1 -- 172.21.15.116:0/2746446008 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4744037070 con 0x55bce2919070 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f4731ffb640 1 -- 172.21.15.116:0/2746446008 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4744009df0 con 0x55bce2919070 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f4731ffb640 1 -- 172.21.15.116:0/2746446008 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f474404a050 con 0x55bce2919070 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f47533b7740 10 client.26201 fetch_fsmap learned FSMap version 49 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f47533b7740 1 -- 172.21.15.116:0/2746446008 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55bce2924a00 con 0x55bce2919070 2024-03-19T01:26:24.914 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.909+0000 7f4731ffb640 1 -- 172.21.15.116:0/2746446008 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4744037070 con 0x55bce2919070 2024-03-19T01:26:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.910+0000 7f4731ffb640 1 -- 172.21.15.116:0/2746446008 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f4744003bf0 con 0x55bce2919070 2024-03-19T01:26:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.910+0000 7f47533b7740 10 client.26201 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:26:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.910+0000 7f47533b7740 10 client.26201 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:26:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.910+0000 7f47533b7740 20 client.26201 populate_metadata read hostname 'smithi116' 2024-03-19T01:26:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.910+0000 7f47533b7740 1 --2- 172.21.15.116:0/2746446008 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55bce2929260 0x55bce2991870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.910+0000 7f47533b7740 4 client.26201 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:26:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.910+0000 7f47533b7740 1 -- 172.21.15.116:0/2746446008 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bce287b470 con 0x55bce2929260 2024-03-19T01:26:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.910+0000 7f474e56f640 1 --2- 172.21.15.116:0/2746446008 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55bce2929260 0x55bce2991870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.911+0000 7f474e56f640 1 --2- 172.21.15.116:0/2746446008 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55bce2929260 0x55bce2991870 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6329f1202084c136 server_cookie=17553c11dc60ecca in_seq=0 out_seq=0 2024-03-19T01:26:24.915 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.911+0000 7f4731ffb640 10 client.26201 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:24.920 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.919+0000 7f4731ffb640 1 -- 172.21.15.116:0/2746446008 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bce287b470 con 0x55bce2929260 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.919+0000 7f4731ffb640 10 client.26201 handle_command_reply: tid=0 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.919+0000 7f47533b7740 1 client.26201 shutdown 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47533b7740 1 -- 172.21.15.116:0/2746446008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2919070 msgr2=0x55bce2923100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47533b7740 1 --2- 172.21.15.116:0/2746446008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2919070 0x55bce2923100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4744009be0 tx=0x7f474401fc00 comp rx=0 tx=0).stop 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47533b7740 1 -- 172.21.15.116:0/2746446008 shutdown_connections 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47533b7740 1 --2- 172.21.15.116:0/2746446008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55bce2919070 0x55bce2923100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47533b7740 1 --2- 172.21.15.116:0/2746446008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55bce2913f00 0x55bce2922160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47533b7740 1 --2- 172.21.15.116:0/2746446008 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55bce2929260 0x55bce2991870 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47533b7740 1 --2- 172.21.15.116:0/2746446008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55bce2913510 0x55bce2921c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f4731ffb640 0 client.26201 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47533b7740 1 -- 172.21.15.116:0/2746446008 >> 172.21.15.116:0/2746446008 conn(0x55bce290a670 msgr2=0x55bce2909220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47533b7740 1 -- 172.21.15.116:0/2746446008 shutdown_connections 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47533b7740 1 -- 172.21.15.116:0/2746446008 wait complete. 2024-03-19T01:26:24.921 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47533b7740 20 client.26201 trim_cache size 0 max 16384 2024-03-19T01:26:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:26:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:26:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:26:24.922 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:26:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47507d3640 1 -- 172.21.15.116:0/3275067733 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47340a28a0 msgr2=0x7f47340a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47507d3640 1 --2- 172.21.15.116:0/3275067733 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47340a28a0 0x7f47340a4d60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f474401e000 tx=0x7f4744020000 comp rx=0 tx=0).stop 2024-03-19T01:26:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47507d3640 1 -- 172.21.15.116:0/3275067733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748101900 msgr2=0x7f474819c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.920+0000 7f47507d3640 1 --2- 172.21.15.116:0/3275067733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748101900 0x7f474819c810 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f473800e9f0 tx=0x7f473800edf0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.921+0000 7f47507d3640 1 -- 172.21.15.116:0/3275067733 shutdown_connections 2024-03-19T01:26:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.921+0000 7f47507d3640 1 --2- 172.21.15.116:0/3275067733 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f47481022c0 0x7f474819ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.921+0000 7f47507d3640 1 --2- 172.21.15.116:0/3275067733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4748101900 0x7f474819c810 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.921+0000 7f47507d3640 1 --2- 172.21.15.116:0/3275067733 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f47340a28a0 0x7f47340a4d60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.921+0000 7f47507d3640 1 --2- 172.21.15.116:0/3275067733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4748100710 0x7f4748069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.921+0000 7f47507d3640 1 -- 172.21.15.116:0/3275067733 >> 172.21.15.116:0/3275067733 conn(0x7f47480fbc80 msgr2=0x7f4748104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:24.923 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.921+0000 7f47507d3640 1 -- 172.21.15.116:0/3275067733 shutdown_connections 2024-03-19T01:26:24.924 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:24.921+0000 7f47507d3640 1 -- 172.21.15.116:0/3275067733 wait complete. 2024-03-19T01:26:24.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:24 smithi156 ceph-mon[66799]: pgmap v765: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 514 op/s 2024-03-19T01:26:26.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 820.0 s, 1731.6 tps, lat 57.815 ms stddev 64.872 2024-03-19T01:26:26.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:26 smithi162 ceph-mon[67178]: pgmap v766: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.3 MiB/s wr, 680 op/s 2024-03-19T01:26:26.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:26 smithi116 ceph-mon[62701]: pgmap v766: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.3 MiB/s wr, 680 op/s 2024-03-19T01:26:26.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:26 smithi156 ceph-mon[66799]: pgmap v766: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.3 MiB/s wr, 680 op/s 2024-03-19T01:26:28.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:28 smithi162 ceph-mon[67178]: pgmap v767: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.9 MiB/s wr, 617 op/s 2024-03-19T01:26:28.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:28 smithi116 ceph-mon[62701]: pgmap v767: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.9 MiB/s wr, 617 op/s 2024-03-19T01:26:28.974 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:28 smithi156 ceph-mon[66799]: pgmap v767: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.9 MiB/s wr, 617 op/s 2024-03-19T01:26:30.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:30 smithi162 ceph-mon[67178]: pgmap v768: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.9 MiB/s wr, 617 op/s 2024-03-19T01:26:30.904 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:30 smithi116 ceph-mon[62701]: pgmap v768: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.9 MiB/s wr, 617 op/s 2024-03-19T01:26:30.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:30 smithi156 ceph-mon[66799]: pgmap v768: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.9 MiB/s wr, 617 op/s 2024-03-19T01:26:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 825.0 s, 1962.0 tps, lat 51.393 ms stddev 61.709 2024-03-19T01:26:32.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:32 smithi162 ceph-mon[67178]: pgmap v769: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.9 MiB/s wr, 638 op/s 2024-03-19T01:26:32.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:32 smithi116 ceph-mon[62701]: pgmap v769: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.9 MiB/s wr, 638 op/s 2024-03-19T01:26:32.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:32 smithi156 ceph-mon[66799]: pgmap v769: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.9 MiB/s wr, 638 op/s 2024-03-19T01:26:34.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:34 smithi162 ceph-mon[67178]: pgmap v770: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.2 MiB/s wr, 554 op/s 2024-03-19T01:26:34.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:34 smithi116 ceph-mon[62701]: pgmap v770: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.2 MiB/s wr, 554 op/s 2024-03-19T01:26:34.973 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:34 smithi156 ceph-mon[66799]: pgmap v770: 97 pgs: 97 active+clean; 1.6 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 5.2 MiB/s wr, 554 op/s 2024-03-19T01:26:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 830.0 s, 1775.2 tps, lat 55.797 ms stddev 62.594 2024-03-19T01:26:37.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:37 smithi156 ceph-mon[66799]: pgmap v771: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 741 op/s 2024-03-19T01:26:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:37 smithi162 ceph-mon[67178]: pgmap v771: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 741 op/s 2024-03-19T01:26:37.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:37 smithi116 ceph-mon[62701]: pgmap v771: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 741 op/s 2024-03-19T01:26:38.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.070+0000 7f9cbbce8640 1 -- 172.21.15.116:0/2131959449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb4108820 msgr2=0x7f9cb410ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:38.071 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.070+0000 7f9cbbce8640 1 --2- 172.21.15.116:0/2131959449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb4108820 0x7f9cb410ac50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f9ca401a910 tx=0x7f9ca403cca0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.073+0000 7f9cbbce8640 1 -- 172.21.15.116:0/2131959449 shutdown_connections 2024-03-19T01:26:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.073+0000 7f9cbbce8640 1 --2- 172.21.15.116:0/2131959449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb410db40 0x7f9cb410dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.073+0000 7f9cbbce8640 1 --2- 172.21.15.116:0/2131959449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cb410b190 0x7f9cb410d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.073 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.073+0000 7f9cbbce8640 1 --2- 172.21.15.116:0/2131959449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb4108820 0x7f9cb410ac50 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.074 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.073+0000 7f9cbbce8640 1 -- 172.21.15.116:0/2131959449 >> 172.21.15.116:0/2131959449 conn(0x7f9cb406e860 msgr2=0x7f9cb4070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:38.076 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.076+0000 7f9cbbce8640 1 -- 172.21.15.116:0/2131959449 shutdown_connections 2024-03-19T01:26:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.076+0000 7f9cbbce8640 1 -- 172.21.15.116:0/2131959449 wait complete. 2024-03-19T01:26:38.077 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.077+0000 7f9cbbce8640 1 Processor -- start 2024-03-19T01:26:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.077+0000 7f9cbbce8640 1 -- start start 2024-03-19T01:26:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.077+0000 7f9cbbce8640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb4108820 0x7f9cb419e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.077+0000 7f9cbbce8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb410b190 0x7f9cb419e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.077+0000 7f9cbbce8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cb410db40 0x7f9cb41a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.078+0000 7f9cbbce8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb4111af0 con 0x7f9cb410b190 2024-03-19T01:26:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.078+0000 7f9cbbce8640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb4111970 con 0x7f9cb410db40 2024-03-19T01:26:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.078+0000 7f9cbbce8640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb4111c70 con 0x7f9cb4108820 2024-03-19T01:26:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.078+0000 7f9cb9a84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb4108820 0x7f9cb419e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.078+0000 7f9cb9a84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb4108820 0x7f9cb419e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:38982/0 (socket says 172.21.15.116:38982) 2024-03-19T01:26:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.078+0000 7f9cb9a84640 1 -- 172.21.15.116:0/3940114046 learned_addr learned my addr 172.21.15.116:0/3940114046 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.078+0000 7f9cb9283640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb410b190 0x7f9cb419e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.079+0000 7f9cba285640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cb410db40 0x7f9cb41a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.079+0000 7f9cb9a84640 1 -- 172.21.15.116:0/3940114046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cb410db40 msgr2=0x7f9cb41a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.079+0000 7f9cb9a84640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cb410db40 0x7f9cb41a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.079+0000 7f9cb9a84640 1 -- 172.21.15.116:0/3940114046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb410b190 msgr2=0x7f9cb419e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.079+0000 7f9cb9a84640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb410b190 0x7f9cb419e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.079+0000 7f9cb9a84640 1 -- 172.21.15.116:0/3940114046 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cb41a5c80 con 0x7f9cb4108820 2024-03-19T01:26:38.080 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.079+0000 7f9cb9a84640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb4108820 0x7f9cb419e060 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9ca403d2b0 tx=0x7f9ca4005750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.080+0000 7f9caaffd640 1 -- 172.21.15.116:0/3940114046 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4052020 con 0x7f9cb4108820 2024-03-19T01:26:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.080+0000 7f9caaffd640 1 -- 172.21.15.116:0/3940114046 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca403dd30 con 0x7f9cb4108820 2024-03-19T01:26:38.081 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.080+0000 7f9caaffd640 1 -- 172.21.15.116:0/3940114046 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4003de0 con 0x7f9cb4108820 2024-03-19T01:26:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.082+0000 7f9cbbce8640 1 -- 172.21.15.116:0/3940114046 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cb41a5f10 con 0x7f9cb4108820 2024-03-19T01:26:38.083 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.083+0000 7f9cbbce8640 1 -- 172.21.15.116:0/3940114046 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cb41a6320 con 0x7f9cb4108820 2024-03-19T01:26:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.087+0000 7f9cbe9ac740 1 Processor -- start 2024-03-19T01:26:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.087+0000 7f9cbe9ac740 1 -- start start 2024-03-19T01:26:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.087+0000 7f9cbe9ac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d29520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.087+0000 7f9cbe9ac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d29af0 0x560c08d2e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.087+0000 7f9cbe9ac740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d2ec60 0x560c08d310f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.087+0000 7f9cbe9ac740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560c08c8dbe0 con 0x560c08d29af0 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.087+0000 7f9cbe9ac740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560c08c36960 con 0x560c08d2ec60 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.087+0000 7f9cbe9ac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560c08c89c80 con 0x560c08d29100 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.088+0000 7f9cb9283640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d2ec60 0x560c08d310f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.088+0000 7f9cb9283640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d2ec60 0x560c08d310f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:32866/0 (socket says 172.21.15.116:32866) 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.088+0000 7f9cb9283640 1 -- 172.21.15.116:0/331668789 learned_addr learned my addr 172.21.15.116:0/331668789 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.088+0000 7f9cba285640 1 --2- 172.21.15.116:0/331668789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d29520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.088+0000 7f9caaffd640 1 -- 172.21.15.116:0/3940114046 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f9ca4055020 con 0x7f9cb4108820 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.088+0000 7f9cb9283640 1 -- 172.21.15.116:0/331668789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 msgr2=0x560c08d29520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.088+0000 7f9cb9283640 1 --2- 172.21.15.116:0/331668789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d29520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.088+0000 7f9cb9283640 1 -- 172.21.15.116:0/331668789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d29af0 msgr2=0x560c08d2e5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:26:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.088+0000 7f9cb9283640 1 --2- 172.21.15.116:0/331668789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d29af0 0x560c08d2e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.088+0000 7f9cb9283640 1 -- 172.21.15.116:0/331668789 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c08c2dd20 con 0x560c08d2ec60 2024-03-19T01:26:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.088+0000 7f9cb9283640 1 --2- 172.21.15.116:0/331668789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d2ec60 0x560c08d310f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000bcb0 tx=0x7f9cb00107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f15a0fedda332e80 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.089+0000 7f9caaffd640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9ca00a2a60 0x7f9ca00a4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.089+0000 7f9c9ffff640 1 -- 172.21.15.116:0/331668789 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0004030 con 0x560c08d2ec60 2024-03-19T01:26:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.089+0000 7f9c9ffff640 1 -- 172.21.15.116:0/331668789 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cb00041d0 con 0x560c08d2ec60 2024-03-19T01:26:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.089+0000 7f9c9ffff640 1 -- 172.21.15.116:0/331668789 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb00116e0 con 0x560c08d2ec60 2024-03-19T01:26:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.089+0000 7f9caaffd640 1 -- 172.21.15.116:0/3940114046 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f9ca4101290 con 0x7f9cb4108820 2024-03-19T01:26:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.089+0000 7f9cba285640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9ca00a2a60 0x7f9ca00a4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.090+0000 7f9cba285640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9ca00a2a60 0x7f9ca00a4f20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x560c08d2a350 tx=0x7f9cac009740 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.090+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/331668789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d2ec60 msgr2=0x560c08d310f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.090+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/331668789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d2ec60 0x560c08d310f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9cb000bcb0 tx=0x7f9cb00107a0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.090+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/331668789 shutdown_connections 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.090+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/331668789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d2ec60 0x560c08d310f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.090+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/331668789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d29af0 0x560c08d2e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.090+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/331668789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d29520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.090+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/331668789 >> 172.21.15.116:0/331668789 conn(0x560c08d24670 msgr2=0x560c08d24a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.090+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/331668789 shutdown_connections 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.090+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/331668789 wait complete. 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.090+0000 7f9cbe9ac740 1 -- start start 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.091+0000 7f9cbe9ac740 1 Processor -- start 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.091+0000 7f9cbe9ac740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29100 0x560c08d33130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.091+0000 7f9cbe9ac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29af0 0x560c08d33670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.091+0000 7f9cbe9ac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d3a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.097 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.091+0000 7f9cbe9ac740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x560c08d31c10 con 0x560c08d2ec60 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.091+0000 7f9cbe9ac740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x560c08c49db0 con 0x560c08d29100 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.091+0000 7f9cbe9ac740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x560c08d31d90 con 0x560c08d29af0 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.091+0000 7f9cbe9ac740 10 client.0 _ll_register_callbacks cb 0x560c08c4ee70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.091+0000 7f9cba285640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d3a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.091+0000 7f9cba285640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d3a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60052/0 (socket says 172.21.15.116:60052) 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.091+0000 7f9cba285640 1 -- 172.21.15.116:0/803016285 learned_addr learned my addr 172.21.15.116:0/803016285 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.092+0000 7f9cb9283640 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29af0 0x560c08d33670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.092+0000 7f9cb9a84640 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29100 0x560c08d33130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.092+0000 7f9cb9a84640 1 -- 172.21.15.116:0/803016285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29af0 msgr2=0x560c08d33670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.092+0000 7f9cb9a84640 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29af0 0x560c08d33670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.092+0000 7f9cb9a84640 1 -- 172.21.15.116:0/803016285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 msgr2=0x560c08d3a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.092+0000 7f9cb9a84640 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d3a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.092+0000 7f9cb9a84640 1 -- 172.21.15.116:0/803016285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c08c48c60 con 0x560c08d29100 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.092+0000 7f9cba285640 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d3a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:26:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.092+0000 7f9cb9a84640 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29100 0x560c08d33130 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x560c08d2a560 tx=0x7f9ca40cdb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c40ebaec360c759 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.092+0000 7f9c9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.092+0000 7f9c9d7fa640 1 -- 172.21.15.116:0/803016285 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca40ce310 con 0x560c08d29100 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.092+0000 7f9c9d7fa640 1 -- 172.21.15.116:0/803016285 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f9ca40035f0 con 0x560c08d29100 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.092+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/803016285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cb00041d0 con 0x560c08d29100 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.093+0000 7f9c9d7fa640 1 -- 172.21.15.116:0/803016285 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f9ca4004ff0 con 0x560c08d29100 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.093+0000 7f9cbe9ac740 10 client.26213 fetch_fsmap learned FSMap version 49 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.093+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/803016285 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x560c089a2210 con 0x560c08d29100 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.093+0000 7f9c9d7fa640 1 -- 172.21.15.116:0/803016285 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cb00041d0 con 0x560c08d29100 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.093+0000 7f9c9d7fa640 1 -- 172.21.15.116:0/803016285 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f9ca40ec1c0 con 0x560c08d29100 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.093+0000 7f9cbe9ac740 10 client.26213 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.093+0000 7f9cbe9ac740 10 client.26213 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.093+0000 7f9cbe9ac740 20 client.26213 populate_metadata read hostname 'smithi116' 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.093+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560c08d3ef30 0x560c08d41340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.093+0000 7f9cbe9ac740 4 client.26213 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.093+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/803016285 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c08c9b120 con 0x560c08d3ef30 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.094+0000 7f9cb9283640 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560c08d3ef30 0x560c08d41340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.094+0000 7f9cb9283640 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560c08d3ef30 0x560c08d41340 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f67f717177e7ae4a server_cookie=fe774212954ca9a6 in_seq=0 out_seq=0 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.094+0000 7f9c9d7fa640 10 client.26213 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9c9d7fa640 1 -- 172.21.15.116:0/803016285 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560c08c9b120 con 0x560c08d3ef30 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9c9d7fa640 10 client.26213 handle_command_reply: tid=0 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9cbe9ac740 1 client.26213 shutdown 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/803016285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29100 msgr2=0x560c08d33130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29100 0x560c08d33130 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x560c08d2a560 tx=0x7f9ca40cdb10 comp rx=0 tx=0).stop 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/803016285 shutdown_connections 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d3a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560c08d3ef30 0x560c08d41340 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29af0 0x560c08d33670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/803016285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29100 0x560c08d33130 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9c9d7fa640 0 client.26213 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/803016285 >> 172.21.15.116:0/803016285 conn(0x560c08d24670 msgr2=0x560c08d258c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/803016285 shutdown_connections 2024-03-19T01:26:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.095+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/803016285 wait complete. 2024-03-19T01:26:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.096+0000 7f9cbe9ac740 20 client.26213 trim_cache size 0 max 16384 2024-03-19T01:26:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.118+0000 7f9cbe9ac740 1 Processor -- start 2024-03-19T01:26:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.118+0000 7f9cbe9ac740 1 -- start start 2024-03-19T01:26:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.118+0000 7f9cbe9ac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d32d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cbe9ac740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29af0 0x560c08d33280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cbe9ac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d2a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cb9a84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d32d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cb9a84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d32d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:39006/0 (socket says 172.21.15.116:39006) 2024-03-19T01:26:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cb9a84640 1 -- 172.21.15.116:0/427844044 learned_addr learned my addr 172.21.15.116:0/427844044 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/427844044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41af270 con 0x560c08d2ec60 2024-03-19T01:26:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/427844044 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41b0db0 con 0x560c08d29af0 2024-03-19T01:26:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/427844044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41af940 con 0x560c08d29100 2024-03-19T01:26:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cba285640 1 --2- 172.21.15.116:0/427844044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d2a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cb9283640 1 --2- 172.21.15.116:0/427844044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29af0 0x560c08d33280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cb9a84640 1 -- 172.21.15.116:0/427844044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29af0 msgr2=0x560c08d33280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cb9a84640 1 --2- 172.21.15.116:0/427844044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29af0 0x560c08d33280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cb9a84640 1 -- 172.21.15.116:0/427844044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 msgr2=0x560c08d2a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cb9a84640 1 --2- 172.21.15.116:0/427844044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d2a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cb9a84640 1 -- 172.21.15.116:0/427844044 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c08a6e620 con 0x560c08d29100 2024-03-19T01:26:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.119+0000 7f9cb9a84640 1 --2- 172.21.15.116:0/427844044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d32d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40ca070 tx=0x7f9ca40cc120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f223b20942701dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.120+0000 7f9c9f7fe640 1 -- 172.21.15.116:0/427844044 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca40ce310 con 0x560c08d29100 2024-03-19T01:26:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.120+0000 7f9c9f7fe640 1 -- 172.21.15.116:0/427844044 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca40cdc10 con 0x560c08d29100 2024-03-19T01:26:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.120+0000 7f9c9f7fe640 1 -- 172.21.15.116:0/427844044 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca40036f0 con 0x560c08d29100 2024-03-19T01:26:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.120+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/427844044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 msgr2=0x560c08d32d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.120+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/427844044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d32d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40ca070 tx=0x7f9ca40cc120 comp rx=0 tx=0).stop 2024-03-19T01:26:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.121+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/427844044 shutdown_connections 2024-03-19T01:26:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.121+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/427844044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d2a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.121+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/427844044 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29af0 0x560c08d33280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.121+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/427844044 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d32d40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.121+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/427844044 >> 172.21.15.116:0/427844044 conn(0x560c08d24670 msgr2=0x560c08d24a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.121+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/427844044 shutdown_connections 2024-03-19T01:26:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.121+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/427844044 wait complete. 2024-03-19T01:26:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.121+0000 7f9cbe9ac740 1 -- start start 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.121+0000 7f9cbe9ac740 1 Processor -- start 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.122+0000 7f9cbe9ac740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d37940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.122+0000 7f9cbe9ac740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29af0 0x560c08d37e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.122+0000 7f9cbe9ac740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d38e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.122+0000 7f9cb9a84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d37940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.122+0000 7f9cb9a84640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d37940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:39014/0 (socket says 172.21.15.116:39014) 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.122+0000 7f9cba285640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d38e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.122+0000 7f9cba285640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d38e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60072/0 (socket says 172.21.15.116:60072) 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.122+0000 7f9cb9283640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29af0 0x560c08d37e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.122+0000 7f9cb9283640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29af0 0x560c08d37e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:32892/0 (socket says 172.21.15.116:32892) 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.122+0000 7f9cb9a84640 1 -- 172.21.15.116:0/2538785328 learned_addr learned my addr 172.21.15.116:0/2538785328 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.122+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/2538785328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41b06e0 con 0x560c08d2ec60 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.122+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/2538785328 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41b0010 con 0x560c08d29af0 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.122+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/2538785328 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb41ad250 con 0x560c08d29100 2024-03-19T01:26:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.123+0000 7f9cbe9ac740 10 client.0 _ll_register_callbacks cb 0x7f9ca40c9f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.123+0000 7f9cb9a84640 1 -- 172.21.15.116:0/2538785328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29af0 msgr2=0x560c08d37e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.123+0000 7f9cb9a84640 1 --2- 172.21.15.116:0/2538785328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29af0 0x560c08d37e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.123+0000 7f9cb9a84640 1 -- 172.21.15.116:0/2538785328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 msgr2=0x560c08d38e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.123+0000 7f9cb9a84640 1 --2- 172.21.15.116:0/2538785328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d38e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.123+0000 7f9cb9a84640 1 -- 172.21.15.116:0/2538785328 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c08d39430 con 0x560c08d29100 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.123+0000 7f9cb9283640 1 --2- 172.21.15.116:0/2538785328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29af0 0x560c08d37e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.123+0000 7f9cb9a84640 1 --2- 172.21.15.116:0/2538785328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d37940 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40ca5e0 tx=0x7f9ca40cd9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16e2a8369998dc61 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.123+0000 7f9c9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.123+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/2538785328 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ca40cdc10 con 0x560c08d29100 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.123+0000 7f9c9d7fa640 1 -- 172.21.15.116:0/2538785328 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca40036d0 con 0x560c08d29100 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.123+0000 7f9c9d7fa640 1 -- 172.21.15.116:0/2538785328 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f9ca40d6130 con 0x560c08d29100 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.124+0000 7f9c9d7fa640 1 -- 172.21.15.116:0/2538785328 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f9ca40b3050 con 0x560c08d29100 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.124+0000 7f9cbe9ac740 10 client.26209 fetch_fsmap learned FSMap version 49 2024-03-19T01:26:38.126 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.124+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/2538785328 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({fsmap=49}) -- 0x560c08d3a5a0 con 0x560c08d29100 2024-03-19T01:26:38.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.124+0000 7f9c9d7fa640 1 -- 172.21.15.116:0/2538785328 <== mon.1 v2:172.21.15.162:3300/0 4 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f9ca403d850 con 0x560c08d29100 2024-03-19T01:26:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.124+0000 7f9cbe9ac740 10 client.26209 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:26:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.124+0000 7f9cbe9ac740 10 client.26209 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:26:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.124+0000 7f9cbe9ac740 20 client.26209 populate_metadata read hostname 'smithi116' 2024-03-19T01:26:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.124+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/2538785328 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560c08da5550 0x560c08da7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.124+0000 7f9cbe9ac740 4 client.26209 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:26:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.124+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/2538785328 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f9ca40cdc10 con 0x560c08da5550 2024-03-19T01:26:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.124+0000 7f9cb9283640 1 --2- 172.21.15.116:0/2538785328 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560c08da5550 0x560c08da7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.125+0000 7f9cb9283640 1 --2- 172.21.15.116:0/2538785328 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560c08da5550 0x560c08da7940 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcf30f107efa9b46 server_cookie=6911c583f737582d in_seq=0 out_seq=0 2024-03-19T01:26:38.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.125+0000 7f9c9d7fa640 10 client.26209 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.137+0000 7f9c9d7fa640 1 -- 172.21.15.116:0/2538785328 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f9ca40cdc10 con 0x560c08da5550 2024-03-19T01:26:38.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.137+0000 7f9c9d7fa640 10 client.26209 handle_command_reply: tid=0 2024-03-19T01:26:38.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.137+0000 7f9cbe9ac740 1 client.26209 shutdown 2024-03-19T01:26:38.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.138+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/2538785328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 msgr2=0x560c08d37940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:38.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.138+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/2538785328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d37940 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40ca5e0 tx=0x7f9ca40cd9e0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.138+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/2538785328 shutdown_connections 2024-03-19T01:26:38.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.138+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/2538785328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x560c08d2ec60 0x560c08d38e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.138+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/2538785328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x560c08d29af0 0x560c08d37e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.138+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/2538785328 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x560c08da5550 0x560c08da7940 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.138+0000 7f9cbe9ac740 1 --2- 172.21.15.116:0/2538785328 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x560c08d29100 0x560c08d37940 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.138+0000 7f9c9d7fa640 0 client.26209 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:38.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.138+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/2538785328 >> 172.21.15.116:0/2538785328 conn(0x560c08d24670 msgr2=0x560c08d32d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:38.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.138+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/2538785328 shutdown_connections 2024-03-19T01:26:38.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.138+0000 7f9cbe9ac740 1 -- 172.21.15.116:0/2538785328 wait complete. 2024-03-19T01:26:38.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.139+0000 7f9cbe9ac740 20 client.26209 trim_cache size 0 max 16384 2024-03-19T01:26:38.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:26:38.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:26:38.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:26:38.140 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:26:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.139+0000 7f9cbbce8640 1 -- 172.21.15.116:0/3940114046 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9ca00a2a60 msgr2=0x7f9ca00a4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.139+0000 7f9cbbce8640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9ca00a2a60 0x7f9ca00a4f20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x560c08d2a350 tx=0x7f9cac009740 comp rx=0 tx=0).stop 2024-03-19T01:26:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.139+0000 7f9cbbce8640 1 -- 172.21.15.116:0/3940114046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb4108820 msgr2=0x7f9cb419e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.139+0000 7f9cbbce8640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb4108820 0x7f9cb419e060 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9ca403d2b0 tx=0x7f9ca4005750 comp rx=0 tx=0).stop 2024-03-19T01:26:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.139+0000 7f9cbbce8640 1 -- 172.21.15.116:0/3940114046 shutdown_connections 2024-03-19T01:26:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.139+0000 7f9cbbce8640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f9ca00a2a60 0x7f9ca00a4f20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.139+0000 7f9cbbce8640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cb410db40 0x7f9cb41a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.139+0000 7f9cbbce8640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cb410b190 0x7f9cb419e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.139+0000 7f9cbbce8640 1 --2- 172.21.15.116:0/3940114046 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9cb4108820 0x7f9cb419e060 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.139+0000 7f9cbbce8640 1 -- 172.21.15.116:0/3940114046 >> 172.21.15.116:0/3940114046 conn(0x7f9cb406e860 msgr2=0x7f9cb4108ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:38.142 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.139+0000 7f9cbbce8640 1 -- 172.21.15.116:0/3940114046 shutdown_connections 2024-03-19T01:26:38.142 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:38.139+0000 7f9cbbce8640 1 -- 172.21.15.116:0/3940114046 wait complete. 2024-03-19T01:26:39.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:39 smithi156 ceph-mon[66799]: pgmap v772: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 MiB/s wr, 613 op/s 2024-03-19T01:26:39.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:39 smithi162 ceph-mon[67178]: pgmap v772: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 MiB/s wr, 613 op/s 2024-03-19T01:26:39.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:39 smithi116 ceph-mon[62701]: pgmap v772: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.9 MiB/s wr, 613 op/s 2024-03-19T01:26:41.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:41 smithi156 ceph-mon[66799]: pgmap v773: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 658 op/s 2024-03-19T01:26:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 835.0 s, 1130.6 tps, lat 87.513 ms stddev 90.189 2024-03-19T01:26:41.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:41 smithi162 ceph-mon[67178]: pgmap v773: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 658 op/s 2024-03-19T01:26:41.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:41 smithi116 ceph-mon[62701]: pgmap v773: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 658 op/s 2024-03-19T01:26:43.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:43 smithi156 ceph-mon[66799]: pgmap v774: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 667 op/s 2024-03-19T01:26:43.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:43 smithi162 ceph-mon[67178]: pgmap v774: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 667 op/s 2024-03-19T01:26:43.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:43 smithi116 ceph-mon[62701]: pgmap v774: 97 pgs: 97 active+clean; 1.6 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 667 op/s 2024-03-19T01:26:45.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:45 smithi156 ceph-mon[66799]: pgmap v775: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.7 MiB/s wr, 589 op/s 2024-03-19T01:26:45.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:45 smithi162 ceph-mon[67178]: pgmap v775: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.7 MiB/s wr, 589 op/s 2024-03-19T01:26:45.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:45 smithi116 ceph-mon[62701]: pgmap v775: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.7 MiB/s wr, 589 op/s 2024-03-19T01:26:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 840.0 s, 1726.2 tps, lat 59.005 ms stddev 80.864 2024-03-19T01:26:47.451 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:47 smithi162 ceph-mon[67178]: pgmap v776: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 731 op/s 2024-03-19T01:26:47.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:47 smithi156 ceph-mon[66799]: pgmap v776: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 731 op/s 2024-03-19T01:26:47.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:47 smithi116 ceph-mon[62701]: pgmap v776: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 MiB/s wr, 731 op/s 2024-03-19T01:26:49.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:49 smithi156 ceph-mon[66799]: pgmap v777: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 574 op/s 2024-03-19T01:26:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:49 smithi162 ceph-mon[67178]: pgmap v777: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 574 op/s 2024-03-19T01:26:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:49 smithi116 ceph-mon[62701]: pgmap v777: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.1 MiB/s wr, 574 op/s 2024-03-19T01:26:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.258+0000 7fd511ec6640 1 -- 172.21.15.116:0/2804863982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd50c101930 msgr2=0x7fd50c101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.258+0000 7fd511ec6640 1 --2- 172.21.15.116:0/2804863982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd50c101930 0x7fd50c101db0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc009860 tx=0x7fd4fc02bb70 comp rx=0 tx=0).stop 2024-03-19T01:26:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.258+0000 7fd511ec6640 1 -- 172.21.15.116:0/2804863982 shutdown_connections 2024-03-19T01:26:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.258+0000 7fd511ec6640 1 --2- 172.21.15.116:0/2804863982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd50c1022f0 0x7fd50c108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.258+0000 7fd511ec6640 1 --2- 172.21.15.116:0/2804863982 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd50c101930 0x7fd50c101db0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.258+0000 7fd511ec6640 1 --2- 172.21.15.116:0/2804863982 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd50c100740 0x7fd50c100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.258+0000 7fd511ec6640 1 -- 172.21.15.116:0/2804863982 >> 172.21.15.116:0/2804863982 conn(0x7fd50c0fbcb0 msgr2=0x7fd50c0fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.258+0000 7fd511ec6640 1 -- 172.21.15.116:0/2804863982 shutdown_connections 2024-03-19T01:26:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.258+0000 7fd511ec6640 1 -- 172.21.15.116:0/2804863982 wait complete. 2024-03-19T01:26:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd511ec6640 1 Processor -- start 2024-03-19T01:26:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd511ec6640 1 -- start start 2024-03-19T01:26:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd511ec6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd50c100740 0x7fd50c069a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd511ec6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd50c101930 0x7fd50c19c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd511ec6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd50c1022f0 0x7fd50c19ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd511ec6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c10c3a0 con 0x7fd50c101930 2024-03-19T01:26:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd511ec6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c10c220 con 0x7fd50c1022f0 2024-03-19T01:26:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd511ec6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c10c520 con 0x7fd50c100740 2024-03-19T01:26:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd50b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd50c100740 0x7fd50c069a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd50affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd50c101930 0x7fd50c19c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd50b7fe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd50c100740 0x7fd50c069a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47138/0 (socket says 172.21.15.116:47138) 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd50bfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd50c1022f0 0x7fd50c19ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd50affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd50c101930 0x7fd50c19c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58066/0 (socket says 172.21.15.116:58066) 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd50b7fe640 1 -- 172.21.15.116:0/3313817728 learned_addr learned my addr 172.21.15.116:0/3313817728 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd50affd640 1 -- 172.21.15.116:0/3313817728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd50c100740 msgr2=0x7fd50c069a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd50affd640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd50c100740 0x7fd50c069a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.259+0000 7fd50affd640 1 -- 172.21.15.116:0/3313817728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd50c1022f0 msgr2=0x7fd50c19ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.260+0000 7fd50affd640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd50c1022f0 0x7fd50c19ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.260+0000 7fd50affd640 1 -- 172.21.15.116:0/3313817728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd50c19f770 con 0x7fd50c101930 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.260+0000 7fd50b7fe640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd50c100740 0x7fd50c069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.260+0000 7fd50bfff640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd50c1022f0 0x7fd50c19ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.260+0000 7fd50affd640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd50c101930 0x7fd50c19c840 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc02c0b0 tx=0x7fd4fc032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.260+0000 7fd508ff9640 1 -- 172.21.15.116:0/3313817728 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc041020 con 0x7fd50c101930 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.260+0000 7fd511ec6640 1 -- 172.21.15.116:0/3313817728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd50c19f9a0 con 0x7fd50c101930 2024-03-19T01:26:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.260+0000 7fd511ec6640 1 -- 172.21.15.116:0/3313817728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd50c19fee0 con 0x7fd50c101930 2024-03-19T01:26:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.260+0000 7fd508ff9640 1 -- 172.21.15.116:0/3313817728 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4fc02c9f0 con 0x7fd50c101930 2024-03-19T01:26:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.260+0000 7fd508ff9640 1 -- 172.21.15.116:0/3313817728 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc039040 con 0x7fd50c101930 2024-03-19T01:26:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.262+0000 7fd514b92740 1 Processor -- start 2024-03-19T01:26:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.263+0000 7fd514b92740 1 -- start start 2024-03-19T01:26:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.263+0000 7fd514b92740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7a080 0x559feae7a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.263+0000 7fd514b92740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7aa70 0x559feae7d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.263+0000 7fd514b92740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae8aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.263+0000 7fd514b92740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559feaddebe0 con 0x559feae7aa70 2024-03-19T01:26:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.263+0000 7fd514b92740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559fead87960 con 0x559feae7dbd0 2024-03-19T01:26:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.263+0000 7fd50b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7aa70 0x559feae7d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.263+0000 7fd50b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7aa70 0x559feae7d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58078/0 (socket says 172.21.15.116:58078) 2024-03-19T01:26:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.263+0000 7fd50b7fe640 1 -- 172.21.15.116:0/79996310 learned_addr learned my addr 172.21.15.116:0/79996310 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.263+0000 7fd508ff9640 1 -- 172.21.15.116:0/3313817728 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fd4fc044020 con 0x7fd50c101930 2024-03-19T01:26:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.263+0000 7fd514b92740 1 -- 172.21.15.116:0/79996310 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559feaddac80 con 0x559feae7a080 2024-03-19T01:26:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.264+0000 7fd50bfff640 1 --2- 172.21.15.116:0/79996310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7a080 0x559feae7a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.264+0000 7fd50affd640 1 --2- 172.21.15.116:0/79996310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae8aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.264+0000 7fd508ff9640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd4f40a29a0 0x7fd4f40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.264+0000 7fd50b7fe640 1 -- 172.21.15.116:0/79996310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7a080 msgr2=0x559feae7a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.264+0000 7fd50b7fe640 1 --2- 172.21.15.116:0/79996310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7a080 0x559feae7a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.264+0000 7fd50b7fe640 1 -- 172.21.15.116:0/79996310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 msgr2=0x559feae8aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.264+0000 7fd50b7fe640 1 --2- 172.21.15.116:0/79996310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae8aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.264+0000 7fd50b7fe640 1 -- 172.21.15.116:0/79996310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fead7ed20 con 0x559feae7aa70 2024-03-19T01:26:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.264+0000 7fd508ff9640 1 -- 172.21.15.116:0/3313817728 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd4fc0efc80 con 0x7fd50c101930 2024-03-19T01:26:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.265+0000 7fd50b7fe640 1 --2- 172.21.15.116:0/79996310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7aa70 0x559feae7d560 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd500009fd0 tx=0x7fd500011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a12cb421f71be8b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.265+0000 7fd50bfff640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd4f40a29a0 0x7fd4f40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.265+0000 7fd4f1ffb640 1 -- 172.21.15.116:0/79996310 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50001e070 con 0x559feae7aa70 2024-03-19T01:26:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.265+0000 7fd4f1ffb640 1 -- 172.21.15.116:0/79996310 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd50000fb20 con 0x559feae7aa70 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.265+0000 7fd4f1ffb640 1 -- 172.21.15.116:0/79996310 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd500019ce0 con 0x559feae7aa70 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.266+0000 7fd50bfff640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd4f40a29a0 0x7fd4f40a4e60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x559feae7b2d0 tx=0x7fd504004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.266+0000 7fd514b92740 1 -- 172.21.15.116:0/79996310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7aa70 msgr2=0x559feae7d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.266+0000 7fd514b92740 1 --2- 172.21.15.116:0/79996310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7aa70 0x559feae7d560 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd500009fd0 tx=0x7fd500011a50 comp rx=0 tx=0).stop 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.266+0000 7fd514b92740 1 -- 172.21.15.116:0/79996310 shutdown_connections 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.266+0000 7fd514b92740 1 --2- 172.21.15.116:0/79996310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae8aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.266+0000 7fd514b92740 1 --2- 172.21.15.116:0/79996310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7aa70 0x559feae7d560 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.266+0000 7fd514b92740 1 --2- 172.21.15.116:0/79996310 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7a080 0x559feae7a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.266+0000 7fd514b92740 1 -- 172.21.15.116:0/79996310 >> 172.21.15.116:0/79996310 conn(0x559feae75670 msgr2=0x559feae75a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.267+0000 7fd514b92740 1 -- 172.21.15.116:0/79996310 shutdown_connections 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.267+0000 7fd514b92740 1 -- 172.21.15.116:0/79996310 wait complete. 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.267+0000 7fd514b92740 1 -- start start 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.267+0000 7fd514b92740 1 Processor -- start 2024-03-19T01:26:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.267+0000 7fd514b92740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 0x559feae8c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.267+0000 7fd514b92740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7aa70 0x559feae8cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.267+0000 7fd514b92740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae93e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd514b92740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559feae8b440 con 0x559feae7a080 2024-03-19T01:26:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd514b92740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559fead9adb0 con 0x559feae7dbd0 2024-03-19T01:26:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd514b92740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x559feae8b5c0 con 0x559feae7aa70 2024-03-19T01:26:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd50affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7aa70 0x559feae8cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd50b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 0x559feae8c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd50affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7aa70 0x559feae8cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47158/0 (socket says 172.21.15.116:47158) 2024-03-19T01:26:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd50affd640 1 -- 172.21.15.116:0/2272750299 learned_addr learned my addr 172.21.15.116:0/2272750299 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd50bfff640 1 --2- 172.21.15.116:0/2272750299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae93e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd514b92740 10 client.0 _ll_register_callbacks cb 0x559fead9fe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:26:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd50bfff640 1 -- 172.21.15.116:0/2272750299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7aa70 msgr2=0x559feae8cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd50bfff640 1 --2- 172.21.15.116:0/2272750299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7aa70 0x559feae8cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd50bfff640 1 -- 172.21.15.116:0/2272750299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 msgr2=0x559feae8c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd50bfff640 1 --2- 172.21.15.116:0/2272750299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 0x559feae8c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd50bfff640 1 -- 172.21.15.116:0/2272750299 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fead99c60 con 0x559feae7dbd0 2024-03-19T01:26:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd50b7fe640 1 --2- 172.21.15.116:0/2272750299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 0x559feae8c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:26:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd50bfff640 1 --2- 172.21.15.116:0/2272750299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae93e00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd504009fd0 tx=0x7fd504004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8bcdb6e6ae3b3b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd4eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:26:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd4eaffd640 1 -- 172.21.15.116:0/2272750299 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50402d060 con 0x559feae7dbd0 2024-03-19T01:26:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.268+0000 7fd514b92740 1 -- 172.21.15.116:0/2272750299 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd50000fb20 con 0x559feae7dbd0 2024-03-19T01:26:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.269+0000 7fd4eaffd640 1 -- 172.21.15.116:0/2272750299 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd504028600 con 0x559feae7dbd0 2024-03-19T01:26:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.269+0000 7fd4eaffd640 1 -- 172.21.15.116:0/2272750299 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fd50401de40 con 0x559feae7dbd0 2024-03-19T01:26:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.269+0000 7fd514b92740 10 client.26231 fetch_fsmap learned FSMap version 49 2024-03-19T01:26:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.269+0000 7fd514b92740 1 -- 172.21.15.116:0/2272750299 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x559feaaf3210 con 0x559feae7dbd0 2024-03-19T01:26:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.269+0000 7fd4eaffd640 1 -- 172.21.15.116:0/2272750299 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd50000fb20 con 0x559feae7dbd0 2024-03-19T01:26:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.269+0000 7fd4eaffd640 1 -- 172.21.15.116:0/2272750299 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fd504033970 con 0x559feae7dbd0 2024-03-19T01:26:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.269+0000 7fd514b92740 10 client.26231 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:26:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.269+0000 7fd514b92740 10 client.26231 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:26:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.269+0000 7fd514b92740 20 client.26231 populate_metadata read hostname 'smithi116' 2024-03-19T01:26:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.269+0000 7fd514b92740 1 --2- 172.21.15.116:0/2272750299 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559feae98750 0x559feae9ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.269+0000 7fd514b92740 4 client.26231 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:26:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.269+0000 7fd514b92740 1 -- 172.21.15.116:0/2272750299 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559feadec120 con 0x559feae98750 2024-03-19T01:26:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.270+0000 7fd50b7fe640 1 --2- 172.21.15.116:0/2272750299 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559feae98750 0x559feae9ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.270+0000 7fd4eaffd640 10 client.26231 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.270+0000 7fd50b7fe640 1 --2- 172.21.15.116:0/2272750299 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559feae98750 0x559feae9ab60 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72801545a3ea7448 server_cookie=33506670f9aa2799 in_seq=0 out_seq=0 2024-03-19T01:26:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.271+0000 7fd4eaffd640 1 -- 172.21.15.116:0/2272750299 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559feadec120 con 0x559feae98750 2024-03-19T01:26:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.271+0000 7fd4eaffd640 10 client.26231 handle_command_reply: tid=0 2024-03-19T01:26:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.271+0000 7fd514b92740 1 client.26231 shutdown 2024-03-19T01:26:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.272+0000 7fd514b92740 1 -- 172.21.15.116:0/2272750299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 msgr2=0x559feae93e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.272+0000 7fd514b92740 1 --2- 172.21.15.116:0/2272750299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae93e00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd504009fd0 tx=0x7fd504004ab0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.272+0000 7fd514b92740 1 -- 172.21.15.116:0/2272750299 shutdown_connections 2024-03-19T01:26:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.272+0000 7fd514b92740 1 --2- 172.21.15.116:0/2272750299 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae93e00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.272+0000 7fd514b92740 1 --2- 172.21.15.116:0/2272750299 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559feae98750 0x559feae9ab60 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.272+0000 7fd514b92740 1 --2- 172.21.15.116:0/2272750299 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7aa70 0x559feae8cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.272+0000 7fd4eaffd640 0 client.26231 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.272+0000 7fd514b92740 1 --2- 172.21.15.116:0/2272750299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 0x559feae8c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.272+0000 7fd514b92740 1 -- 172.21.15.116:0/2272750299 >> 172.21.15.116:0/2272750299 conn(0x559feae75670 msgr2=0x559feae768c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.272+0000 7fd514b92740 1 -- 172.21.15.116:0/2272750299 shutdown_connections 2024-03-19T01:26:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.272+0000 7fd514b92740 1 -- 172.21.15.116:0/2272750299 wait complete. 2024-03-19T01:26:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.272+0000 7fd514b92740 20 client.26231 trim_cache size 0 max 16384 2024-03-19T01:26:51.315 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.314+0000 7fd514b92740 1 Processor -- start 2024-03-19T01:26:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.316+0000 7fd514b92740 1 -- start start 2024-03-19T01:26:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.317+0000 7fd514b92740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 0x559feae8bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.317+0000 7fd514b92740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7aa70 0x559feae93dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.317 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.317+0000 7fd514b92740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae8c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.317+0000 7fd50affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7aa70 0x559feae93dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.317+0000 7fd50affd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7aa70 0x559feae93dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:47160/0 (socket says 172.21.15.116:47160) 2024-03-19T01:26:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.317+0000 7fd50affd640 1 -- 172.21.15.116:0/2717580366 learned_addr learned my addr 172.21.15.116:0/2717580366 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:51.318 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.318+0000 7fd50bfff640 1 --2- 172.21.15.116:0/2717580366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae8c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.318+0000 7fd514b92740 1 -- 172.21.15.116:0/2717580366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c1a6bd0 con 0x559feae7a080 2024-03-19T01:26:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.318+0000 7fd514b92740 1 -- 172.21.15.116:0/2717580366 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c1a8710 con 0x559feae7dbd0 2024-03-19T01:26:51.319 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.318+0000 7fd514b92740 1 -- 172.21.15.116:0/2717580366 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c1a72a0 con 0x559feae7aa70 2024-03-19T01:26:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.324+0000 7fd50bfff640 1 -- 172.21.15.116:0/2717580366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7aa70 msgr2=0x559feae93dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:51.324 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.324+0000 7fd50bfff640 1 --2- 172.21.15.116:0/2717580366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7aa70 0x559feae93dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.324+0000 7fd50bfff640 1 -- 172.21.15.116:0/2717580366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 msgr2=0x559feae8bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.324+0000 7fd50bfff640 1 --2- 172.21.15.116:0/2717580366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 0x559feae8bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.325 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.324+0000 7fd50bfff640 1 -- 172.21.15.116:0/2717580366 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559feabbf620 con 0x559feae7dbd0 2024-03-19T01:26:51.328 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.328+0000 7fd50bfff640 1 --2- 172.21.15.116:0/2717580366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae8c0b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd504033e60 tx=0x7fd504005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8612149501c48723 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:51.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.332+0000 7fd4f17fa640 1 -- 172.21.15.116:0/2717580366 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd504004d20 con 0x559feae7dbd0 2024-03-19T01:26:51.332 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.332+0000 7fd4f17fa640 1 -- 172.21.15.116:0/2717580366 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5040338e0 con 0x559feae7dbd0 2024-03-19T01:26:51.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.333+0000 7fd514b92740 1 -- 172.21.15.116:0/2717580366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 msgr2=0x559feae8c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:51.333 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.333+0000 7fd514b92740 1 --2- 172.21.15.116:0/2717580366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae8c0b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd504033e60 tx=0x7fd504005fd0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.334+0000 7fd514b92740 1 -- 172.21.15.116:0/2717580366 shutdown_connections 2024-03-19T01:26:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.334+0000 7fd514b92740 1 --2- 172.21.15.116:0/2717580366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7dbd0 0x559feae8c0b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.334+0000 7fd514b92740 1 --2- 172.21.15.116:0/2717580366 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7aa70 0x559feae93dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.334+0000 7fd514b92740 1 --2- 172.21.15.116:0/2717580366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 0x559feae8bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.334+0000 7fd514b92740 1 -- 172.21.15.116:0/2717580366 >> 172.21.15.116:0/2717580366 conn(0x559feae75670 msgr2=0x559feae8a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.342+0000 7fd514b92740 1 -- 172.21.15.116:0/2717580366 shutdown_connections 2024-03-19T01:26:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.342+0000 7fd514b92740 1 -- 172.21.15.116:0/2717580366 wait complete. 2024-03-19T01:26:51.342 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.342+0000 7fd514b92740 1 -- start start 2024-03-19T01:26:51.343 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.343+0000 7fd514b92740 1 Processor -- start 2024-03-19T01:26:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.343+0000 7fd514b92740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 0x559feae90da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.343+0000 7fd514b92740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7aa70 0x559feae912e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.344 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.343+0000 7fd50b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 0x559feae90da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.343+0000 7fd50b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 0x559feae90da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58110/0 (socket says 172.21.15.116:58110) 2024-03-19T01:26:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.344+0000 7fd50b7fe640 1 -- 172.21.15.116:0/1665395811 learned_addr learned my addr 172.21.15.116:0/1665395811 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:26:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.344+0000 7fd514b92740 1 --2- 172.21.15.116:0/1665395811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7dbd0 0x559feaefca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.344+0000 7fd50affd640 1 --2- 172.21.15.116:0/1665395811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7aa70 0x559feae912e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.344+0000 7fd514b92740 1 -- 172.21.15.116:0/1665395811 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c1a8040 con 0x559feae7a080 2024-03-19T01:26:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.344+0000 7fd514b92740 1 -- 172.21.15.116:0/1665395811 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c1a7970 con 0x559feae7aa70 2024-03-19T01:26:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.344+0000 7fd514b92740 1 -- 172.21.15.116:0/1665395811 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fd50c1a4bb0 con 0x559feae7dbd0 2024-03-19T01:26:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.344+0000 7fd514b92740 10 client.0 _ll_register_callbacks cb 0x7fd504011a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:26:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.345+0000 7fd50affd640 1 -- 172.21.15.116:0/1665395811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7dbd0 msgr2=0x559feaefca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.345+0000 7fd50affd640 1 --2- 172.21.15.116:0/1665395811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7dbd0 0x559feaefca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.345+0000 7fd50affd640 1 -- 172.21.15.116:0/1665395811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 msgr2=0x559feae90da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:26:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.345+0000 7fd50affd640 1 --2- 172.21.15.116:0/1665395811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 0x559feae90da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.346 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.345+0000 7fd50affd640 1 -- 172.21.15.116:0/1665395811 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559feaefcfd0 con 0x559feae7aa70 2024-03-19T01:26:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.349+0000 7fd50affd640 1 --2- 172.21.15.116:0/1665395811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7aa70 0x559feae912e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc000c00 tx=0x7fd4fc02bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e9522bc87e3d731 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:26:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.349+0000 7fd4eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:26:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.354+0000 7fd4eaffd640 1 -- 172.21.15.116:0/1665395811 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc0bc0e0 con 0x559feae7aa70 2024-03-19T01:26:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.354+0000 7fd4eaffd640 1 -- 172.21.15.116:0/1665395811 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fd4fc032640 con 0x559feae7aa70 2024-03-19T01:26:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.355+0000 7fd514b92740 1 -- 172.21.15.116:0/1665395811 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5040338e0 con 0x559feae7aa70 2024-03-19T01:26:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.356+0000 7fd4eaffd640 1 -- 172.21.15.116:0/1665395811 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fd4fc0d5050 con 0x559feae7aa70 2024-03-19T01:26:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.357+0000 7fd514b92740 10 client.26237 fetch_fsmap learned FSMap version 49 2024-03-19T01:26:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.357+0000 7fd514b92740 1 -- 172.21.15.116:0/1665395811 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x559feaefe370 con 0x559feae7aa70 2024-03-19T01:26:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.358+0000 7fd4eaffd640 1 -- 172.21.15.116:0/1665395811 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5040338e0 con 0x559feae7aa70 2024-03-19T01:26:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.358+0000 7fd4eaffd640 1 -- 172.21.15.116:0/1665395811 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fd4fc03c030 con 0x559feae7aa70 2024-03-19T01:26:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.358+0000 7fd514b92740 10 client.26237 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:26:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.358+0000 7fd514b92740 10 client.26237 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:26:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.358+0000 7fd514b92740 20 client.26237 populate_metadata read hostname 'smithi116' 2024-03-19T01:26:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.358+0000 7fd514b92740 1 --2- 172.21.15.116:0/1665395811 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559feaefedb0 0x559feaf011d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:26:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.358+0000 7fd514b92740 4 client.26237 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:26:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.358+0000 7fd514b92740 1 -- 172.21.15.116:0/1665395811 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559feade6470 con 0x559feaefedb0 2024-03-19T01:26:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.365+0000 7fd50b7fe640 1 --2- 172.21.15.116:0/1665395811 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559feaefedb0 0x559feaf011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:26:51.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.370+0000 7fd50b7fe640 1 --2- 172.21.15.116:0/1665395811 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559feaefedb0 0x559feaf011d0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9dac7b40d72980b server_cookie=663f5db1fb44dc94 in_seq=0 out_seq=0 2024-03-19T01:26:51.371 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.370+0000 7fd4eaffd640 10 client.26237 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.400+0000 7fd4eaffd640 1 -- 172.21.15.116:0/1665395811 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559feade6470 con 0x559feaefedb0 2024-03-19T01:26:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.400+0000 7fd4eaffd640 10 client.26237 handle_command_reply: tid=0 2024-03-19T01:26:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.400+0000 7fd514b92740 1 client.26237 shutdown 2024-03-19T01:26:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.402+0000 7fd514b92740 1 -- 172.21.15.116:0/1665395811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7aa70 msgr2=0x559feae912e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.402+0000 7fd514b92740 1 --2- 172.21.15.116:0/1665395811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7aa70 0x559feae912e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc000c00 tx=0x7fd4fc02bb70 comp rx=0 tx=0).stop 2024-03-19T01:26:51.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.404+0000 7fd514b92740 1 -- 172.21.15.116:0/1665395811 shutdown_connections 2024-03-19T01:26:51.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.404+0000 7fd514b92740 1 --2- 172.21.15.116:0/1665395811 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x559feaefedb0 0x559feaf011d0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.404+0000 7fd514b92740 1 --2- 172.21.15.116:0/1665395811 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x559feae7dbd0 0x559feaefca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.404+0000 7fd514b92740 1 --2- 172.21.15.116:0/1665395811 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559feae7aa70 0x559feae912e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.404 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.404+0000 7fd514b92740 1 --2- 172.21.15.116:0/1665395811 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559feae7a080 0x559feae90da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.404+0000 7fd514b92740 1 -- 172.21.15.116:0/1665395811 >> 172.21.15.116:0/1665395811 conn(0x559feae75670 msgr2=0x559feae86a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:51.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.404+0000 7fd4eaffd640 0 client.26237 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:26:51.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.404+0000 7fd514b92740 1 -- 172.21.15.116:0/1665395811 shutdown_connections 2024-03-19T01:26:51.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.405+0000 7fd514b92740 1 -- 172.21.15.116:0/1665395811 wait complete. 2024-03-19T01:26:51.405 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.405+0000 7fd514b92740 20 client.26237 trim_cache size 0 max 16384 2024-03-19T01:26:51.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:26:51.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:26:51.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:26:51.406 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:26:51.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.405+0000 7fd511ec6640 1 -- 172.21.15.116:0/3313817728 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd4f40a29a0 msgr2=0x7fd4f40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:51.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.405+0000 7fd511ec6640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd4f40a29a0 0x7fd4f40a4e60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x559feae7b2d0 tx=0x7fd504004a40 comp rx=0 tx=0).stop 2024-03-19T01:26:51.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.405+0000 7fd511ec6640 1 -- 172.21.15.116:0/3313817728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd50c101930 msgr2=0x7fd50c19c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:26:51.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.405+0000 7fd511ec6640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd50c101930 0x7fd50c19c840 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc02c0b0 tx=0x7fd4fc032430 comp rx=0 tx=0).stop 2024-03-19T01:26:51.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.405+0000 7fd511ec6640 1 -- 172.21.15.116:0/3313817728 shutdown_connections 2024-03-19T01:26:51.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.405+0000 7fd511ec6640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fd4f40a29a0 0x7fd4f40a4e60 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.405+0000 7fd511ec6640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd50c1022f0 0x7fd50c19ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.406+0000 7fd511ec6640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd50c101930 0x7fd50c19c840 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.406+0000 7fd511ec6640 1 --2- 172.21.15.116:0/3313817728 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fd50c100740 0x7fd50c069a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:26:51.407 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.406+0000 7fd511ec6640 1 -- 172.21.15.116:0/3313817728 >> 172.21.15.116:0/3313817728 conn(0x7fd50c0fbcb0 msgr2=0x7fd50c104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:26:51.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.406+0000 7fd511ec6640 1 -- 172.21.15.116:0/3313817728 shutdown_connections 2024-03-19T01:26:51.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:26:51.406+0000 7fd511ec6640 1 -- 172.21.15.116:0/3313817728 wait complete. 2024-03-19T01:26:51.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:51 smithi156 ceph-mon[66799]: pgmap v778: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.5 MiB/s wr, 591 op/s 2024-03-19T01:26:51.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 845.0 s, 1707.8 tps, lat 57.785 ms stddev 67.185 2024-03-19T01:26:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:51 smithi162 ceph-mon[67178]: pgmap v778: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.5 MiB/s wr, 591 op/s 2024-03-19T01:26:51.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:51 smithi116 ceph-mon[62701]: pgmap v778: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.5 MiB/s wr, 591 op/s 2024-03-19T01:26:53.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:53 smithi156 ceph-mon[66799]: pgmap v779: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.6 MiB/s wr, 623 op/s 2024-03-19T01:26:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:53 smithi162 ceph-mon[67178]: pgmap v779: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.6 MiB/s wr, 623 op/s 2024-03-19T01:26:53.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:53 smithi116 ceph-mon[62701]: pgmap v779: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.6 MiB/s wr, 623 op/s 2024-03-19T01:26:55.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:55 smithi156 ceph-mon[66799]: pgmap v780: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.3 MiB/s wr, 558 op/s 2024-03-19T01:26:55.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:55 smithi162 ceph-mon[67178]: pgmap v780: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.3 MiB/s wr, 558 op/s 2024-03-19T01:26:55.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:55 smithi116 ceph-mon[62701]: pgmap v780: 97 pgs: 97 active+clean; 1.6 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.3 MiB/s wr, 558 op/s 2024-03-19T01:26:56.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 850.0 s, 1404.8 tps, lat 71.783 ms stddev 86.635 2024-03-19T01:26:57.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:57 smithi156 ceph-mon[66799]: pgmap v781: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.2 MiB/s wr, 663 op/s 2024-03-19T01:26:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:57 smithi162 ceph-mon[67178]: pgmap v781: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.2 MiB/s wr, 663 op/s 2024-03-19T01:26:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:57 smithi116 ceph-mon[62701]: pgmap v781: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.2 MiB/s wr, 663 op/s 2024-03-19T01:26:59.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:26:59 smithi156 ceph-mon[66799]: pgmap v782: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 MiB/s wr, 579 op/s 2024-03-19T01:26:59.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:26:59 smithi162 ceph-mon[67178]: pgmap v782: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 MiB/s wr, 579 op/s 2024-03-19T01:26:59.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:26:59 smithi116 ceph-mon[62701]: pgmap v782: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.4 MiB/s wr, 579 op/s 2024-03-19T01:27:01.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:01 smithi156 ceph-mon[66799]: pgmap v783: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.4 MiB/s wr, 580 op/s 2024-03-19T01:27:01.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 855.0 s, 1324.0 tps, lat 74.753 ms stddev 81.401 2024-03-19T01:27:01.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:01 smithi162 ceph-mon[67178]: pgmap v783: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.4 MiB/s wr, 580 op/s 2024-03-19T01:27:01.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:01 smithi116 ceph-mon[62701]: pgmap v783: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 5.4 MiB/s wr, 580 op/s 2024-03-19T01:27:03.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:03 smithi156 ceph-mon[66799]: pgmap v784: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.1 MiB/s wr, 645 op/s 2024-03-19T01:27:03.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:03 smithi162 ceph-mon[67178]: pgmap v784: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.1 MiB/s wr, 645 op/s 2024-03-19T01:27:03.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:03 smithi116 ceph-mon[62701]: pgmap v784: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.1 MiB/s wr, 645 op/s 2024-03-19T01:27:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.555+0000 7fb55cdda640 1 -- 172.21.15.116:0/281819696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb55810b190 msgr2=0x7fb55810d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:04.558 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.555+0000 7fb55cdda640 1 --2- 172.21.15.116:0/281819696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb55810b190 0x7fb55810d600 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb54801a910 tx=0x7fb54803cc60 comp rx=0 tx=0).stop 2024-03-19T01:27:04.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.558+0000 7fb55cdda640 1 -- 172.21.15.116:0/281819696 shutdown_connections 2024-03-19T01:27:04.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.558+0000 7fb55cdda640 1 --2- 172.21.15.116:0/281819696 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb55810db40 0x7fb55810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.558+0000 7fb55cdda640 1 --2- 172.21.15.116:0/281819696 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb55810b190 0x7fb55810d600 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.558+0000 7fb55cdda640 1 --2- 172.21.15.116:0/281819696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb558108820 0x7fb55810ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.558+0000 7fb55cdda640 1 -- 172.21.15.116:0/281819696 >> 172.21.15.116:0/281819696 conn(0x7fb55806e860 msgr2=0x7fb558070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:04.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.558+0000 7fb55cdda640 1 -- 172.21.15.116:0/281819696 shutdown_connections 2024-03-19T01:27:04.560 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.558+0000 7fb55cdda640 1 -- 172.21.15.116:0/281819696 wait complete. 2024-03-19T01:27:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.559+0000 7fb55cdda640 1 Processor -- start 2024-03-19T01:27:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.559+0000 7fb55cdda640 1 -- start start 2024-03-19T01:27:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.559+0000 7fb55cdda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb558108820 0x7fb55819e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55cdda640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb55810b190 0x7fb55819e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55cdda640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb55810db40 0x7fb5581a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55cdda640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb558111af0 con 0x7fb558108820 2024-03-19T01:27:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55cdda640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb558111970 con 0x7fb55810db40 2024-03-19T01:27:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55cdda640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb558111c70 con 0x7fb55810b190 2024-03-19T01:27:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb558108820 0x7fb55819e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb558108820 0x7fb55819e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48492/0 (socket says 172.21.15.116:48492) 2024-03-19T01:27:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55659c640 1 -- 172.21.15.116:0/3909972626 learned_addr learned my addr 172.21.15.116:0/3909972626 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb555d9b640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb55810b190 0x7fb55819e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb556d9d640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb55810db40 0x7fb5581a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55659c640 1 -- 172.21.15.116:0/3909972626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb55810b190 msgr2=0x7fb55819e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55659c640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb55810b190 0x7fb55819e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55659c640 1 -- 172.21.15.116:0/3909972626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb55810db40 msgr2=0x7fb5581a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55659c640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb55810db40 0x7fb5581a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55659c640 1 -- 172.21.15.116:0/3909972626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5581a5c40 con 0x7fb558108820 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb555d9b640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb55810b190 0x7fb55819e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb55659c640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb558108820 0x7fb55819e020 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00d950 tx=0x7fb54c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.560+0000 7fb5477fe640 1 -- 172.21.15.116:0/3909972626 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c017030 con 0x7fb558108820 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.561+0000 7fb55cdda640 1 -- 172.21.15.116:0/3909972626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5581a5f30 con 0x7fb558108820 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.561+0000 7fb55cdda640 1 -- 172.21.15.116:0/3909972626 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5581a63a0 con 0x7fb558108820 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.561+0000 7fb5477fe640 1 -- 172.21.15.116:0/3909972626 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb54c004020 con 0x7fb558108820 2024-03-19T01:27:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.561+0000 7fb5477fe640 1 -- 172.21.15.116:0/3909972626 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c0047a0 con 0x7fb558108820 2024-03-19T01:27:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.564+0000 7fb5477fe640 1 -- 172.21.15.116:0/3909972626 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fb54c009a30 con 0x7fb558108820 2024-03-19T01:27:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.564+0000 7fb55f9bd740 1 Processor -- start 2024-03-19T01:27:04.567 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.564+0000 7fb55f9bd740 1 -- start start 2024-03-19T01:27:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.564+0000 7fb55f9bd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6100 0x55fccadd6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.564+0000 7fb55f9bd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccadd6af0 0x55fccaddb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb55f9bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccaddbc60 0x55fccadde0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb55f9bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fccad3abe0 con 0x55fccaddbc60 2024-03-19T01:27:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb55f9bd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fccace3960 con 0x55fccadd6100 2024-03-19T01:27:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb55f9bd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fccad36c80 con 0x55fccadd6af0 2024-03-19T01:27:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb55659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccadd6af0 0x55fccaddb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb55659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccadd6af0 0x55fccaddb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51024/0 (socket says 172.21.15.116:51024) 2024-03-19T01:27:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb55659c640 1 -- 172.21.15.116:0/3880355763 learned_addr learned my addr 172.21.15.116:0/3880355763 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb5477fe640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb5400a28f0 0x7fb5400a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb555d9b640 1 --2- 172.21.15.116:0/3880355763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccaddbc60 0x55fccadde0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.568 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb55659c640 1 -- 172.21.15.116:0/3880355763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6100 msgr2=0x55fccadd6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb55659c640 1 --2- 172.21.15.116:0/3880355763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6100 0x55fccadd6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb55659c640 1 -- 172.21.15.116:0/3880355763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccaddbc60 msgr2=0x55fccadde0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb55659c640 1 --2- 172.21.15.116:0/3880355763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccaddbc60 0x55fccadde0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb55659c640 1 -- 172.21.15.116:0/3880355763 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fccacdad20 con 0x55fccadd6af0 2024-03-19T01:27:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.565+0000 7fb5477fe640 1 -- 172.21.15.116:0/3909972626 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb54c0ce3f0 con 0x7fb558108820 2024-03-19T01:27:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.566+0000 7fb55659c640 1 --2- 172.21.15.116:0/3880355763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccadd6af0 0x55fccaddb5f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb54c097260 tx=0x7fb54c096aa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60596e8b1d6cc041 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.566+0000 7fb556d9d640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb5400a28f0 0x7fb5400a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.566+0000 7fb544ff9640 1 -- 172.21.15.116:0/3880355763 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c0d1e50 con 0x55fccadd6af0 2024-03-19T01:27:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.566+0000 7fb544ff9640 1 -- 172.21.15.116:0/3880355763 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb54c0056a0 con 0x55fccadd6af0 2024-03-19T01:27:04.569 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.566+0000 7fb544ff9640 1 -- 172.21.15.116:0/3880355763 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c09ef50 con 0x55fccadd6af0 2024-03-19T01:27:04.573 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.567+0000 7fb556d9d640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb5400a28f0 0x7fb5400a4db0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x55fccadd7350 tx=0x7fb5500094a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.567+0000 7fb55f9bd740 1 -- 172.21.15.116:0/3880355763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccadd6af0 msgr2=0x55fccaddb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.567+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/3880355763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccadd6af0 0x55fccaddb5f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb54c097260 tx=0x7fb54c096aa0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.567+0000 7fb55f9bd740 1 -- 172.21.15.116:0/3880355763 shutdown_connections 2024-03-19T01:27:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.567+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/3880355763 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccaddbc60 0x55fccadde0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.567+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/3880355763 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccadd6af0 0x55fccaddb5f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.567+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/3880355763 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6100 0x55fccadd6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.567+0000 7fb55f9bd740 1 -- 172.21.15.116:0/3880355763 >> 172.21.15.116:0/3880355763 conn(0x55fccadd1670 msgr2=0x55fccadd1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.567+0000 7fb55f9bd740 1 -- 172.21.15.116:0/3880355763 shutdown_connections 2024-03-19T01:27:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.567+0000 7fb55f9bd740 1 -- 172.21.15.116:0/3880355763 wait complete. 2024-03-19T01:27:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.567+0000 7fb55f9bd740 1 -- start start 2024-03-19T01:27:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb55f9bd740 1 Processor -- start 2024-03-19T01:27:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb55f9bd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6100 0x55fccade00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.574 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb55f9bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6af0 0x55fccade05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb55f9bd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 0x55fccade7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb55f9bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55fccaddebb0 con 0x55fccadd6af0 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb55f9bd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55fccacf6db0 con 0x55fccadd6100 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb55f9bd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55fccadded30 con 0x55fccaddbc60 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb55f9bd740 10 client.0 _ll_register_callbacks cb 0x55fccacfbe70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb55659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6100 0x55fccade00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb555d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6af0 0x55fccade05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb555d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6af0 0x55fccade05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48508/0 (socket says 172.21.15.116:48508) 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb55659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6100 0x55fccade00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:52540/0 (socket says 172.21.15.116:52540) 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb555d9b640 1 -- 172.21.15.116:0/3147577797 learned_addr learned my addr 172.21.15.116:0/3147577797 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb556d9d640 1 --2- 172.21.15.116:0/3147577797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 0x55fccade7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb555d9b640 1 -- 172.21.15.116:0/3147577797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 msgr2=0x55fccade7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb555d9b640 1 --2- 172.21.15.116:0/3147577797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 0x55fccade7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb555d9b640 1 -- 172.21.15.116:0/3147577797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6100 msgr2=0x55fccade00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:04.575 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb555d9b640 1 --2- 172.21.15.116:0/3147577797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6100 0x55fccade00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.568+0000 7fb555d9b640 1 -- 172.21.15.116:0/3147577797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fccacf5c60 con 0x55fccadd6af0 2024-03-19T01:27:04.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.569+0000 7fb555d9b640 1 --2- 172.21.15.116:0/3147577797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6af0 0x55fccade05e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb54801a910 tx=0x7fb54803da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d352a48de01ba44d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:04.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.569+0000 7fb53dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:27:04.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.569+0000 7fb53dffb640 1 -- 172.21.15.116:0/3147577797 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54803ddc0 con 0x55fccadd6af0 2024-03-19T01:27:04.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.569+0000 7fb53dffb640 1 -- 172.21.15.116:0/3147577797 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb5480035f0 con 0x55fccadd6af0 2024-03-19T01:27:04.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.569+0000 7fb55f9bd740 1 -- 172.21.15.116:0/3147577797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb54c0056a0 con 0x55fccadd6af0 2024-03-19T01:27:04.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.569+0000 7fb53dffb640 1 -- 172.21.15.116:0/3147577797 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fb5480037f0 con 0x55fccadd6af0 2024-03-19T01:27:04.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.569+0000 7fb55f9bd740 10 client.16395 fetch_fsmap learned FSMap version 49 2024-03-19T01:27:04.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.569+0000 7fb55f9bd740 1 -- 172.21.15.116:0/3147577797 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55fccaa4f210 con 0x55fccadd6af0 2024-03-19T01:27:04.576 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.570+0000 7fb53dffb640 1 -- 172.21.15.116:0/3147577797 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb54c0056a0 con 0x55fccadd6af0 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.570+0000 7fb53dffb640 1 -- 172.21.15.116:0/3147577797 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fb548045aa0 con 0x55fccadd6af0 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.570+0000 7fb55f9bd740 10 client.16395 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.570+0000 7fb55f9bd740 10 client.16395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.570+0000 7fb55f9bd740 20 client.16395 populate_metadata read hostname 'smithi116' 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.570+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/3147577797 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fccadebcf0 0x55fccadee0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.570+0000 7fb55f9bd740 4 client.16395 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.570+0000 7fb55f9bd740 1 -- 172.21.15.116:0/3147577797 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fccad48120 con 0x55fccadebcf0 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.570+0000 7fb55659c640 1 --2- 172.21.15.116:0/3147577797 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fccadebcf0 0x55fccadee0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.571+0000 7fb55659c640 1 --2- 172.21.15.116:0/3147577797 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fccadebcf0 0x55fccadee0e0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=546c96987874eec7 server_cookie=37f572f146c58b02 in_seq=0 out_seq=0 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.571+0000 7fb53dffb640 10 client.16395 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb53dffb640 1 -- 172.21.15.116:0/3147577797 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fccad48120 con 0x55fccadebcf0 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb53dffb640 10 client.16395 handle_command_reply: tid=0 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb55f9bd740 1 client.16395 shutdown 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb55f9bd740 1 -- 172.21.15.116:0/3147577797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6af0 msgr2=0x55fccade05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:04.577 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/3147577797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6af0 0x55fccade05e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fb54801a910 tx=0x7fb54803da40 comp rx=0 tx=0).stop 2024-03-19T01:27:04.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb55f9bd740 1 -- 172.21.15.116:0/3147577797 shutdown_connections 2024-03-19T01:27:04.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/3147577797 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fccadebcf0 0x55fccadee0e0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/3147577797 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 0x55fccade7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/3147577797 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6af0 0x55fccade05e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/3147577797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6100 0x55fccade00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb55f9bd740 1 -- 172.21.15.116:0/3147577797 >> 172.21.15.116:0/3147577797 conn(0x55fccadd1670 msgr2=0x55fccadd28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:04.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb53dffb640 0 client.16395 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:04.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb55f9bd740 1 -- 172.21.15.116:0/3147577797 shutdown_connections 2024-03-19T01:27:04.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb55f9bd740 1 -- 172.21.15.116:0/3147577797 wait complete. 2024-03-19T01:27:04.578 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.572+0000 7fb55f9bd740 20 client.16395 trim_cache size 0 max 16384 2024-03-19T01:27:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55f9bd740 1 Processor -- start 2024-03-19T01:27:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55f9bd740 1 -- start start 2024-03-19T01:27:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55f9bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 0x55fccadd4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55f9bd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6af0 0x55fccade75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55f9bd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 0x55fccadf29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55f9bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb5581af3a0 con 0x55fccadd6100 2024-03-19T01:27:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55f9bd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb5581b0ee0 con 0x55fccadd6af0 2024-03-19T01:27:04.604 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55f9bd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb5581afa70 con 0x55fccaddbc60 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 0x55fccadd4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 0x55fccadd4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48524/0 (socket says 172.21.15.116:48524) 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55659c640 1 -- 172.21.15.116:0/2686433659 learned_addr learned my addr 172.21.15.116:0/2686433659 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb556d9d640 1 --2- 172.21.15.116:0/2686433659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 0x55fccadf29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb555d9b640 1 --2- 172.21.15.116:0/2686433659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6af0 0x55fccade75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55659c640 1 -- 172.21.15.116:0/2686433659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 msgr2=0x55fccadf29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55659c640 1 --2- 172.21.15.116:0/2686433659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 0x55fccadf29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55659c640 1 -- 172.21.15.116:0/2686433659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6af0 msgr2=0x55fccade75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55659c640 1 --2- 172.21.15.116:0/2686433659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6af0 0x55fccade75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55659c640 1 -- 172.21.15.116:0/2686433659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fccab1b620 con 0x55fccadd6100 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.599+0000 7fb55659c640 1 --2- 172.21.15.116:0/2686433659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 0x55fccadd4fe0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb54c09ab00 tx=0x7fb54c0a1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d308008618a0d9c server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.600+0000 7fb53ffff640 1 -- 172.21.15.116:0/2686433659 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c0d3e60 con 0x55fccadd6100 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.600+0000 7fb53ffff640 1 -- 172.21.15.116:0/2686433659 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb54c0d3070 con 0x55fccadd6100 2024-03-19T01:27:04.605 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.600+0000 7fb53ffff640 1 -- 172.21.15.116:0/2686433659 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c0041d0 con 0x55fccadd6100 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.600+0000 7fb55f9bd740 1 -- 172.21.15.116:0/2686433659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 msgr2=0x55fccadd4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.600+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/2686433659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 0x55fccadd4fe0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fb54c09ab00 tx=0x7fb54c0a1040 comp rx=0 tx=0).stop 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.600+0000 7fb55f9bd740 1 -- 172.21.15.116:0/2686433659 shutdown_connections 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.600+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/2686433659 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 0x55fccadf29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.600+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/2686433659 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6af0 0x55fccade75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.600+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/2686433659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 0x55fccadd4fe0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.600+0000 7fb55f9bd740 1 -- 172.21.15.116:0/2686433659 >> 172.21.15.116:0/2686433659 conn(0x55fccadd1670 msgr2=0x55fccadd48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.600+0000 7fb55f9bd740 1 -- 172.21.15.116:0/2686433659 shutdown_connections 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55f9bd740 1 -- 172.21.15.116:0/2686433659 wait complete. 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55f9bd740 1 -- start start 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55f9bd740 1 Processor -- start 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55f9bd740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 0x55fccade4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55f9bd740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6af0 0x55fccade5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.606 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55f9bd740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 0x55fccade6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55f9bd740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb5581b0810 con 0x55fccadd6100 2024-03-19T01:27:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55f9bd740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb5581b0140 con 0x55fccadd6af0 2024-03-19T01:27:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55f9bd740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb5581ad380 con 0x55fccaddbc60 2024-03-19T01:27:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55f9bd740 10 client.0 _ll_register_callbacks cb 0x7fb548004500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:27:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 0x55fccade4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 0x55fccade4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48526/0 (socket says 172.21.15.116:48526) 2024-03-19T01:27:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55659c640 1 -- 172.21.15.116:0/1950395777 learned_addr learned my addr 172.21.15.116:0/1950395777 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb555d9b640 1 --2- 172.21.15.116:0/1950395777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6af0 0x55fccade5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb556d9d640 1 --2- 172.21.15.116:0/1950395777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 0x55fccade6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55659c640 1 -- 172.21.15.116:0/1950395777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 msgr2=0x55fccade6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55659c640 1 --2- 172.21.15.116:0/1950395777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 0x55fccade6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55659c640 1 -- 172.21.15.116:0/1950395777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6af0 msgr2=0x55fccade5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:04.607 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55659c640 1 --2- 172.21.15.116:0/1950395777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6af0 0x55fccade5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.601+0000 7fb55659c640 1 -- 172.21.15.116:0/1950395777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fccade6830 con 0x55fccadd6100 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb55659c640 1 --2- 172.21.15.116:0/1950395777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 0x55fccade4cf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0a2fe0 tx=0x7fb54c009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22cc513e9ca31e8f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb53dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb53dffb640 1 -- 172.21.15.116:0/1950395777 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb54c0b7020 con 0x55fccadd6100 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb53dffb640 1 -- 172.21.15.116:0/1950395777 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fb54c0041d0 con 0x55fccadd6100 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb55f9bd740 1 -- 172.21.15.116:0/1950395777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb54c0d3070 con 0x55fccadd6100 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb53dffb640 1 -- 172.21.15.116:0/1950395777 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fb54c09a760 con 0x55fccadd6100 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb55f9bd740 10 client.16401 fetch_fsmap learned FSMap version 49 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb55f9bd740 1 -- 172.21.15.116:0/1950395777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55fccade6830 con 0x55fccadd6100 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb53dffb640 1 -- 172.21.15.116:0/1950395777 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb54c0d3070 con 0x55fccadd6100 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb53dffb640 1 -- 172.21.15.116:0/1950395777 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fb54c04c020 con 0x55fccadd6100 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb55f9bd740 10 client.16401 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb55f9bd740 10 client.16401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb55f9bd740 20 client.16401 populate_metadata read hostname 'smithi116' 2024-03-19T01:27:04.608 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/1950395777 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fccae52110 0x55fccae54530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb55f9bd740 4 client.16401 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:27:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.602+0000 7fb55f9bd740 1 -- 172.21.15.116:0/1950395777 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fccad42470 con 0x55fccae52110 2024-03-19T01:27:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.603+0000 7fb555d9b640 1 --2- 172.21.15.116:0/1950395777 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fccae52110 0x55fccae54530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.603+0000 7fb53dffb640 10 client.16401 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.603+0000 7fb555d9b640 1 --2- 172.21.15.116:0/1950395777 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fccae52110 0x55fccae54530 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a375f0651a1de10e server_cookie=5deb403d07e12f91 in_seq=0 out_seq=0 2024-03-19T01:27:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.610+0000 7fb53dffb640 1 -- 172.21.15.116:0/1950395777 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fccad42470 con 0x55fccae52110 2024-03-19T01:27:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.610+0000 7fb53dffb640 10 client.16401 handle_command_reply: tid=0 2024-03-19T01:27:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.610+0000 7fb55f9bd740 1 client.16401 shutdown 2024-03-19T01:27:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55f9bd740 1 -- 172.21.15.116:0/1950395777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 msgr2=0x55fccade4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/1950395777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 0x55fccade4cf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0a2fe0 tx=0x7fb54c009740 comp rx=0 tx=0).stop 2024-03-19T01:27:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55f9bd740 1 -- 172.21.15.116:0/1950395777 shutdown_connections 2024-03-19T01:27:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/1950395777 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55fccae52110 0x55fccae54530 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb53dffb640 0 client.16401 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/1950395777 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55fccaddbc60 0x55fccade6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/1950395777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55fccadd6af0 0x55fccade5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55f9bd740 1 --2- 172.21.15.116:0/1950395777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55fccadd6100 0x55fccade4cf0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55f9bd740 1 -- 172.21.15.116:0/1950395777 >> 172.21.15.116:0/1950395777 conn(0x55fccadd1670 msgr2=0x55fccadd5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55f9bd740 1 -- 172.21.15.116:0/1950395777 shutdown_connections 2024-03-19T01:27:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55f9bd740 1 -- 172.21.15.116:0/1950395777 wait complete. 2024-03-19T01:27:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55f9bd740 20 client.16401 trim_cache size 0 max 16384 2024-03-19T01:27:04.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:27:04.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:27:04.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:27:04.614 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:27:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55cdda640 1 -- 172.21.15.116:0/3909972626 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb5400a28f0 msgr2=0x7fb5400a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55cdda640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb5400a28f0 0x7fb5400a4db0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x55fccadd7350 tx=0x7fb5500094a0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55cdda640 1 -- 172.21.15.116:0/3909972626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb558108820 msgr2=0x7fb55819e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55cdda640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb558108820 0x7fb55819e020 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fb54c00d950 tx=0x7fb54c00de10 comp rx=0 tx=0).stop 2024-03-19T01:27:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55cdda640 1 -- 172.21.15.116:0/3909972626 shutdown_connections 2024-03-19T01:27:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55cdda640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb5400a28f0 0x7fb5400a4db0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55cdda640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb55810db40 0x7fb5581a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.611+0000 7fb55cdda640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb55810b190 0x7fb55819e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.612+0000 7fb55cdda640 1 --2- 172.21.15.116:0/3909972626 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb558108820 0x7fb55819e020 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.612+0000 7fb55cdda640 1 -- 172.21.15.116:0/3909972626 >> 172.21.15.116:0/3909972626 conn(0x7fb55806e860 msgr2=0x7fb55810b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.612+0000 7fb55cdda640 1 -- 172.21.15.116:0/3909972626 shutdown_connections 2024-03-19T01:27:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:04.612+0000 7fb55cdda640 1 -- 172.21.15.116:0/3909972626 wait complete. 2024-03-19T01:27:05.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:05 smithi162 ceph-mon[67178]: pgmap v785: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.0 MiB/s wr, 535 op/s 2024-03-19T01:27:05.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:05 smithi116 ceph-mon[62701]: pgmap v785: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.0 MiB/s wr, 535 op/s 2024-03-19T01:27:05.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:05 smithi156 ceph-mon[66799]: pgmap v785: 97 pgs: 97 active+clean; 1.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.0 MiB/s wr, 535 op/s 2024-03-19T01:27:06.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 860.0 s, 1481.0 tps, lat 67.611 ms stddev 75.490 2024-03-19T01:27:07.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:07 smithi162 ceph-mon[67178]: pgmap v786: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.2 MiB/s wr, 655 op/s 2024-03-19T01:27:07.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:07 smithi116 ceph-mon[62701]: pgmap v786: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.2 MiB/s wr, 655 op/s 2024-03-19T01:27:07.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:07 smithi156 ceph-mon[66799]: pgmap v786: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.2 MiB/s wr, 655 op/s 2024-03-19T01:27:09.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:09 smithi162 ceph-mon[67178]: pgmap v787: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.0 MiB/s wr, 598 op/s 2024-03-19T01:27:09.659 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:09 smithi116 ceph-mon[62701]: pgmap v787: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.0 MiB/s wr, 598 op/s 2024-03-19T01:27:09.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:09 smithi156 ceph-mon[66799]: pgmap v787: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 6.0 MiB/s wr, 598 op/s 2024-03-19T01:27:11.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 865.0 s, 1433.4 tps, lat 70.172 ms stddev 81.413 2024-03-19T01:27:11.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:11 smithi162 ceph-mon[67178]: pgmap v788: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 599 op/s 2024-03-19T01:27:11.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:11 smithi116 ceph-mon[62701]: pgmap v788: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 599 op/s 2024-03-19T01:27:11.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:11 smithi156 ceph-mon[66799]: pgmap v788: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 599 op/s 2024-03-19T01:27:13.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:13 smithi162 ceph-mon[67178]: pgmap v789: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.0 MiB/s wr, 691 op/s 2024-03-19T01:27:13.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:13 smithi116 ceph-mon[62701]: pgmap v789: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.0 MiB/s wr, 691 op/s 2024-03-19T01:27:13.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:13 smithi156 ceph-mon[66799]: pgmap v789: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 7.0 MiB/s wr, 691 op/s 2024-03-19T01:27:15.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:15 smithi162 ceph-mon[67178]: pgmap v790: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.7 MiB/s wr, 574 op/s 2024-03-19T01:27:15.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:15 smithi116 ceph-mon[62701]: pgmap v790: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.7 MiB/s wr, 574 op/s 2024-03-19T01:27:15.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:15 smithi156 ceph-mon[66799]: pgmap v790: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 5.7 MiB/s wr, 574 op/s 2024-03-19T01:27:16.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 870.0 s, 1129.6 tps, lat 88.901 ms stddev 95.883 2024-03-19T01:27:17.525 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:17 smithi156 ceph-mon[66799]: pgmap v791: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.2 MiB/s wr, 724 op/s 2024-03-19T01:27:17.526 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:17 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:27:17.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:17 smithi162 ceph-mon[67178]: pgmap v791: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.2 MiB/s wr, 724 op/s 2024-03-19T01:27:17.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:17 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:27:17.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:17 smithi116 ceph-mon[62701]: pgmap v791: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 8.2 MiB/s wr, 724 op/s 2024-03-19T01:27:17.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:17 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:27:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.742+0000 7f7560661640 1 -- 172.21.15.116:0/2316827345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7558103420 msgr2=0x7f7558109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:17.742 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.742+0000 7f7560661640 1 --2- 172.21.15.116:0/2316827345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7558103420 0x7f7558109cb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7550009860 tx=0x7f755002bb70 comp rx=0 tx=0).stop 2024-03-19T01:27:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.744+0000 7f7560661640 1 -- 172.21.15.116:0/2316827345 shutdown_connections 2024-03-19T01:27:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.744+0000 7f7560661640 1 --2- 172.21.15.116:0/2316827345 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7558103420 0x7f7558109cb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.744+0000 7f7560661640 1 --2- 172.21.15.116:0/2316827345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7558102a60 0x7f7558102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.744+0000 7f7560661640 1 --2- 172.21.15.116:0/2316827345 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7558101870 0x7f7558101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.744+0000 7f7560661640 1 -- 172.21.15.116:0/2316827345 >> 172.21.15.116:0/2316827345 conn(0x7f75580fcde0 msgr2=0x7f75580ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:17.744 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.744+0000 7f7560661640 1 -- 172.21.15.116:0/2316827345 shutdown_connections 2024-03-19T01:27:17.745 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.744+0000 7f7560661640 1 -- 172.21.15.116:0/2316827345 wait complete. 2024-03-19T01:27:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.744+0000 7f7560661640 1 Processor -- start 2024-03-19T01:27:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.744+0000 7f7560661640 1 -- start start 2024-03-19T01:27:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f7560661640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7558101870 0x7f7558197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f7560661640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7558102a60 0x7f7558197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f7560661640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7558103420 0x7f755819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f755e3fd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7558101870 0x7f7558197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f7560661640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f755810d4f0 con 0x7f7558101870 2024-03-19T01:27:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f7560661640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f755810d370 con 0x7f7558103420 2024-03-19T01:27:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f7560661640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f755810d670 con 0x7f7558102a60 2024-03-19T01:27:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f755e3fd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7558101870 0x7f7558197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57714/0 (socket says 172.21.15.116:57714) 2024-03-19T01:27:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f755e3fd640 1 -- 172.21.15.116:0/2502059246 learned_addr learned my addr 172.21.15.116:0/2502059246 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f755dbfc640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7558102a60 0x7f7558197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f755ebfe640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7558103420 0x7f755819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f755e3fd640 1 -- 172.21.15.116:0/2502059246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7558102a60 msgr2=0x7f7558197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f755e3fd640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7558102a60 0x7f7558197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f755e3fd640 1 -- 172.21.15.116:0/2502059246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7558103420 msgr2=0x7f755819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f755e3fd640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7558103420 0x7f755819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f755e3fd640 1 -- 172.21.15.116:0/2502059246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f755819f690 con 0x7f7558101870 2024-03-19T01:27:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.745+0000 7f755e3fd640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7558101870 0x7f7558197a70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f754800eb70 tx=0x7f754800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.746+0000 7f75577fe640 1 -- 172.21.15.116:0/2502059246 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7548003b70 con 0x7f7558101870 2024-03-19T01:27:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.746+0000 7f7560661640 1 -- 172.21.15.116:0/2502059246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f755819f980 con 0x7f7558101870 2024-03-19T01:27:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.746+0000 7f7560661640 1 -- 172.21.15.116:0/2502059246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f755819fed0 con 0x7f7558101870 2024-03-19T01:27:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.746+0000 7f75577fe640 1 -- 172.21.15.116:0/2502059246 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f754800ee70 con 0x7f7558101870 2024-03-19T01:27:17.748 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.746+0000 7f75577fe640 1 -- 172.21.15.116:0/2502059246 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75480107a0 con 0x7f7558101870 2024-03-19T01:27:17.749 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.748+0000 7f75577fe640 1 -- 172.21.15.116:0/2502059246 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7548004000 con 0x7f7558101870 2024-03-19T01:27:17.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.749+0000 7f75577fe640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f75400a28a0 0x7f75400a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.749+0000 7f7563252740 1 Processor -- start 2024-03-19T01:27:17.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.749+0000 7f75577fe640 1 -- 172.21.15.116:0/2502059246 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f75480cf010 con 0x7f7558101870 2024-03-19T01:27:17.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.749+0000 7f755dbfc640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f75400a28a0 0x7f75400a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.749+0000 7f7563252740 1 -- start start 2024-03-19T01:27:17.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.749+0000 7f7563252740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d638100 0x55f02d638520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.751 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f7563252740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638af0 0x55f02d63d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f7563252740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d63dc60 0x55f02d6400f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f7563252740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f02d59cbe0 con 0x55f02d63dc60 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f7563252740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f02d545960 con 0x55f02d638af0 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f7563252740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f02d598c80 con 0x55f02d638100 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f755ebfe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d63dc60 0x55f02d6400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f755ebfe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d63dc60 0x55f02d6400f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57724/0 (socket says 172.21.15.116:57724) 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f755ebfe640 1 -- 172.21.15.116:0/3996917963 learned_addr learned my addr 172.21.15.116:0/3996917963 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f755dbfc640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f75400a28a0 0x7f75400a4d60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7558198f80 tx=0x7f754c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f755dbfc640 1 --2- 172.21.15.116:0/3996917963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638af0 0x55f02d63d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f755e3fd640 1 --2- 172.21.15.116:0/3996917963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d638100 0x55f02d638520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f755ebfe640 1 -- 172.21.15.116:0/3996917963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d638100 msgr2=0x55f02d638520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f755ebfe640 1 --2- 172.21.15.116:0/3996917963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d638100 0x55f02d638520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f755ebfe640 1 -- 172.21.15.116:0/3996917963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638af0 msgr2=0x55f02d63d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:17.752 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f755ebfe640 1 --2- 172.21.15.116:0/3996917963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638af0 0x55f02d63d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f755ebfe640 1 -- 172.21.15.116:0/3996917963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f02d53cd20 con 0x55f02d63dc60 2024-03-19T01:27:17.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f755dbfc640 1 --2- 172.21.15.116:0/3996917963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638af0 0x55f02d63d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:27:17.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f755ebfe640 1 --2- 172.21.15.116:0/3996917963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d63dc60 0x55f02d6400f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7550034b10 tx=0x7f7550005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53aa296e42a1254 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:17.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.750+0000 7f7554bfe640 1 -- 172.21.15.116:0/3996917963 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7550043020 con 0x55f02d63dc60 2024-03-19T01:27:17.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.751+0000 7f7554bfe640 1 -- 172.21.15.116:0/3996917963 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7550034db0 con 0x55f02d63dc60 2024-03-19T01:27:17.753 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.751+0000 7f7554bfe640 1 -- 172.21.15.116:0/3996917963 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7550039040 con 0x55f02d63dc60 2024-03-19T01:27:17.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.751+0000 7f7563252740 1 -- 172.21.15.116:0/3996917963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d63dc60 msgr2=0x55f02d6400f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:17.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.751+0000 7f7563252740 1 --2- 172.21.15.116:0/3996917963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d63dc60 0x55f02d6400f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f7550034b10 tx=0x7f7550005b20 comp rx=0 tx=0).stop 2024-03-19T01:27:17.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.751+0000 7f7563252740 1 -- 172.21.15.116:0/3996917963 shutdown_connections 2024-03-19T01:27:17.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.751+0000 7f7563252740 1 --2- 172.21.15.116:0/3996917963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d63dc60 0x55f02d6400f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.751+0000 7f7563252740 1 --2- 172.21.15.116:0/3996917963 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638af0 0x55f02d63d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.751+0000 7f7563252740 1 --2- 172.21.15.116:0/3996917963 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d638100 0x55f02d638520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.757 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.751+0000 7f7563252740 1 -- 172.21.15.116:0/3996917963 >> 172.21.15.116:0/3996917963 conn(0x55f02d633670 msgr2=0x55f02d633a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.751+0000 7f7563252740 1 -- 172.21.15.116:0/3996917963 shutdown_connections 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.751+0000 7f7563252740 1 -- 172.21.15.116:0/3996917963 wait complete. 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f7563252740 1 -- start start 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f7563252740 1 Processor -- start 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f7563252740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638100 0x55f02d642040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f7563252740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d638af0 0x55f02d642580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f7563252740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d63dc60 0x55f02d6495c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f7563252740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55f02d640bb0 con 0x55f02d638100 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f7563252740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55f02d558db0 con 0x55f02d63dc60 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f7563252740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55f02d640d30 con 0x55f02d638af0 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f755dbfc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d638af0 0x55f02d642580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f755dbfc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d638af0 0x55f02d642580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:48198/0 (socket says 172.21.15.116:48198) 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f755e3fd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638100 0x55f02d642040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.758 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f755dbfc640 1 -- 172.21.15.116:0/1998219878 learned_addr learned my addr 172.21.15.116:0/1998219878 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f755ebfe640 1 --2- 172.21.15.116:0/1998219878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d63dc60 0x55f02d6495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f7563252740 10 client.0 _ll_register_callbacks cb 0x55f02d55de70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f755dbfc640 1 -- 172.21.15.116:0/1998219878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d63dc60 msgr2=0x55f02d6495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f755dbfc640 1 --2- 172.21.15.116:0/1998219878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d63dc60 0x55f02d6495c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f755dbfc640 1 -- 172.21.15.116:0/1998219878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638100 msgr2=0x55f02d642040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.752+0000 7f755dbfc640 1 --2- 172.21.15.116:0/1998219878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638100 0x55f02d642040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.753+0000 7f755dbfc640 1 -- 172.21.15.116:0/1998219878 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f02d557c60 con 0x55f02d638af0 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.753+0000 7f755dbfc640 1 --2- 172.21.15.116:0/1998219878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d638af0 0x55f02d642580 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f754c01b990 tx=0x7f754c013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2910d1f87a4f88c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.753+0000 7f753dffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.753+0000 7f753dffb640 1 -- 172.21.15.116:0/1998219878 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754c02c060 con 0x55f02d638af0 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.753+0000 7f753dffb640 1 -- 172.21.15.116:0/1998219878 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f754c00ed40 con 0x55f02d638af0 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.753+0000 7f7563252740 1 -- 172.21.15.116:0/1998219878 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7550034db0 con 0x55f02d638af0 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.753+0000 7f753dffb640 1 -- 172.21.15.116:0/1998219878 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f754c009e10 con 0x55f02d638af0 2024-03-19T01:27:17.759 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.753+0000 7f7563252740 10 client.26254 fetch_fsmap learned FSMap version 49 2024-03-19T01:27:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.753+0000 7f7563252740 1 -- 172.21.15.116:0/1998219878 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55f02d2b1210 con 0x55f02d638af0 2024-03-19T01:27:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.754+0000 7f753dffb640 1 -- 172.21.15.116:0/1998219878 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f754c0328e0 con 0x55f02d638af0 2024-03-19T01:27:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.754+0000 7f753dffb640 1 -- 172.21.15.116:0/1998219878 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f754c026070 con 0x55f02d638af0 2024-03-19T01:27:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.754+0000 7f7563252740 10 client.26254 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:27:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.754+0000 7f7563252740 10 client.26254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:27:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.754+0000 7f7563252740 20 client.26254 populate_metadata read hostname 'smithi116' 2024-03-19T01:27:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.754+0000 7f7563252740 1 --2- 172.21.15.116:0/1998219878 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f02d64df00 0x55f02d650390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.754+0000 7f7563252740 4 client.26254 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:27:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.754+0000 7f7563252740 1 -- 172.21.15.116:0/1998219878 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7550034db0 con 0x55f02d64df00 2024-03-19T01:27:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.754+0000 7f755e3fd640 1 --2- 172.21.15.116:0/1998219878 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f02d64df00 0x55f02d650390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.760 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.754+0000 7f755e3fd640 1 --2- 172.21.15.116:0/1998219878 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f02d64df00 0x55f02d650390 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=582021eeeaa87d14 server_cookie=ff93feb81c23e321 in_seq=0 out_seq=0 2024-03-19T01:27:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.754+0000 7f753dffb640 10 client.26254 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.755+0000 7f753dffb640 1 -- 172.21.15.116:0/1998219878 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f7550034db0 con 0x55f02d64df00 2024-03-19T01:27:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.755+0000 7f753dffb640 10 client.26254 handle_command_reply: tid=0 2024-03-19T01:27:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.755+0000 7f7563252740 1 client.26254 shutdown 2024-03-19T01:27:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.756+0000 7f7563252740 1 -- 172.21.15.116:0/1998219878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d638af0 msgr2=0x55f02d642580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.756+0000 7f7563252740 1 --2- 172.21.15.116:0/1998219878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d638af0 0x55f02d642580 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f754c01b990 tx=0x7f754c013040 comp rx=0 tx=0).stop 2024-03-19T01:27:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.756+0000 7f7563252740 1 -- 172.21.15.116:0/1998219878 shutdown_connections 2024-03-19T01:27:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.756+0000 7f7563252740 1 --2- 172.21.15.116:0/1998219878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d63dc60 0x55f02d6495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.756+0000 7f7563252740 1 --2- 172.21.15.116:0/1998219878 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f02d64df00 0x55f02d650390 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.756+0000 7f753dffb640 0 client.26254 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.756+0000 7f7563252740 1 --2- 172.21.15.116:0/1998219878 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d638af0 0x55f02d642580 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.756+0000 7f7563252740 1 --2- 172.21.15.116:0/1998219878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638100 0x55f02d642040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.761 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.756+0000 7f7563252740 1 -- 172.21.15.116:0/1998219878 >> 172.21.15.116:0/1998219878 conn(0x55f02d633670 msgr2=0x55f02d6348c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:17.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.756+0000 7f7563252740 1 -- 172.21.15.116:0/1998219878 shutdown_connections 2024-03-19T01:27:17.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.756+0000 7f7563252740 1 -- 172.21.15.116:0/1998219878 wait complete. 2024-03-19T01:27:17.762 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.756+0000 7f7563252740 20 client.26254 trim_cache size 0 max 16384 2024-03-19T01:27:17.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.777+0000 7f7563252740 1 Processor -- start 2024-03-19T01:27:17.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.777+0000 7f7563252740 1 -- start start 2024-03-19T01:27:17.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.777+0000 7f7563252740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638100 0x55f02d633a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.777+0000 7f7563252740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638af0 0x55f02d649580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.777+0000 7f7563252740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d63dc60 0x55f02d632220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.777+0000 7f7563252740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f75581a8c60 con 0x55f02d638100 2024-03-19T01:27:17.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.777+0000 7f7563252740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f75581aa7a0 con 0x55f02d638af0 2024-03-19T01:27:17.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.777+0000 7f7563252740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f75581a9330 con 0x55f02d63dc60 2024-03-19T01:27:17.779 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f755e3fd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638100 0x55f02d633a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f755e3fd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638100 0x55f02d633a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57750/0 (socket says 172.21.15.116:57750) 2024-03-19T01:27:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f755e3fd640 1 -- 172.21.15.116:0/1172182710 learned_addr learned my addr 172.21.15.116:0/1172182710 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f755dbfc640 1 --2- 172.21.15.116:0/1172182710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638af0 0x55f02d649580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f755ebfe640 1 --2- 172.21.15.116:0/1172182710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d63dc60 0x55f02d632220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f755e3fd640 1 -- 172.21.15.116:0/1172182710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d63dc60 msgr2=0x55f02d632220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f755e3fd640 1 --2- 172.21.15.116:0/1172182710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d63dc60 0x55f02d632220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f755e3fd640 1 -- 172.21.15.116:0/1172182710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638af0 msgr2=0x55f02d649580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f755e3fd640 1 --2- 172.21.15.116:0/1172182710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638af0 0x55f02d649580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f755e3fd640 1 -- 172.21.15.116:0/1172182710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f02d37d620 con 0x55f02d638100 2024-03-19T01:27:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f755e3fd640 1 --2- 172.21.15.116:0/1172182710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638100 0x55f02d633a50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f754809b9c0 tx=0x7f75480cca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f4c52841a720ac server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f753ffff640 1 -- 172.21.15.116:0/1172182710 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75480ba300 con 0x55f02d638100 2024-03-19T01:27:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f753ffff640 1 -- 172.21.15.116:0/1172182710 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75480a41d0 con 0x55f02d638100 2024-03-19T01:27:17.780 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.778+0000 7f753ffff640 1 -- 172.21.15.116:0/1172182710 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75480042f0 con 0x55f02d638100 2024-03-19T01:27:17.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.779+0000 7f7563252740 1 -- 172.21.15.116:0/1172182710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638100 msgr2=0x55f02d633a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:17.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.779+0000 7f7563252740 1 --2- 172.21.15.116:0/1172182710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638100 0x55f02d633a50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f754809b9c0 tx=0x7f75480cca60 comp rx=0 tx=0).stop 2024-03-19T01:27:17.784 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 -- 172.21.15.116:0/1172182710 shutdown_connections 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 --2- 172.21.15.116:0/1172182710 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d63dc60 0x55f02d632220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 --2- 172.21.15.116:0/1172182710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638af0 0x55f02d649580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 --2- 172.21.15.116:0/1172182710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638100 0x55f02d633a50 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 -- 172.21.15.116:0/1172182710 >> 172.21.15.116:0/1172182710 conn(0x55f02d633670 msgr2=0x55f02d62e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 -- 172.21.15.116:0/1172182710 shutdown_connections 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 -- 172.21.15.116:0/1172182710 wait complete. 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 -- start start 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 Processor -- start 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638100 0x55f02d646bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638af0 0x55f02d647100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d63dc60 0x55f02d6480c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f75581aa0d0 con 0x55f02d638af0 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f75581a9a00 con 0x55f02d638100 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.780+0000 7f7563252740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f75581a6c40 con 0x55f02d63dc60 2024-03-19T01:27:17.785 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f7563252740 10 client.0 _ll_register_callbacks cb 0x7f754c009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f755dbfc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638af0 0x55f02d647100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f755e3fd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638100 0x55f02d646bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f755dbfc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638af0 0x55f02d647100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57758/0 (socket says 172.21.15.116:57758) 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f755dbfc640 1 -- 172.21.15.116:0/1482340503 learned_addr learned my addr 172.21.15.116:0/1482340503 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f755ebfe640 1 --2- 172.21.15.116:0/1482340503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d63dc60 0x55f02d6480c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f755dbfc640 1 -- 172.21.15.116:0/1482340503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d63dc60 msgr2=0x55f02d6480c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f755dbfc640 1 --2- 172.21.15.116:0/1482340503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d63dc60 0x55f02d6480c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f755dbfc640 1 -- 172.21.15.116:0/1482340503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638100 msgr2=0x55f02d646bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f755dbfc640 1 --2- 172.21.15.116:0/1482340503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638100 0x55f02d646bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f755dbfc640 1 -- 172.21.15.116:0/1482340503 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f02d6486b0 con 0x55f02d638af0 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f755ebfe640 1 --2- 172.21.15.116:0/1482340503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d63dc60 0x55f02d6480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f755dbfc640 1 --2- 172.21.15.116:0/1482340503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638af0 0x55f02d647100 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f754c013010 tx=0x7f754c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ad94f5b9684c84 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.781+0000 7f753dffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:27:17.786 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.782+0000 7f753dffb640 1 -- 172.21.15.116:0/1482340503 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754c01b7a0 con 0x55f02d638af0 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.782+0000 7f753dffb640 1 -- 172.21.15.116:0/1482340503 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f754c005040 con 0x55f02d638af0 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.782+0000 7f7563252740 1 -- 172.21.15.116:0/1482340503 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75480a41d0 con 0x55f02d638af0 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.782+0000 7f753dffb640 1 -- 172.21.15.116:0/1482340503 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f754c027e50 con 0x55f02d638af0 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.782+0000 7f7563252740 10 client.16419 fetch_fsmap learned FSMap version 49 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.782+0000 7f7563252740 1 -- 172.21.15.116:0/1482340503 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55f02d6b42d0 con 0x55f02d638af0 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.783+0000 7f753dffb640 1 -- 172.21.15.116:0/1482340503 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75480a41d0 con 0x55f02d638af0 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.783+0000 7f753dffb640 1 -- 172.21.15.116:0/1482340503 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f754c039020 con 0x55f02d638af0 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.783+0000 7f7563252740 10 client.16419 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.783+0000 7f7563252740 10 client.16419 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.783+0000 7f7563252740 20 client.16419 populate_metadata read hostname 'smithi116' 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.783+0000 7f7563252740 1 --2- 172.21.15.116:0/1482340503 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f02d64df00 0x55f02d6b6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.783+0000 7f7563252740 4 client.16419 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.783+0000 7f7563252740 1 -- 172.21.15.116:0/1482340503 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f02d5aa120 con 0x55f02d64df00 2024-03-19T01:27:17.787 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.783+0000 7f755e3fd640 1 --2- 172.21.15.116:0/1482340503 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f02d64df00 0x55f02d6b6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.784+0000 7f755e3fd640 1 --2- 172.21.15.116:0/1482340503 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f02d64df00 0x55f02d6b6bb0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7df4c1a0a80ab06 server_cookie=6e8a54d13d65310b in_seq=0 out_seq=0 2024-03-19T01:27:17.788 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.784+0000 7f753dffb640 10 client.16419 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.793+0000 7f753dffb640 1 -- 172.21.15.116:0/1482340503 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f02d5aa120 con 0x55f02d64df00 2024-03-19T01:27:17.794 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.793+0000 7f753dffb640 10 client.16419 handle_command_reply: tid=0 2024-03-19T01:27:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.793+0000 7f7563252740 1 client.16419 shutdown 2024-03-19T01:27:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.794+0000 7f7563252740 1 -- 172.21.15.116:0/1482340503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638af0 msgr2=0x55f02d647100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.794+0000 7f7563252740 1 --2- 172.21.15.116:0/1482340503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638af0 0x55f02d647100 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f754c013010 tx=0x7f754c00edf0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.794+0000 7f7563252740 1 -- 172.21.15.116:0/1482340503 shutdown_connections 2024-03-19T01:27:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.794+0000 7f7563252740 1 --2- 172.21.15.116:0/1482340503 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55f02d64df00 0x55f02d6b6bb0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.794+0000 7f7563252740 1 --2- 172.21.15.116:0/1482340503 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55f02d63dc60 0x55f02d6480c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.794+0000 7f7563252740 1 --2- 172.21.15.116:0/1482340503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55f02d638af0 0x55f02d647100 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.794+0000 7f7563252740 1 --2- 172.21.15.116:0/1482340503 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55f02d638100 0x55f02d646bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.794+0000 7f753dffb640 0 client.16419 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.794+0000 7f7563252740 1 -- 172.21.15.116:0/1482340503 >> 172.21.15.116:0/1482340503 conn(0x55f02d633670 msgr2=0x55f02d633a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.794+0000 7f7563252740 1 -- 172.21.15.116:0/1482340503 shutdown_connections 2024-03-19T01:27:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.794+0000 7f7563252740 1 -- 172.21.15.116:0/1482340503 wait complete. 2024-03-19T01:27:17.795 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.794+0000 7f7563252740 20 client.16419 trim_cache size 0 max 16384 2024-03-19T01:27:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:27:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:27:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:27:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:27:17.796 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.795+0000 7f7560661640 1 -- 172.21.15.116:0/2502059246 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f75400a28a0 msgr2=0x7f75400a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.795+0000 7f7560661640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f75400a28a0 0x7f75400a4d60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7558198f80 tx=0x7f754c009260 comp rx=0 tx=0).stop 2024-03-19T01:27:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.795+0000 7f7560661640 1 -- 172.21.15.116:0/2502059246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7558101870 msgr2=0x7f7558197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.795+0000 7f7560661640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7558101870 0x7f7558197a70 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f754800eb70 tx=0x7f754800c6a0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.795+0000 7f7560661640 1 -- 172.21.15.116:0/2502059246 shutdown_connections 2024-03-19T01:27:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.795+0000 7f7560661640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f75400a28a0 0x7f75400a4d60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.795+0000 7f7560661640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7558103420 0x7f755819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.795+0000 7f7560661640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7558102a60 0x7f7558197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.795+0000 7f7560661640 1 --2- 172.21.15.116:0/2502059246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7558101870 0x7f7558197a70 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.795+0000 7f7560661640 1 -- 172.21.15.116:0/2502059246 >> 172.21.15.116:0/2502059246 conn(0x7f75580fcde0 msgr2=0x7f7558105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.795+0000 7f7560661640 1 -- 172.21.15.116:0/2502059246 shutdown_connections 2024-03-19T01:27:17.797 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:17.795+0000 7f7560661640 1 -- 172.21.15.116:0/2502059246 wait complete. 2024-03-19T01:27:18.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:18 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:27:18.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:18 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:27:18.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:18 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:27:18.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:18 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:27:18.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:18 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:27:18.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:18 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:27:18.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:18 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:27:18.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:18 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:27:18.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:18 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:27:19.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:19 smithi162 ceph-mon[67178]: pgmap v792: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.6 MiB/s wr, 651 op/s 2024-03-19T01:27:19.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:19 smithi116 ceph-mon[62701]: pgmap v792: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.6 MiB/s wr, 651 op/s 2024-03-19T01:27:19.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:19 smithi156 ceph-mon[66799]: pgmap v792: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 8.6 MiB/s wr, 651 op/s 2024-03-19T01:27:21.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 875.0 s, 1270.6 tps, lat 77.585 ms stddev 94.488 2024-03-19T01:27:21.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:21 smithi162 ceph-mon[67178]: pgmap v793: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 660 op/s 2024-03-19T01:27:21.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:21 smithi116 ceph-mon[62701]: pgmap v793: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 660 op/s 2024-03-19T01:27:21.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:21 smithi156 ceph-mon[66799]: pgmap v793: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 660 op/s 2024-03-19T01:27:23.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:23 smithi162 ceph-mon[67178]: pgmap v794: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 12 MiB/s wr, 726 op/s 2024-03-19T01:27:23.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:23 smithi116 ceph-mon[62701]: pgmap v794: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 12 MiB/s wr, 726 op/s 2024-03-19T01:27:23.723 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:23 smithi156 ceph-mon[66799]: pgmap v794: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 12 MiB/s wr, 726 op/s 2024-03-19T01:27:25.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:25 smithi162 ceph-mon[67178]: pgmap v795: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 12 MiB/s wr, 625 op/s 2024-03-19T01:27:25.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:25 smithi116 ceph-mon[62701]: pgmap v795: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 12 MiB/s wr, 625 op/s 2024-03-19T01:27:25.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:25 smithi156 ceph-mon[66799]: pgmap v795: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 12 MiB/s wr, 625 op/s 2024-03-19T01:27:26.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 880.0 s, 1517.6 tps, lat 66.274 ms stddev 80.251 2024-03-19T01:27:27.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:27 smithi162 ceph-mon[67178]: pgmap v796: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 MiB/s wr, 757 op/s 2024-03-19T01:27:27.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:27 smithi116 ceph-mon[62701]: pgmap v796: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 MiB/s wr, 757 op/s 2024-03-19T01:27:27.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:27 smithi156 ceph-mon[66799]: pgmap v796: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 MiB/s wr, 757 op/s 2024-03-19T01:27:29.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:29 smithi162 ceph-mon[67178]: pgmap v797: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 14 MiB/s wr, 626 op/s 2024-03-19T01:27:29.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:29 smithi116 ceph-mon[62701]: pgmap v797: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 14 MiB/s wr, 626 op/s 2024-03-19T01:27:29.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:29 smithi156 ceph-mon[66799]: pgmap v797: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 14 MiB/s wr, 626 op/s 2024-03-19T01:27:30.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.932+0000 7f8a7546c640 1 -- 172.21.15.116:0/60340151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7010b190 msgr2=0x7f8a7010b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:30.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.932+0000 7f8a7546c640 1 --2- 172.21.15.116:0/60340151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7010b190 0x7f8a7010b5f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009900 tx=0x7f8a6002bc50 comp rx=0 tx=0).stop 2024-03-19T01:27:30.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.932+0000 7f8a7546c640 1 -- 172.21.15.116:0/60340151 shutdown_connections 2024-03-19T01:27:30.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.932+0000 7f8a7546c640 1 --2- 172.21.15.116:0/60340151 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a70111f60 0x7f8a70114370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.932+0000 7f8a7546c640 1 --2- 172.21.15.116:0/60340151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7010b190 0x7f8a7010b5f0 secure :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009900 tx=0x7f8a6002bc50 comp rx=0 tx=0).stop 2024-03-19T01:27:30.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.932+0000 7f8a7546c640 1 --2- 172.21.15.116:0/60340151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a70108820 0x7f8a7010ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.933 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.932+0000 7f8a7546c640 1 -- 172.21.15.116:0/60340151 >> 172.21.15.116:0/60340151 conn(0x7f8a7006e860 msgr2=0x7f8a70070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.933+0000 7f8a7546c640 1 -- 172.21.15.116:0/60340151 shutdown_connections 2024-03-19T01:27:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.933+0000 7f8a7546c640 1 -- 172.21.15.116:0/60340151 wait complete. 2024-03-19T01:27:30.934 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.934+0000 7f8a7546c640 1 Processor -- start 2024-03-19T01:27:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.934+0000 7f8a7546c640 1 -- start start 2024-03-19T01:27:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.934+0000 7f8a7546c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a70108820 0x7f8a70119590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.934+0000 7f8a7546c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a70111f60 0x7f8a70119ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.934+0000 7f8a7546c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7011eae0 0x7f8a701b7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.934+0000 7f8a7546c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7010ce40 con 0x7f8a7011eae0 2024-03-19T01:27:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.934+0000 7f8a7546c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7010ccc0 con 0x7f8a70111f60 2024-03-19T01:27:30.935 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.934+0000 7f8a7546c640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7010cfc0 con 0x7f8a70108820 2024-03-19T01:27:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.935+0000 7f8a6e7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a70111f60 0x7f8a70119ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.935+0000 7f8a6f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7011eae0 0x7f8a701b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.935+0000 7f8a6f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7011eae0 0x7f8a701b7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42714/0 (socket says 172.21.15.116:42714) 2024-03-19T01:27:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.935+0000 7f8a6f7fe640 1 -- 172.21.15.116:0/3563056442 learned_addr learned my addr 172.21.15.116:0/3563056442 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.935+0000 7f8a6effd640 1 --2- 172.21.15.116:0/3563056442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a70108820 0x7f8a70119590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.935+0000 7f8a6f7fe640 1 -- 172.21.15.116:0/3563056442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a70108820 msgr2=0x7f8a70119590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.935+0000 7f8a6f7fe640 1 --2- 172.21.15.116:0/3563056442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a70108820 0x7f8a70119590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.935+0000 7f8a6f7fe640 1 -- 172.21.15.116:0/3563056442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a70111f60 msgr2=0x7f8a70119ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.935+0000 7f8a6f7fe640 1 --2- 172.21.15.116:0/3563056442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a70111f60 0x7f8a70119ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.936 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.935+0000 7f8a6f7fe640 1 -- 172.21.15.116:0/3563056442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7011f010 con 0x7f8a7011eae0 2024-03-19T01:27:30.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.936+0000 7f8a6f7fe640 1 --2- 172.21.15.116:0/3563056442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7011eae0 0x7f8a701b7160 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400d750 tx=0x7f8a6400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:30.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.936+0000 7f8a57fff640 1 -- 172.21.15.116:0/3563056442 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64017070 con 0x7f8a7011eae0 2024-03-19T01:27:30.937 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.936+0000 7f8a7546c640 1 -- 172.21.15.116:0/3563056442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a701b76a0 con 0x7f8a7011eae0 2024-03-19T01:27:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.936+0000 7f8a7546c640 1 -- 172.21.15.116:0/3563056442 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a701b7aa0 con 0x7f8a7011eae0 2024-03-19T01:27:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.937+0000 7f8a57fff640 1 -- 172.21.15.116:0/3563056442 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a64005500 con 0x7f8a7011eae0 2024-03-19T01:27:30.938 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.937+0000 7f8a57fff640 1 -- 172.21.15.116:0/3563056442 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64004d90 con 0x7f8a7011eae0 2024-03-19T01:27:30.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.940+0000 7f8a57fff640 1 -- 172.21.15.116:0/3563056442 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f8a64010460 con 0x7f8a7011eae0 2024-03-19T01:27:30.941 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a78058740 1 Processor -- start 2024-03-19T01:27:30.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a78058740 1 -- start start 2024-03-19T01:27:30.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a78058740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813100 0x55948b813520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.942 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a78058740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b8185f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a78058740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 0x55948b81b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a78058740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55948b777be0 con 0x55948b818c60 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a78058740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55948b720960 con 0x55948b813af0 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a78058740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55948b773c80 con 0x55948b813100 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a57fff640 1 --2- 172.21.15.116:0/3563056442 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8a580a29f0 0x7f8a580a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a6effd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b8185f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a6effd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b8185f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:57966/0 (socket says 172.21.15.116:57966) 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a6effd640 1 -- 172.21.15.116:0/2953704043 learned_addr learned my addr 172.21.15.116:0/2953704043 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a6e7fc640 1 --2- 172.21.15.116:0/2953704043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813100 0x55948b813520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a57fff640 1 -- 172.21.15.116:0/3563056442 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8a640ce380 con 0x7f8a7011eae0 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a6effd640 1 -- 172.21.15.116:0/2953704043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813100 msgr2=0x55948b813520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.941+0000 7f8a6effd640 1 --2- 172.21.15.116:0/2953704043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813100 0x55948b813520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.942+0000 7f8a6effd640 1 -- 172.21.15.116:0/2953704043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 msgr2=0x55948b81b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:30.943 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.942+0000 7f8a6effd640 1 --2- 172.21.15.116:0/2953704043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 0x55948b81b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.942+0000 7f8a6effd640 1 -- 172.21.15.116:0/2953704043 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55948b717d20 con 0x55948b813af0 2024-03-19T01:27:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.942+0000 7f8a6effd640 1 --2- 172.21.15.116:0/2953704043 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b8185f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8a68010400 tx=0x7f8a680108c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e31dfa1ffdac685a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.942+0000 7f8a6f7fe640 1 --2- 172.21.15.116:0/3563056442 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8a580a29f0 0x7f8a580a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.943+0000 7f8a557fa640 1 -- 172.21.15.116:0/2953704043 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68003b70 con 0x55948b813af0 2024-03-19T01:27:30.944 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.943+0000 7f8a557fa640 1 -- 172.21.15.116:0/2953704043 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a68003d10 con 0x55948b813af0 2024-03-19T01:27:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.943+0000 7f8a557fa640 1 -- 172.21.15.116:0/2953704043 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68033b50 con 0x55948b813af0 2024-03-19T01:27:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.944+0000 7f8a78058740 1 -- 172.21.15.116:0/2953704043 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 msgr2=0x55948b8185f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:30.945 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.944+0000 7f8a78058740 1 --2- 172.21.15.116:0/2953704043 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b8185f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8a68010400 tx=0x7f8a680108c0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.946+0000 7f8a6f7fe640 1 --2- 172.21.15.116:0/3563056442 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8a580a29f0 0x7f8a580a4eb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8a64097670 tx=0x7f8a640ce5d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.946+0000 7f8a78058740 1 -- 172.21.15.116:0/2953704043 shutdown_connections 2024-03-19T01:27:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.946+0000 7f8a78058740 1 --2- 172.21.15.116:0/2953704043 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 0x55948b81b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.946+0000 7f8a78058740 1 --2- 172.21.15.116:0/2953704043 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b8185f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.946+0000 7f8a78058740 1 --2- 172.21.15.116:0/2953704043 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813100 0x55948b813520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.947 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.946+0000 7f8a78058740 1 -- 172.21.15.116:0/2953704043 >> 172.21.15.116:0/2953704043 conn(0x55948b80e670 msgr2=0x55948b80ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.946+0000 7f8a78058740 1 -- 172.21.15.116:0/2953704043 shutdown_connections 2024-03-19T01:27:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.946+0000 7f8a78058740 1 -- 172.21.15.116:0/2953704043 wait complete. 2024-03-19T01:27:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.947+0000 7f8a78058740 1 -- start start 2024-03-19T01:27:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.947+0000 7f8a78058740 1 Processor -- start 2024-03-19T01:27:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.947+0000 7f8a78058740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813100 0x55948b81d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.947+0000 7f8a78058740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b81d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.947+0000 7f8a78058740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 0x55948b824700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.948 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.947+0000 7f8a78058740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55948b81bc60 con 0x55948b818c60 2024-03-19T01:27:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.947+0000 7f8a78058740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55948b733db0 con 0x55948b813af0 2024-03-19T01:27:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.947+0000 7f8a78058740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x55948b81bde0 con 0x55948b813100 2024-03-19T01:27:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.947+0000 7f8a78058740 10 client.0 _ll_register_callbacks cb 0x55948b738e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:27:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.947+0000 7f8a6e7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813100 0x55948b81d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.947+0000 7f8a6e7fc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813100 0x55948b81d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55950/0 (socket says 172.21.15.116:55950) 2024-03-19T01:27:30.950 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.947+0000 7f8a6e7fc640 1 -- 172.21.15.116:0/1125202786 learned_addr learned my addr 172.21.15.116:0/1125202786 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.947+0000 7f8a6effd640 1 --2- 172.21.15.116:0/1125202786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b81d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.948+0000 7f8a6e7fc640 1 --2- 172.21.15.116:0/1125202786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 0x55948b824700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.948+0000 7f8a6effd640 1 -- 172.21.15.116:0/1125202786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813100 msgr2=0x55948b81d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.948+0000 7f8a6effd640 1 --2- 172.21.15.116:0/1125202786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813100 0x55948b81d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.948+0000 7f8a6effd640 1 -- 172.21.15.116:0/1125202786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 msgr2=0x55948b824700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.948+0000 7f8a6effd640 1 --2- 172.21.15.116:0/1125202786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 0x55948b824700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.948+0000 7f8a6effd640 1 -- 172.21.15.116:0/1125202786 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55948b732c60 con 0x55948b813af0 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.948+0000 7f8a6effd640 1 --2- 172.21.15.116:0/1125202786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b81d6c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x55948b80f5e0 tx=0x7f8a68004c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f0b149b24c0ef8a server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.948+0000 7f8a52ff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.948+0000 7f8a52ff5640 1 -- 172.21.15.116:0/1125202786 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68033850 con 0x55948b813af0 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.948+0000 7f8a78058740 1 -- 172.21.15.116:0/1125202786 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a68003d10 con 0x55948b813af0 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.948+0000 7f8a52ff5640 1 -- 172.21.15.116:0/1125202786 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8a680339f0 con 0x55948b813af0 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.948+0000 7f8a52ff5640 1 -- 172.21.15.116:0/1125202786 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f8a6804f050 con 0x55948b813af0 2024-03-19T01:27:30.951 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.949+0000 7f8a78058740 10 client.26282 fetch_fsmap learned FSMap version 49 2024-03-19T01:27:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.949+0000 7f8a78058740 1 -- 172.21.15.116:0/1125202786 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55948b48c210 con 0x55948b813af0 2024-03-19T01:27:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.949+0000 7f8a52ff5640 1 -- 172.21.15.116:0/1125202786 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a68003d10 con 0x55948b813af0 2024-03-19T01:27:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.949+0000 7f8a52ff5640 1 -- 172.21.15.116:0/1125202786 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f8a68005040 con 0x55948b813af0 2024-03-19T01:27:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.949+0000 7f8a78058740 10 client.26282 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:27:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.949+0000 7f8a78058740 10 client.26282 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:27:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.949+0000 7f8a78058740 20 client.26282 populate_metadata read hostname 'smithi116' 2024-03-19T01:27:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.949+0000 7f8a78058740 1 --2- 172.21.15.116:0/1125202786 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55948b828c50 0x55948b82b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.949+0000 7f8a78058740 4 client.26282 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:27:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.949+0000 7f8a78058740 1 -- 172.21.15.116:0/1125202786 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55948b785120 con 0x55948b828c50 2024-03-19T01:27:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.949+0000 7f8a6e7fc640 1 --2- 172.21.15.116:0/1125202786 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55948b828c50 0x55948b82b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.950+0000 7f8a52ff5640 10 client.26282 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:30.952 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.950+0000 7f8a6e7fc640 1 --2- 172.21.15.116:0/1125202786 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55948b828c50 0x55948b82b070 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d2b3b200dc4f14 server_cookie=6af6dfee3ff53ac2 in_seq=0 out_seq=0 2024-03-19T01:27:30.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.951+0000 7f8a52ff5640 1 -- 172.21.15.116:0/1125202786 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55948b785120 con 0x55948b828c50 2024-03-19T01:27:30.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.951+0000 7f8a52ff5640 10 client.26282 handle_command_reply: tid=0 2024-03-19T01:27:30.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.951+0000 7f8a78058740 1 client.26282 shutdown 2024-03-19T01:27:30.953 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.951+0000 7f8a78058740 1 -- 172.21.15.116:0/1125202786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 msgr2=0x55948b81d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.951+0000 7f8a78058740 1 --2- 172.21.15.116:0/1125202786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b81d6c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x55948b80f5e0 tx=0x7f8a68004c70 comp rx=0 tx=0).stop 2024-03-19T01:27:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.951+0000 7f8a78058740 1 -- 172.21.15.116:0/1125202786 shutdown_connections 2024-03-19T01:27:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.951+0000 7f8a78058740 1 --2- 172.21.15.116:0/1125202786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 0x55948b824700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.951+0000 7f8a78058740 1 --2- 172.21.15.116:0/1125202786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b81d6c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.951+0000 7f8a78058740 1 --2- 172.21.15.116:0/1125202786 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55948b828c50 0x55948b82b070 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.951+0000 7f8a78058740 1 --2- 172.21.15.116:0/1125202786 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813100 0x55948b81d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.951+0000 7f8a78058740 1 -- 172.21.15.116:0/1125202786 >> 172.21.15.116:0/1125202786 conn(0x55948b80e670 msgr2=0x55948b80f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.951+0000 7f8a52ff5640 0 client.26282 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.951+0000 7f8a78058740 1 -- 172.21.15.116:0/1125202786 shutdown_connections 2024-03-19T01:27:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.952+0000 7f8a78058740 1 -- 172.21.15.116:0/1125202786 wait complete. 2024-03-19T01:27:30.954 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.952+0000 7f8a78058740 20 client.26282 trim_cache size 0 max 16384 2024-03-19T01:27:30.986 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a78058740 1 Processor -- start 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a78058740 1 -- start start 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a78058740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813100 0x55948b81cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a78058740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813af0 0x55948b8246c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a78058740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 0x55948b80d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a78058740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c00e4d0 con 0x55948b818c60 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a78058740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c010010 con 0x55948b813100 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a78058740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c00eba0 con 0x55948b813af0 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a6e7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813100 0x55948b81cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a6effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813af0 0x55948b8246c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a6effd640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813af0 0x55948b8246c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:55958/0 (socket says 172.21.15.116:55958) 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a6e7fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813100 0x55948b81cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:57984/0 (socket says 172.21.15.116:57984) 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a6effd640 1 -- 172.21.15.116:0/1776849095 learned_addr learned my addr 172.21.15.116:0/1776849095 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.985+0000 7f8a6e7fc640 1 --2- 172.21.15.116:0/1776849095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 0x55948b80d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.987 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.986+0000 7f8a6effd640 1 -- 172.21.15.116:0/1776849095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813100 msgr2=0x55948b81cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.986+0000 7f8a6effd640 1 --2- 172.21.15.116:0/1776849095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813100 0x55948b81cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.986+0000 7f8a6effd640 1 -- 172.21.15.116:0/1776849095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 msgr2=0x55948b80d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.986+0000 7f8a6effd640 1 --2- 172.21.15.116:0/1776849095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 0x55948b80d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.986+0000 7f8a6effd640 1 -- 172.21.15.116:0/1776849095 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55948b558620 con 0x55948b813af0 2024-03-19T01:27:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.986+0000 7f8a6effd640 1 --2- 172.21.15.116:0/1776849095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813af0 0x55948b8246c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8a68006010 tx=0x7f8a68009ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fab6bd7f7bced8a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.986+0000 7f8a54ff9640 1 -- 172.21.15.116:0/1776849095 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68009b00 con 0x55948b813af0 2024-03-19T01:27:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.986+0000 7f8a54ff9640 1 -- 172.21.15.116:0/1776849095 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a68005650 con 0x55948b813af0 2024-03-19T01:27:30.988 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.986+0000 7f8a54ff9640 1 -- 172.21.15.116:0/1776849095 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68003d10 con 0x55948b813af0 2024-03-19T01:27:30.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.987+0000 7f8a78058740 1 -- 172.21.15.116:0/1776849095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813af0 msgr2=0x55948b8246c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:30.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.987+0000 7f8a78058740 1 --2- 172.21.15.116:0/1776849095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813af0 0x55948b8246c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8a68006010 tx=0x7f8a68009ef0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.987+0000 7f8a78058740 1 -- 172.21.15.116:0/1776849095 shutdown_connections 2024-03-19T01:27:30.991 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.987+0000 7f8a78058740 1 --2- 172.21.15.116:0/1776849095 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b818c60 0x55948b80d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.987+0000 7f8a78058740 1 --2- 172.21.15.116:0/1776849095 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b813af0 0x55948b8246c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.987+0000 7f8a78058740 1 --2- 172.21.15.116:0/1776849095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813100 0x55948b81cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.987+0000 7f8a78058740 1 -- 172.21.15.116:0/1776849095 >> 172.21.15.116:0/1776849095 conn(0x55948b80e670 msgr2=0x55948b81c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.987+0000 7f8a78058740 1 -- 172.21.15.116:0/1776849095 shutdown_connections 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.987+0000 7f8a78058740 1 -- 172.21.15.116:0/1776849095 wait complete. 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.988+0000 7f8a78058740 1 -- start start 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.988+0000 7f8a78058740 1 Processor -- start 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.988+0000 7f8a78058740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b813100 0x55948b821940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.988+0000 7f8a78058740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b821e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.988+0000 7f8a78058740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b818c60 0x55948b822e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.988+0000 7f8a78058740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c00f940 con 0x55948b813100 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.988+0000 7f8a78058740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c00f270 con 0x55948b813af0 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.988+0000 7f8a6e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b813100 0x55948b821940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.992 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.988+0000 7f8a6e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b813100 0x55948b821940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42750/0 (socket says 172.21.15.116:42750) 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.988+0000 7f8a6e7fc640 1 -- 172.21.15.116:0/1353324769 learned_addr learned my addr 172.21.15.116:0/1353324769 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a6effd640 1 --2- 172.21.15.116:0/1353324769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b821e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a78058740 1 -- 172.21.15.116:0/1353324769 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c00c4b0 con 0x55948b818c60 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a78058740 10 client.0 _ll_register_callbacks cb 0x7f8a60009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a6e7fc640 1 --2- 172.21.15.116:0/1353324769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b818c60 0x55948b822e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a6effd640 1 -- 172.21.15.116:0/1353324769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b818c60 msgr2=0x55948b822e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a6effd640 1 --2- 172.21.15.116:0/1353324769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b818c60 0x55948b822e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a6effd640 1 -- 172.21.15.116:0/1353324769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b813100 msgr2=0x55948b821940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a6effd640 1 --2- 172.21.15.116:0/1353324769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b813100 0x55948b821940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a6effd640 1 -- 172.21.15.116:0/1353324769 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55948b823410 con 0x55948b813af0 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a6effd640 1 --2- 172.21.15.116:0/1353324769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b821e80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8a68010ec0 tx=0x7f8a68009d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=702ab871c8bd6241 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a52ff5640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a52ff5640 1 -- 172.21.15.116:0/1353324769 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68004f70 con 0x55948b813af0 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a78058740 1 -- 172.21.15.116:0/1353324769 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a68005650 con 0x55948b813af0 2024-03-19T01:27:30.993 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.989+0000 7f8a52ff5640 1 -- 172.21.15.116:0/1353324769 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8a68058040 con 0x55948b813af0 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.990+0000 7f8a52ff5640 1 -- 172.21.15.116:0/1353324769 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f8a68062070 con 0x55948b813af0 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.990+0000 7f8a78058740 10 client.26288 fetch_fsmap learned FSMap version 49 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.990+0000 7f8a78058740 1 -- 172.21.15.116:0/1353324769 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55948b824720 con 0x55948b813af0 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.990+0000 7f8a52ff5640 1 -- 172.21.15.116:0/1353324769 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a68005650 con 0x55948b813af0 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.990+0000 7f8a52ff5640 1 -- 172.21.15.116:0/1353324769 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f8a680486f0 con 0x55948b813af0 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.990+0000 7f8a78058740 10 client.26288 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.990+0000 7f8a78058740 10 client.26288 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.990+0000 7f8a78058740 20 client.26288 populate_metadata read hostname 'smithi116' 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.990+0000 7f8a78058740 1 --2- 172.21.15.116:0/1353324769 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55948b88f3e0 0x55948b891800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.990+0000 7f8a78058740 4 client.26288 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.990+0000 7f8a78058740 1 -- 172.21.15.116:0/1353324769 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55948b77f470 con 0x55948b88f3e0 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.990+0000 7f8a6e7fc640 1 --2- 172.21.15.116:0/1353324769 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55948b88f3e0 0x55948b891800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.991+0000 7f8a52ff5640 10 client.26288 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:30.994 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:30.991+0000 7f8a6e7fc640 1 --2- 172.21.15.116:0/1353324769 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55948b88f3e0 0x55948b891800 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=121d5f5bc449f8d4 server_cookie=df778e2fc6551f96 in_seq=0 out_seq=0 2024-03-19T01:27:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a52ff5640 1 -- 172.21.15.116:0/1353324769 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55948b77f470 con 0x55948b88f3e0 2024-03-19T01:27:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a52ff5640 10 client.26288 handle_command_reply: tid=0 2024-03-19T01:27:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a78058740 1 client.26288 shutdown 2024-03-19T01:27:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a78058740 1 -- 172.21.15.116:0/1353324769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 msgr2=0x55948b821e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a78058740 1 --2- 172.21.15.116:0/1353324769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b821e80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8a68010ec0 tx=0x7f8a68009d30 comp rx=0 tx=0).stop 2024-03-19T01:27:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a78058740 1 -- 172.21.15.116:0/1353324769 shutdown_connections 2024-03-19T01:27:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a78058740 1 --2- 172.21.15.116:0/1353324769 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x55948b88f3e0 0x55948b891800 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a52ff5640 0 client.26288 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a78058740 1 --2- 172.21.15.116:0/1353324769 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x55948b818c60 0x55948b822e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a78058740 1 --2- 172.21.15.116:0/1353324769 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55948b813af0 0x55948b821e80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:31.001 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a78058740 1 --2- 172.21.15.116:0/1353324769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55948b813100 0x55948b821940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:31.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a78058740 1 -- 172.21.15.116:0/1353324769 >> 172.21.15.116:0/1353324769 conn(0x55948b80e670 msgr2=0x55948b80d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:31.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a78058740 1 -- 172.21.15.116:0/1353324769 shutdown_connections 2024-03-19T01:27:31.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a78058740 1 -- 172.21.15.116:0/1353324769 wait complete. 2024-03-19T01:27:31.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.000+0000 7f8a78058740 20 client.26288 trim_cache size 0 max 16384 2024-03-19T01:27:31.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:27:31.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:27:31.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:27:31.002 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:27:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.001+0000 7f8a7546c640 1 -- 172.21.15.116:0/3563056442 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8a580a29f0 msgr2=0x7f8a580a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.001+0000 7f8a7546c640 1 --2- 172.21.15.116:0/3563056442 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8a580a29f0 0x7f8a580a4eb0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8a64097670 tx=0x7f8a640ce5d0 comp rx=0 tx=0).stop 2024-03-19T01:27:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.001+0000 7f8a7546c640 1 -- 172.21.15.116:0/3563056442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7011eae0 msgr2=0x7f8a701b7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.001+0000 7f8a7546c640 1 --2- 172.21.15.116:0/3563056442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7011eae0 0x7f8a701b7160 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400d750 tx=0x7f8a6400dc10 comp rx=0 tx=0).stop 2024-03-19T01:27:31.003 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.001+0000 7f8a7546c640 1 -- 172.21.15.116:0/3563056442 shutdown_connections 2024-03-19T01:27:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.001+0000 7f8a7546c640 1 --2- 172.21.15.116:0/3563056442 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a7011eae0 0x7f8a701b7160 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.001+0000 7f8a7546c640 1 --2- 172.21.15.116:0/3563056442 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8a580a29f0 0x7f8a580a4eb0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.001+0000 7f8a7546c640 1 --2- 172.21.15.116:0/3563056442 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a70111f60 0x7f8a70119ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.001+0000 7f8a7546c640 1 --2- 172.21.15.116:0/3563056442 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8a70108820 0x7f8a70119590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.001+0000 7f8a7546c640 1 -- 172.21.15.116:0/3563056442 >> 172.21.15.116:0/3563056442 conn(0x7f8a7006e860 msgr2=0x7f8a70112750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.001+0000 7f8a7546c640 1 -- 172.21.15.116:0/3563056442 shutdown_connections 2024-03-19T01:27:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:31.001+0000 7f8a7546c640 1 -- 172.21.15.116:0/3563056442 wait complete. 2024-03-19T01:27:31.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 885.0 s, 1516.0 tps, lat 66.055 ms stddev 75.940 2024-03-19T01:27:31.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:31 smithi162 ceph-mon[67178]: pgmap v798: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 636 op/s 2024-03-19T01:27:31.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:31 smithi116 ceph-mon[62701]: pgmap v798: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 636 op/s 2024-03-19T01:27:31.725 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:31 smithi156 ceph-mon[66799]: pgmap v798: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 636 op/s 2024-03-19T01:27:33.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:33 smithi162 ceph-mon[67178]: pgmap v799: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 15 MiB/s wr, 684 op/s 2024-03-19T01:27:33.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:33 smithi116 ceph-mon[62701]: pgmap v799: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 15 MiB/s wr, 684 op/s 2024-03-19T01:27:33.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:33 smithi156 ceph-mon[66799]: pgmap v799: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 15 MiB/s wr, 684 op/s 2024-03-19T01:27:35.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:35 smithi162 ceph-mon[67178]: pgmap v800: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 13 MiB/s wr, 571 op/s 2024-03-19T01:27:35.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:35 smithi116 ceph-mon[62701]: pgmap v800: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 13 MiB/s wr, 571 op/s 2024-03-19T01:27:35.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:35 smithi156 ceph-mon[66799]: pgmap v800: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 13 MiB/s wr, 571 op/s 2024-03-19T01:27:36.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 890.0 s, 1273.4 tps, lat 78.842 ms stddev 86.378 2024-03-19T01:27:37.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:37 smithi162 ceph-mon[67178]: pgmap v801: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 646 op/s 2024-03-19T01:27:37.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:37 smithi116 ceph-mon[62701]: pgmap v801: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 646 op/s 2024-03-19T01:27:37.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:37 smithi156 ceph-mon[66799]: pgmap v801: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 MiB/s wr, 646 op/s 2024-03-19T01:27:39.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:39 smithi162 ceph-mon[67178]: pgmap v802: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 581 op/s 2024-03-19T01:27:39.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:39 smithi116 ceph-mon[62701]: pgmap v802: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 581 op/s 2024-03-19T01:27:39.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:39 smithi156 ceph-mon[66799]: pgmap v802: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 581 op/s 2024-03-19T01:27:41.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 895.0 s, 1414.4 tps, lat 70.791 ms stddev 77.131 2024-03-19T01:27:41.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:41 smithi162 ceph-mon[67178]: pgmap v803: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 13 MiB/s wr, 648 op/s 2024-03-19T01:27:41.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:41 smithi116 ceph-mon[62701]: pgmap v803: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 13 MiB/s wr, 648 op/s 2024-03-19T01:27:41.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:41 smithi156 ceph-mon[66799]: pgmap v803: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 13 MiB/s wr, 648 op/s 2024-03-19T01:27:43.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:43 smithi162 ceph-mon[67178]: pgmap v804: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 14 MiB/s wr, 716 op/s 2024-03-19T01:27:43.657 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:43 smithi116 ceph-mon[62701]: pgmap v804: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 14 MiB/s wr, 716 op/s 2024-03-19T01:27:43.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:43 smithi156 ceph-mon[66799]: pgmap v804: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 14 MiB/s wr, 716 op/s 2024-03-19T01:27:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.117+0000 7f4fcc27e640 1 -- 172.21.15.116:0/2444896974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc4101930 msgr2=0x7f4fc4101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:44.117 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.117+0000 7f4fcc27e640 1 --2- 172.21.15.116:0/2444896974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc4101930 0x7f4fc4101db0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009900 tx=0x7f4fb402bc30 comp rx=0 tx=0).stop 2024-03-19T01:27:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.118+0000 7f4fcc27e640 1 -- 172.21.15.116:0/2444896974 shutdown_connections 2024-03-19T01:27:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.118+0000 7f4fcc27e640 1 --2- 172.21.15.116:0/2444896974 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fc41022f0 0x7f4fc4108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.118+0000 7f4fcc27e640 1 --2- 172.21.15.116:0/2444896974 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc4101930 0x7f4fc4101db0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.118+0000 7f4fcc27e640 1 --2- 172.21.15.116:0/2444896974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fc4100740 0x7f4fc4100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.118+0000 7f4fcc27e640 1 -- 172.21.15.116:0/2444896974 >> 172.21.15.116:0/2444896974 conn(0x7f4fc40fbcb0 msgr2=0x7f4fc40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.118+0000 7f4fcc27e640 1 -- 172.21.15.116:0/2444896974 shutdown_connections 2024-03-19T01:27:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.118+0000 7f4fcc27e640 1 -- 172.21.15.116:0/2444896974 wait complete. 2024-03-19T01:27:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.119+0000 7f4fcc27e640 1 Processor -- start 2024-03-19T01:27:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.119+0000 7f4fcc27e640 1 -- start start 2024-03-19T01:27:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.119+0000 7f4fcc27e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc4100740 0x7f4fc4197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.119+0000 7f4fcc27e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fc4101930 0x7f4fc4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.119+0000 7f4fcc27e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fc41022f0 0x7f4fc419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.119+0000 7f4fcc27e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc410c3a0 con 0x7f4fc4101930 2024-03-19T01:27:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.119+0000 7f4fcc27e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc410c220 con 0x7f4fc4100740 2024-03-19T01:27:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.119+0000 7f4fcc27e640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc410c520 con 0x7f4fc41022f0 2024-03-19T01:27:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.120+0000 7f4fc9819640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fc4101930 0x7f4fc4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.120+0000 7f4fca81b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fc41022f0 0x7f4fc419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.120+0000 7f4fca01a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc4100740 0x7f4fc4197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.120+0000 7f4fca81b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fc41022f0 0x7f4fc419f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52428/0 (socket says 172.21.15.116:52428) 2024-03-19T01:27:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.120+0000 7f4fca81b640 1 -- 172.21.15.116:0/997040518 learned_addr learned my addr 172.21.15.116:0/997040518 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.120+0000 7f4fca01a640 1 -- 172.21.15.116:0/997040518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fc41022f0 msgr2=0x7f4fc419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.120+0000 7f4fca01a640 1 --2- 172.21.15.116:0/997040518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fc41022f0 0x7f4fc419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.120+0000 7f4fca01a640 1 -- 172.21.15.116:0/997040518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fc4101930 msgr2=0x7f4fc4198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:44.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.120+0000 7f4fca01a640 1 --2- 172.21.15.116:0/997040518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fc4101930 0x7f4fc4198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.120+0000 7f4fca01a640 1 -- 172.21.15.116:0/997040518 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc419f6f0 con 0x7f4fc4100740 2024-03-19T01:27:44.122 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.121+0000 7f4fca01a640 1 --2- 172.21.15.116:0/997040518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc4100740 0x7f4fc4197ad0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000e9c0 tx=0x7f4fc000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:44.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.121+0000 7f4fbb7fe640 1 -- 172.21.15.116:0/997040518 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc000cd80 con 0x7f4fc4100740 2024-03-19T01:27:44.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.121+0000 7f4fbb7fe640 1 -- 172.21.15.116:0/997040518 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fc0010910 con 0x7f4fc4100740 2024-03-19T01:27:44.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.122+0000 7f4fbb7fe640 1 -- 172.21.15.116:0/997040518 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc0018980 con 0x7f4fc4100740 2024-03-19T01:27:44.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.122+0000 7f4fcc27e640 1 -- 172.21.15.116:0/997040518 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc419f9e0 con 0x7f4fc4100740 2024-03-19T01:27:44.123 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.122+0000 7f4fcc27e640 1 -- 172.21.15.116:0/997040518 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc419feb0 con 0x7f4fc4100740 2024-03-19T01:27:44.127 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.126+0000 7f4fcee69740 1 Processor -- start 2024-03-19T01:27:44.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.127+0000 7f4fcee69740 1 -- start start 2024-03-19T01:27:44.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.127+0000 7f4fcee69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d2e100 0x5630c8d2e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.127+0000 7f4fcee69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d335f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.127+0000 7f4fcee69740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d33c60 0x5630c8d360f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.127+0000 7f4fcee69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5630c8c92be0 con 0x5630c8d2e100 2024-03-19T01:27:44.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.127+0000 7f4fcee69740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5630c8c3b960 con 0x5630c8d33c60 2024-03-19T01:27:44.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.127+0000 7f4fcee69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5630c8c8ec80 con 0x5630c8d2eaf0 2024-03-19T01:27:44.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.127+0000 7f4fbb7fe640 1 -- 172.21.15.116:0/997040518 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f4fc0018b20 con 0x7f4fc4100740 2024-03-19T01:27:44.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.127+0000 7f4fc9819640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d33c60 0x5630c8d360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.127+0000 7f4fca81b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d2e100 0x5630c8d2e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.128 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.127+0000 7f4fc9819640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d33c60 0x5630c8d360f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:38630/0 (socket says 172.21.15.116:38630) 2024-03-19T01:27:44.129 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.127+0000 7f4fc9819640 1 -- 172.21.15.116:0/2919530008 learned_addr learned my addr 172.21.15.116:0/2919530008 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:44.129 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.128+0000 7f4fca01a640 1 --2- 172.21.15.116:0/2919530008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.129 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.128+0000 7f4fc9819640 1 -- 172.21.15.116:0/2919530008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 msgr2=0x5630c8d335f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.128+0000 7f4fc9819640 1 --2- 172.21.15.116:0/2919530008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d335f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.128+0000 7f4fc9819640 1 -- 172.21.15.116:0/2919530008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d2e100 msgr2=0x5630c8d2e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.128+0000 7f4fc9819640 1 --2- 172.21.15.116:0/2919530008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d2e100 0x5630c8d2e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.128+0000 7f4fc9819640 1 -- 172.21.15.116:0/2919530008 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630c8c32d20 con 0x5630c8d33c60 2024-03-19T01:27:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.128+0000 7f4fca81b640 1 --2- 172.21.15.116:0/2919530008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d2e100 0x5630c8d2e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:27:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.128+0000 7f4fca01a640 1 --2- 172.21.15.116:0/2919530008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d335f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:27:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.129+0000 7f4fc9819640 1 --2- 172.21.15.116:0/2919530008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d33c60 0x5630c8d360f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009900 tx=0x7f4fb402c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5727839ee9c7c8f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.129+0000 7f4fb8bfe640 1 -- 172.21.15.116:0/2919530008 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb402cb10 con 0x5630c8d33c60 2024-03-19T01:27:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.129+0000 7f4fb8bfe640 1 -- 172.21.15.116:0/2919530008 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fb402c5d0 con 0x5630c8d33c60 2024-03-19T01:27:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.129+0000 7f4fbb7fe640 1 --2- 172.21.15.116:0/997040518 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4fb00a2790 0x7f4fb00a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.129+0000 7f4fbb7fe640 1 -- 172.21.15.116:0/997040518 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4fc00ceb80 con 0x7f4fc4100740 2024-03-19T01:27:44.131 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.129+0000 7f4fb8bfe640 1 -- 172.21.15.116:0/2919530008 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb40046c0 con 0x5630c8d33c60 2024-03-19T01:27:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.130+0000 7f4fca81b640 1 --2- 172.21.15.116:0/997040518 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4fb00a2790 0x7f4fb00a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.130+0000 7f4fcee69740 1 -- 172.21.15.116:0/2919530008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d33c60 msgr2=0x5630c8d360f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.130+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2919530008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d33c60 0x5630c8d360f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4009900 tx=0x7f4fb402c7e0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.130+0000 7f4fcee69740 1 -- 172.21.15.116:0/2919530008 shutdown_connections 2024-03-19T01:27:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.130+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2919530008 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d33c60 0x5630c8d360f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.130+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2919530008 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d335f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.130+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2919530008 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d2e100 0x5630c8d2e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.130+0000 7f4fcee69740 1 -- 172.21.15.116:0/2919530008 >> 172.21.15.116:0/2919530008 conn(0x5630c8d29670 msgr2=0x5630c8d29a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.130+0000 7f4fcee69740 1 -- 172.21.15.116:0/2919530008 shutdown_connections 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.130+0000 7f4fca81b640 1 --2- 172.21.15.116:0/997040518 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4fb00a2790 0x7f4fb00a4c50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x5630c8d2f350 tx=0x7f4fbc009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.130+0000 7f4fcee69740 1 -- 172.21.15.116:0/2919530008 wait complete. 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.131+0000 7f4fcee69740 1 -- start start 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.131+0000 7f4fcee69740 1 Processor -- start 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.131+0000 7f4fcee69740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2e100 0x5630c8d3c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.131+0000 7f4fcee69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d3c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.131+0000 7f4fcee69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d33c60 0x5630c8d43a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.131+0000 7f4fcee69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5630c8d36c50 con 0x5630c8d33c60 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.131+0000 7f4fcee69740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5630c8c4edb0 con 0x5630c8d2e100 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.131+0000 7f4fcee69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5630c8d36dd0 con 0x5630c8d2eaf0 2024-03-19T01:27:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.131+0000 7f4fcee69740 10 client.0 _ll_register_callbacks cb 0x5630c8c53e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:27:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.132+0000 7f4fc9819640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d3c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.132+0000 7f4fc9819640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d3c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52444/0 (socket says 172.21.15.116:52444) 2024-03-19T01:27:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.132+0000 7f4fc9819640 1 -- 172.21.15.116:0/2786989459 learned_addr learned my addr 172.21.15.116:0/2786989459 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:44.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.132+0000 7f4fca01a640 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2e100 0x5630c8d3c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.135 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.133+0000 7f4fca81b640 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d33c60 0x5630c8d43a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.133+0000 7f4fca01a640 1 -- 172.21.15.116:0/2786989459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 msgr2=0x5630c8d3c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.133+0000 7f4fca01a640 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d3c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.133+0000 7f4fca01a640 1 -- 172.21.15.116:0/2786989459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d33c60 msgr2=0x5630c8d43a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.133+0000 7f4fca01a640 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d33c60 0x5630c8d43a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.133+0000 7f4fca01a640 1 -- 172.21.15.116:0/2786989459 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630c8c4dc60 con 0x5630c8d2e100 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.133+0000 7f4fca81b640 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d33c60 0x5630c8d43a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.133+0000 7f4fca01a640 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2e100 0x5630c8d3c4b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x5630c8d2f560 tx=0x7f4fc009ae00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca151fb9b95a77e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.133+0000 7f4fadffb640 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.134+0000 7f4fadffb640 1 -- 172.21.15.116:0/2786989459 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fc00a39b0 con 0x5630c8d2e100 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.134+0000 7f4fcee69740 1 -- 172.21.15.116:0/2786989459 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fb402c5d0 con 0x5630c8d2e100 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.134+0000 7f4fadffb640 1 -- 172.21.15.116:0/2786989459 <== mon.2 v2:172.21.15.156:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4fc009f150 con 0x5630c8d2e100 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.134+0000 7f4fadffb640 1 -- 172.21.15.116:0/2786989459 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f4fc009f5f0 con 0x5630c8d2e100 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.134+0000 7f4fcee69740 10 client.26300 fetch_fsmap learned FSMap version 49 2024-03-19T01:27:44.136 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.134+0000 7f4fcee69740 1 -- 172.21.15.116:0/2786989459 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5630c89a7210 con 0x5630c8d2e100 2024-03-19T01:27:44.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.135+0000 7f4fadffb640 1 -- 172.21.15.116:0/2786989459 <== mon.2 v2:172.21.15.156:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fb402c5d0 con 0x5630c8d2e100 2024-03-19T01:27:44.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.135+0000 7f4fadffb640 1 -- 172.21.15.116:0/2786989459 <== mon.2 v2:172.21.15.156:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f4fc009a8f0 con 0x5630c8d2e100 2024-03-19T01:27:44.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.135+0000 7f4fcee69740 10 client.26300 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:27:44.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.135+0000 7f4fcee69740 10 client.26300 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:27:44.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.135+0000 7f4fcee69740 20 client.26300 populate_metadata read hostname 'smithi116' 2024-03-19T01:27:44.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.135+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5630c8d48710 0x5630c8d4ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.135+0000 7f4fcee69740 4 client.26300 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:27:44.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.135+0000 7f4fcee69740 1 -- 172.21.15.116:0/2786989459 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630c8ca0120 con 0x5630c8d48710 2024-03-19T01:27:44.137 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.135+0000 7f4fc9819640 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5630c8d48710 0x5630c8d4ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.136+0000 7f4fc9819640 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5630c8d48710 0x5630c8d4ab80 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81d13f6099ed0419 server_cookie=30a07c45adfc31c4 in_seq=0 out_seq=0 2024-03-19T01:27:44.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.136+0000 7f4fadffb640 10 client.26300 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:44.138 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.136+0000 7f4fadffb640 1 -- 172.21.15.116:0/2786989459 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5630c8ca0120 con 0x5630c8d48710 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.136+0000 7f4fadffb640 10 client.26300 handle_command_reply: tid=0 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.136+0000 7f4fcee69740 1 client.26300 shutdown 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.137+0000 7f4fcee69740 1 -- 172.21.15.116:0/2786989459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2e100 msgr2=0x5630c8d3c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.137+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2e100 0x5630c8d3c4b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x5630c8d2f560 tx=0x7f4fc009ae00 comp rx=0 tx=0).stop 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.138+0000 7f4fcee69740 1 -- 172.21.15.116:0/2786989459 shutdown_connections 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.138+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d33c60 0x5630c8d43a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.138+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5630c8d48710 0x5630c8d4ab80 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.138+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d3c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.138+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2786989459 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2e100 0x5630c8d3c4b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.138+0000 7f4fcee69740 1 -- 172.21.15.116:0/2786989459 >> 172.21.15.116:0/2786989459 conn(0x5630c8d29670 msgr2=0x5630c8d2a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.138+0000 7f4fadffb640 0 client.26300 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.138+0000 7f4fcee69740 1 -- 172.21.15.116:0/2786989459 shutdown_connections 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.138+0000 7f4fcee69740 1 -- 172.21.15.116:0/2786989459 wait complete. 2024-03-19T01:27:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.138+0000 7f4fcee69740 20 client.26300 trim_cache size 0 max 16384 2024-03-19T01:27:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.166+0000 7f4fcee69740 1 Processor -- start 2024-03-19T01:27:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.166+0000 7f4fcee69740 1 -- start start 2024-03-19T01:27:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.166+0000 7f4fcee69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d2e100 0x5630c8d36050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.166+0000 7f4fcee69740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2eaf0 0x5630c8d2f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.166+0000 7f4fcee69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d33c60 0x5630c8d2fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.170 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.166+0000 7f4fcee69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc41a6a30 con 0x5630c8d2e100 2024-03-19T01:27:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.166+0000 7f4fcee69740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc41a8570 con 0x5630c8d2eaf0 2024-03-19T01:27:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.166+0000 7f4fcee69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc41a7100 con 0x5630c8d33c60 2024-03-19T01:27:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.166+0000 7f4fca81b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d33c60 0x5630c8d2fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.166+0000 7f4fca81b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d33c60 0x5630c8d2fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52446/0 (socket says 172.21.15.116:52446) 2024-03-19T01:27:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.166+0000 7f4fca81b640 1 -- 172.21.15.116:0/3410685961 learned_addr learned my addr 172.21.15.116:0/3410685961 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.167+0000 7f4fc9819640 1 --2- 172.21.15.116:0/3410685961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2eaf0 0x5630c8d2f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.168+0000 7f4fca81b640 1 -- 172.21.15.116:0/3410685961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2eaf0 msgr2=0x5630c8d2f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.168+0000 7f4fca81b640 1 --2- 172.21.15.116:0/3410685961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2eaf0 0x5630c8d2f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.168+0000 7f4fca81b640 1 -- 172.21.15.116:0/3410685961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d2e100 msgr2=0x5630c8d36050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.168+0000 7f4fca81b640 1 --2- 172.21.15.116:0/3410685961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d2e100 0x5630c8d36050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.171 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.168+0000 7f4fca81b640 1 -- 172.21.15.116:0/3410685961 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630c8a73620 con 0x5630c8d33c60 2024-03-19T01:27:44.179 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.177+0000 7f4fca81b640 1 --2- 172.21.15.116:0/3410685961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d33c60 0x5630c8d2fad0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00f8d0 tx=0x7f4fbc017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cce1fb86dd14617 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.181+0000 7f4faffff640 1 -- 172.21.15.116:0/3410685961 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc009e40 con 0x5630c8d33c60 2024-03-19T01:27:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.181+0000 7f4faffff640 1 -- 172.21.15.116:0/3410685961 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fbc04e070 con 0x5630c8d33c60 2024-03-19T01:27:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.181+0000 7f4faffff640 1 -- 172.21.15.116:0/3410685961 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc049910 con 0x5630c8d33c60 2024-03-19T01:27:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.182+0000 7f4fcee69740 1 -- 172.21.15.116:0/3410685961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d33c60 msgr2=0x5630c8d2fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:44.183 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.182+0000 7f4fcee69740 1 --2- 172.21.15.116:0/3410685961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d33c60 0x5630c8d2fad0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00f8d0 tx=0x7f4fbc017000 comp rx=0 tx=0).stop 2024-03-19T01:27:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.184+0000 7f4fcee69740 1 -- 172.21.15.116:0/3410685961 shutdown_connections 2024-03-19T01:27:44.186 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.184+0000 7f4fcee69740 1 --2- 172.21.15.116:0/3410685961 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d33c60 0x5630c8d2fad0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.184+0000 7f4fcee69740 1 --2- 172.21.15.116:0/3410685961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2eaf0 0x5630c8d2f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.184+0000 7f4fcee69740 1 --2- 172.21.15.116:0/3410685961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d2e100 0x5630c8d36050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.184+0000 7f4fcee69740 1 -- 172.21.15.116:0/3410685961 >> 172.21.15.116:0/3410685961 conn(0x5630c8d29670 msgr2=0x5630c8d28220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:44.187 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.185+0000 7f4fcee69740 1 -- 172.21.15.116:0/3410685961 shutdown_connections 2024-03-19T01:27:44.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.189+0000 7f4fcee69740 1 -- 172.21.15.116:0/3410685961 wait complete. 2024-03-19T01:27:44.190 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.189+0000 7f4fcee69740 1 -- start start 2024-03-19T01:27:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.190+0000 7f4fcee69740 1 Processor -- start 2024-03-19T01:27:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.190+0000 7f4fcee69740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2e100 0x5630c8d3bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.190+0000 7f4fcee69740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d3c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.190+0000 7f4fcee69740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d33c60 0x5630c8da8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.190+0000 7f4fcee69740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc41a7ea0 con 0x5630c8d33c60 2024-03-19T01:27:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.190+0000 7f4fcee69740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc41a77d0 con 0x5630c8d2e100 2024-03-19T01:27:44.191 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.190+0000 7f4fcee69740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc41a4a10 con 0x5630c8d2eaf0 2024-03-19T01:27:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.190+0000 7f4fcee69740 10 client.0 _ll_register_callbacks cb 0x7f4fc0009310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:27:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.191+0000 7f4fc9819640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d3c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.191+0000 7f4fc9819640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d3c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:52462/0 (socket says 172.21.15.116:52462) 2024-03-19T01:27:44.192 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.191+0000 7f4fc9819640 1 -- 172.21.15.116:0/2005426752 learned_addr learned my addr 172.21.15.116:0/2005426752 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.191+0000 7f4fc9819640 1 -- 172.21.15.116:0/2005426752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2e100 msgr2=0x5630c8d3bcd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-19T01:27:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.191+0000 7f4fc9819640 1 --2- 172.21.15.116:0/2005426752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2e100 0x5630c8d3bcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.191+0000 7f4fc9819640 1 -- 172.21.15.116:0/2005426752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d33c60 msgr2=0x5630c8da8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.191+0000 7f4fc9819640 1 --2- 172.21.15.116:0/2005426752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d33c60 0x5630c8da8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.191+0000 7f4fc9819640 1 -- 172.21.15.116:0/2005426752 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630c8d43c50 con 0x5630c8d2eaf0 2024-03-19T01:27:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.191+0000 7f4fc9819640 1 --2- 172.21.15.116:0/2005426752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d3c210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4005940 tx=0x7f4fb40095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=901760029ce56791 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.191+0000 7f4fadffb640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:27:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.191+0000 7f4fadffb640 1 -- 172.21.15.116:0/2005426752 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4057060 con 0x5630c8d2eaf0 2024-03-19T01:27:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.192+0000 7f4fcee69740 1 -- 172.21.15.116:0/2005426752 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fbc04e070 con 0x5630c8d2eaf0 2024-03-19T01:27:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.192+0000 7f4fadffb640 1 -- 172.21.15.116:0/2005426752 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f4fb402cc90 con 0x5630c8d2eaf0 2024-03-19T01:27:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.192+0000 7f4fadffb640 1 -- 172.21.15.116:0/2005426752 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f4fb4004040 con 0x5630c8d2eaf0 2024-03-19T01:27:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.192+0000 7f4fcee69740 10 client.26293 fetch_fsmap learned FSMap version 49 2024-03-19T01:27:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.192+0000 7f4fcee69740 1 -- 172.21.15.116:0/2005426752 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5630c8dbaf70 con 0x5630c8d2eaf0 2024-03-19T01:27:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.192+0000 7f4fadffb640 1 -- 172.21.15.116:0/2005426752 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fbc04e070 con 0x5630c8d2eaf0 2024-03-19T01:27:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.192+0000 7f4fadffb640 1 -- 172.21.15.116:0/2005426752 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f4fb40035f0 con 0x5630c8d2eaf0 2024-03-19T01:27:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.192+0000 7f4fcee69740 10 client.26293 fetch_fsmap finished waiting for FSMap version 49 2024-03-19T01:27:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.192+0000 7f4fcee69740 10 client.26293 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:27:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.192+0000 7f4fcee69740 20 client.26293 populate_metadata read hostname 'smithi116' 2024-03-19T01:27:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.192+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2005426752 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5630c8dbb9b0 0x5630c8dbde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.192+0000 7f4fcee69740 4 client.26293 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:27:44.194 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.192+0000 7f4fcee69740 1 -- 172.21.15.116:0/2005426752 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5630c8c9a470 con 0x5630c8dbb9b0 2024-03-19T01:27:44.199 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.195+0000 7f4fca01a640 1 --2- 172.21.15.116:0/2005426752 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5630c8dbb9b0 0x5630c8dbde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.206+0000 7f4fca01a640 1 --2- 172.21.15.116:0/2005426752 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5630c8dbb9b0 0x5630c8dbde70 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=771b395d5c3b2ff2 server_cookie=c7bc2672841311f7 in_seq=0 out_seq=0 2024-03-19T01:27:44.207 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.207+0000 7f4fadffb640 10 client.26293 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fadffb640 1 -- 172.21.15.116:0/2005426752 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5630c8c9a470 con 0x5630c8dbb9b0 2024-03-19T01:27:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fadffb640 10 client.26293 handle_command_reply: tid=0 2024-03-19T01:27:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fcee69740 1 client.26293 shutdown 2024-03-19T01:27:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fcee69740 1 -- 172.21.15.116:0/2005426752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 msgr2=0x5630c8d3c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2005426752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d3c210 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4005940 tx=0x7f4fb40095f0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.264 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fcee69740 1 -- 172.21.15.116:0/2005426752 shutdown_connections 2024-03-19T01:27:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2005426752 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5630c8d33c60 0x5630c8da8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2005426752 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5630c8dbb9b0 0x5630c8dbde70 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2005426752 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5630c8d2eaf0 0x5630c8d3c210 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fcee69740 1 --2- 172.21.15.116:0/2005426752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5630c8d2e100 0x5630c8d3bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fcee69740 1 -- 172.21.15.116:0/2005426752 >> 172.21.15.116:0/2005426752 conn(0x5630c8d29670 msgr2=0x5630c8d2f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fadffb640 0 client.26293 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fcee69740 1 -- 172.21.15.116:0/2005426752 shutdown_connections 2024-03-19T01:27:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fcee69740 1 -- 172.21.15.116:0/2005426752 wait complete. 2024-03-19T01:27:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.263+0000 7f4fcee69740 20 client.26293 trim_cache size 0 max 16384 2024-03-19T01:27:44.265 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:27:44.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:27:44.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:27:44.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:27:44.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.264+0000 7f4fcc27e640 1 -- 172.21.15.116:0/997040518 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4fb00a2790 msgr2=0x7f4fb00a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:44.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.264+0000 7f4fcc27e640 1 --2- 172.21.15.116:0/997040518 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4fb00a2790 0x7f4fb00a4c50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x5630c8d2f350 tx=0x7f4fbc009500 comp rx=0 tx=0).stop 2024-03-19T01:27:44.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.264+0000 7f4fcc27e640 1 -- 172.21.15.116:0/997040518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc4100740 msgr2=0x7f4fc4197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:44.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.264+0000 7f4fcc27e640 1 --2- 172.21.15.116:0/997040518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc4100740 0x7f4fc4197ad0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000e9c0 tx=0x7f4fc000ee80 comp rx=0 tx=0).stop 2024-03-19T01:27:44.266 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.264+0000 7f4fcc27e640 1 -- 172.21.15.116:0/997040518 shutdown_connections 2024-03-19T01:27:44.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.264+0000 7f4fcc27e640 1 --2- 172.21.15.116:0/997040518 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4fc41022f0 0x7f4fc419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.264+0000 7f4fcc27e640 1 --2- 172.21.15.116:0/997040518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fc4101930 0x7f4fc4198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.264+0000 7f4fcc27e640 1 --2- 172.21.15.116:0/997040518 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4fb00a2790 0x7f4fb00a4c50 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.264+0000 7f4fcc27e640 1 --2- 172.21.15.116:0/997040518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4fc4100740 0x7f4fc4197ad0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:44.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.264+0000 7f4fcc27e640 1 -- 172.21.15.116:0/997040518 >> 172.21.15.116:0/997040518 conn(0x7f4fc40fbcb0 msgr2=0x7f4fc40fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:44.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.265+0000 7f4fcc27e640 1 -- 172.21.15.116:0/997040518 shutdown_connections 2024-03-19T01:27:44.267 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:44.265+0000 7f4fcc27e640 1 -- 172.21.15.116:0/997040518 wait complete. 2024-03-19T01:27:45.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:45 smithi162 ceph-mon[67178]: pgmap v805: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 617 op/s 2024-03-19T01:27:45.654 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:45 smithi116 ceph-mon[62701]: pgmap v805: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 617 op/s 2024-03-19T01:27:45.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:45 smithi156 ceph-mon[66799]: pgmap v805: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 12 MiB/s wr, 617 op/s 2024-03-19T01:27:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:progress: 900.0 s, 1489.0 tps, lat 66.531 ms stddev 79.418 2024-03-19T01:27:46.774 INFO:teuthology.orchestra.run.smithi116.stdout:transaction type: 2024-03-19T01:27:46.774 INFO:teuthology.orchestra.run.smithi116.stdout:scaling factor: 32 2024-03-19T01:27:46.774 INFO:teuthology.orchestra.run.smithi116.stdout:query mode: simple 2024-03-19T01:27:46.775 INFO:teuthology.orchestra.run.smithi116.stdout:number of clients: 100 2024-03-19T01:27:46.775 INFO:teuthology.orchestra.run.smithi116.stdout:number of threads: 4 2024-03-19T01:27:46.775 INFO:teuthology.orchestra.run.smithi116.stdout:duration: 900 s 2024-03-19T01:27:46.775 INFO:teuthology.orchestra.run.smithi116.stdout:number of transactions actually processed: 2106795 2024-03-19T01:27:46.775 INFO:teuthology.orchestra.run.smithi116.stdout:latency average = 42.719 ms 2024-03-19T01:27:46.775 INFO:teuthology.orchestra.run.smithi116.stdout:latency stddev = 54.643 ms 2024-03-19T01:27:46.775 INFO:teuthology.orchestra.run.smithi116.stdout:tps = 2340.282810 (including connections establishing) 2024-03-19T01:27:46.775 INFO:teuthology.orchestra.run.smithi116.stdout:tps = 2340.289052 (excluding connections establishing) 2024-03-19T01:27:46.778 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl stop postgresql' 2024-03-19T01:27:47.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:47 smithi162 ceph-mon[67178]: pgmap v806: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 720 op/s 2024-03-19T01:27:47.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:47 smithi116 ceph-mon[62701]: pgmap v806: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 720 op/s 2024-03-19T01:27:47.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:47 smithi156 ceph-mon[66799]: pgmap v806: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 14 MiB/s wr, 720 op/s 2024-03-19T01:27:49.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:49 smithi162 ceph-mon[67178]: pgmap v807: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 13 MiB/s wr, 667 op/s 2024-03-19T01:27:49.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:49 smithi116 ceph-mon[62701]: pgmap v807: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 13 MiB/s wr, 667 op/s 2024-03-19T01:27:49.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:49 smithi156 ceph-mon[66799]: pgmap v807: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 13 MiB/s wr, 667 op/s 2024-03-19T01:27:51.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:51 smithi162 ceph-mon[67178]: pgmap v808: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 MiB/s wr, 867 op/s 2024-03-19T01:27:51.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:51 smithi116 ceph-mon[62701]: pgmap v808: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 MiB/s wr, 867 op/s 2024-03-19T01:27:51.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:51 smithi156 ceph-mon[66799]: pgmap v808: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 17 MiB/s wr, 867 op/s 2024-03-19T01:27:53.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:53 smithi162 ceph-mon[67178]: pgmap v809: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 31 MiB/s wr, 1.61k op/s 2024-03-19T01:27:53.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:53 smithi116 ceph-mon[62701]: pgmap v809: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 31 MiB/s wr, 1.61k op/s 2024-03-19T01:27:53.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:53 smithi156 ceph-mon[66799]: pgmap v809: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 31 MiB/s wr, 1.61k op/s 2024-03-19T01:27:55.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:55 smithi162 ceph-mon[67178]: pgmap v810: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 32 MiB/s wr, 1.66k op/s 2024-03-19T01:27:55.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:55 smithi116 ceph-mon[62701]: pgmap v810: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 32 MiB/s wr, 1.66k op/s 2024-03-19T01:27:55.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:55 smithi156 ceph-mon[66799]: pgmap v810: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 32 MiB/s wr, 1.66k op/s 2024-03-19T01:27:56.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:56 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:27:56.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:56 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:27:56.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:56 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:27:57.377 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.374+0000 7ff09d177640 1 -- 172.21.15.116:0/1001930197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff098102a60 msgr2=0x7ff098102ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.374+0000 7ff09d177640 1 --2- 172.21.15.116:0/1001930197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff098102a60 0x7ff098102ee0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff08c009900 tx=0x7ff08c02bc30 comp rx=0 tx=0).stop 2024-03-19T01:27:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.374+0000 7ff09d177640 1 -- 172.21.15.116:0/1001930197 shutdown_connections 2024-03-19T01:27:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.374+0000 7ff09d177640 1 --2- 172.21.15.116:0/1001930197 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff098103420 0x7ff098109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.374+0000 7ff09d177640 1 --2- 172.21.15.116:0/1001930197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff098102a60 0x7ff098102ee0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.374+0000 7ff09d177640 1 --2- 172.21.15.116:0/1001930197 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff098101870 0x7ff098101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.375+0000 7ff09d177640 1 -- 172.21.15.116:0/1001930197 >> 172.21.15.116:0/1001930197 conn(0x7ff0980fcde0 msgr2=0x7ff0980ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.375+0000 7ff09d177640 1 -- 172.21.15.116:0/1001930197 shutdown_connections 2024-03-19T01:27:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.375+0000 7ff09d177640 1 -- 172.21.15.116:0/1001930197 wait complete. 2024-03-19T01:27:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.375+0000 7ff09d177640 1 Processor -- start 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff09d177640 1 -- start start 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff09d177640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff098101870 0x7ff098197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff09d177640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff098102a60 0x7ff098198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff09d177640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff098103420 0x7ff09819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff09d177640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff09810d4d0 con 0x7ff098101870 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff09d177640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff09810d350 con 0x7ff098102a60 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff09d177640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff09810d650 con 0x7ff098103420 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff096d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff098101870 0x7ff098197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff096d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff098101870 0x7ff098197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59734/0 (socket says 172.21.15.116:59734) 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff096d9d640 1 -- 172.21.15.116:0/3838597350 learned_addr learned my addr 172.21.15.116:0/3838597350 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff09659c640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff098102a60 0x7ff098198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff09759e640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff098103420 0x7ff09819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff096d9d640 1 -- 172.21.15.116:0/3838597350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff098103420 msgr2=0x7ff09819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:57.379 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff096d9d640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff098103420 0x7ff09819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff096d9d640 1 -- 172.21.15.116:0/3838597350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff098102a60 msgr2=0x7ff098198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff096d9d640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff098102a60 0x7ff098198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.376+0000 7ff096d9d640 1 -- 172.21.15.116:0/3838597350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09819f6e0 con 0x7ff098101870 2024-03-19T01:27:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.377+0000 7ff096d9d640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff098101870 0x7ff098197ac0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff08800d770 tx=0x7ff08800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.377+0000 7ff07ffff640 1 -- 172.21.15.116:0/3838597350 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088017070 con 0x7ff098101870 2024-03-19T01:27:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.377+0000 7ff07ffff640 1 -- 172.21.15.116:0/3838597350 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff088009c30 con 0x7ff098101870 2024-03-19T01:27:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.377+0000 7ff09d177640 1 -- 172.21.15.116:0/3838597350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff09819f9d0 con 0x7ff098101870 2024-03-19T01:27:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.377+0000 7ff07ffff640 1 -- 172.21.15.116:0/3838597350 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff088003dd0 con 0x7ff098101870 2024-03-19T01:27:57.380 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.377+0000 7ff09d177640 1 -- 172.21.15.116:0/3838597350 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff09819ff40 con 0x7ff098101870 2024-03-19T01:27:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff07ffff640 1 -- 172.21.15.116:0/3838597350 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7ff088003f70 con 0x7ff098101870 2024-03-19T01:27:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff09fd66740 1 Processor -- start 2024-03-19T01:27:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff09fd66740 1 -- start start 2024-03-19T01:27:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff09fd66740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92ad520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff09fd66740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92adaf0 0x5601c92b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff09fd66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff09fd66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5601c9211be0 con 0x5601c92b2c60 2024-03-19T01:27:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff09fd66740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5601c91ba960 con 0x5601c92adaf0 2024-03-19T01:27:57.385 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff09fd66740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5601c920dc80 con 0x5601c92ad100 2024-03-19T01:27:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff09759e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92ad520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff09659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff09759e640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92ad520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51138/0 (socket says 172.21.15.116:51138) 2024-03-19T01:27:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff09659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92b50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59744/0 (socket says 172.21.15.116:59744) 2024-03-19T01:27:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff09759e640 1 -- 172.21.15.116:0/2575595377 learned_addr learned my addr 172.21.15.116:0/2575595377 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.380+0000 7ff096d9d640 1 --2- 172.21.15.116:0/2575595377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92adaf0 0x5601c92b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.381+0000 7ff07ffff640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff0800a2920 0x7ff0800a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.381+0000 7ff09759e640 1 -- 172.21.15.116:0/2575595377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92adaf0 msgr2=0x5601c92b25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.381+0000 7ff09759e640 1 --2- 172.21.15.116:0/2575595377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92adaf0 0x5601c92b25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.381+0000 7ff09759e640 1 -- 172.21.15.116:0/2575595377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 msgr2=0x5601c92b50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.381+0000 7ff09759e640 1 --2- 172.21.15.116:0/2575595377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92b50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.381+0000 7ff09759e640 1 -- 172.21.15.116:0/2575595377 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601c91b1d20 con 0x5601c92ad100 2024-03-19T01:27:57.386 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.381+0000 7ff09659c640 1 --2- 172.21.15.116:0/2575595377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.381+0000 7ff07ffff640 1 -- 172.21.15.116:0/3838597350 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7ff0880ce2d0 con 0x7ff098101870 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.381+0000 7ff09759e640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff0800a2920 0x7ff0800a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.381+0000 7ff09759e640 1 --2- 172.21.15.116:0/2575595377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92ad520 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff0900039f0 tx=0x7ff090003eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38acb3b2c1e1482d server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.381+0000 7ff07d7fa640 1 -- 172.21.15.116:0/2575595377 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff090022070 con 0x5601c92ad100 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.381+0000 7ff07d7fa640 1 -- 172.21.15.116:0/2575595377 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff09001cb00 con 0x5601c92ad100 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.381+0000 7ff07d7fa640 1 -- 172.21.15.116:0/2575595377 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff090010a60 con 0x5601c92ad100 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.382+0000 7ff09fd66740 1 -- 172.21.15.116:0/2575595377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 msgr2=0x5601c92ad520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.382+0000 7ff09fd66740 1 --2- 172.21.15.116:0/2575595377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92ad520 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff0900039f0 tx=0x7ff090003eb0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.382+0000 7ff09759e640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff0800a2920 0x7ff0800a4de0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff09001f000 tx=0x7ff090020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.382+0000 7ff09fd66740 1 -- 172.21.15.116:0/2575595377 shutdown_connections 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.382+0000 7ff09fd66740 1 --2- 172.21.15.116:0/2575595377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92b50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.382+0000 7ff09fd66740 1 --2- 172.21.15.116:0/2575595377 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92adaf0 0x5601c92b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.382+0000 7ff09fd66740 1 --2- 172.21.15.116:0/2575595377 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92ad520 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.382+0000 7ff09fd66740 1 -- 172.21.15.116:0/2575595377 >> 172.21.15.116:0/2575595377 conn(0x5601c92a8670 msgr2=0x5601c92a8a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:57.387 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.382+0000 7ff09fd66740 1 -- 172.21.15.116:0/2575595377 shutdown_connections 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.382+0000 7ff09fd66740 1 -- 172.21.15.116:0/2575595377 wait complete. 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.382+0000 7ff09fd66740 1 -- start start 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.382+0000 7ff09fd66740 1 Processor -- start 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09fd66740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92ad100 0x5601c92b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09fd66740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92adaf0 0x5601c92b75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09fd66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92be620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09fd66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x5601c92b5c10 con 0x5601c92b2c60 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09fd66740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5601c91cddb0 con 0x5601c92ad100 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09fd66740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x5601c92b5d90 con 0x5601c92adaf0 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09fd66740 10 client.0 _ll_register_callbacks cb 0x5601c91d2e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92be620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92adaf0 0x5601c92b75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09659c640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92adaf0 0x5601c92b75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.162:3300/0 says I am v2:172.21.15.116:51144/0 (socket says 172.21.15.116:51144) 2024-03-19T01:27:57.388 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09659c640 1 -- 172.21.15.116:0/222583600 learned_addr learned my addr 172.21.15.116:0/222583600 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff096d9d640 1 --2- 172.21.15.116:0/222583600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92ad100 0x5601c92b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09659c640 1 -- 172.21.15.116:0/222583600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92ad100 msgr2=0x5601c92b70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09659c640 1 --2- 172.21.15.116:0/222583600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92ad100 0x5601c92b70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09659c640 1 -- 172.21.15.116:0/222583600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 msgr2=0x5601c92be620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09659c640 1 --2- 172.21.15.116:0/222583600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92be620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.383+0000 7ff09659c640 1 -- 172.21.15.116:0/222583600 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601c91ccc60 con 0x5601c92adaf0 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.384+0000 7ff09659c640 1 --2- 172.21.15.116:0/222583600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92adaf0 0x5601c92b75e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff08c036040 tx=0x7ff08c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=555e7a5705eb4050 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.384+0000 7ff07aff5640 10 client.0 ms_handle_connect on v2:172.21.15.162:3300/0 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.384+0000 7ff07aff5640 1 -- 172.21.15.116:0/222583600 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c005230 con 0x5601c92adaf0 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.384+0000 7ff09fd66740 1 -- 172.21.15.116:0/222583600 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff09001cb00 con 0x5601c92adaf0 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.384+0000 7ff07aff5640 1 -- 172.21.15.116:0/222583600 <== mon.1 v2:172.21.15.162:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7ff08c0053d0 con 0x5601c92adaf0 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.384+0000 7ff07aff5640 1 -- 172.21.15.116:0/222583600 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7ff08c0055d0 con 0x5601c92adaf0 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.384+0000 7ff09fd66740 10 client.26305 fetch_fsmap learned FSMap version 50 2024-03-19T01:27:57.389 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.384+0000 7ff09fd66740 1 -- 172.21.15.116:0/222583600 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x5601c8f26210 con 0x5601c92adaf0 2024-03-19T01:27:57.390 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.384+0000 7ff07aff5640 1 -- 172.21.15.116:0/222583600 <== mon.1 v2:172.21.15.162:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff09001cb00 con 0x5601c92adaf0 2024-03-19T01:27:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.385+0000 7ff07aff5640 1 -- 172.21.15.116:0/222583600 <== mon.1 v2:172.21.15.162:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7ff08c003bf0 con 0x5601c92adaf0 2024-03-19T01:27:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.385+0000 7ff09fd66740 10 client.26305 fetch_fsmap finished waiting for FSMap version 50 2024-03-19T01:27:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.385+0000 7ff09fd66740 10 client.26305 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:27:57.391 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.385+0000 7ff09fd66740 20 client.26305 populate_metadata read hostname 'smithi116' 2024-03-19T01:27:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.385+0000 7ff09fd66740 1 --2- 172.21.15.116:0/222583600 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5601c92c3380 0x5601c92c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.385+0000 7ff09fd66740 4 client.26305 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-19T01:27:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.385+0000 7ff09fd66740 1 -- 172.21.15.116:0/222583600 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601c921f120 con 0x5601c92c3380 2024-03-19T01:27:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.385+0000 7ff096d9d640 1 --2- 172.21.15.116:0/222583600 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5601c92c3380 0x5601c92c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.386+0000 7ff096d9d640 1 --2- 172.21.15.116:0/222583600 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5601c92c3380 0x5601c92c57e0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a8d00f954425182 server_cookie=9ead1ed7f4d5e0be in_seq=0 out_seq=0 2024-03-19T01:27:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.386+0000 7ff07aff5640 10 client.26305 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff07aff5640 1 -- 172.21.15.116:0/222583600 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5601c921f120 con 0x5601c92c3380 2024-03-19T01:27:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff07aff5640 10 client.26305 handle_command_reply: tid=0 2024-03-19T01:27:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff09fd66740 1 client.26305 shutdown 2024-03-19T01:27:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff09fd66740 1 -- 172.21.15.116:0/222583600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92adaf0 msgr2=0x5601c92b75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff09fd66740 1 --2- 172.21.15.116:0/222583600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92adaf0 0x5601c92b75e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff08c036040 tx=0x7ff08c004270 comp rx=0 tx=0).stop 2024-03-19T01:27:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff09fd66740 1 -- 172.21.15.116:0/222583600 shutdown_connections 2024-03-19T01:27:57.392 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff09fd66740 1 --2- 172.21.15.116:0/222583600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92be620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff09fd66740 1 --2- 172.21.15.116:0/222583600 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5601c92c3380 0x5601c92c57e0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff09fd66740 1 --2- 172.21.15.116:0/222583600 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92adaf0 0x5601c92b75e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff09fd66740 1 --2- 172.21.15.116:0/222583600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92ad100 0x5601c92b70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff09fd66740 1 -- 172.21.15.116:0/222583600 >> 172.21.15.116:0/222583600 conn(0x5601c92a8670 msgr2=0x5601c92a98c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff07aff5640 0 client.26305 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff09fd66740 1 -- 172.21.15.116:0/222583600 shutdown_connections 2024-03-19T01:27:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff09fd66740 1 -- 172.21.15.116:0/222583600 wait complete. 2024-03-19T01:27:57.393 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.387+0000 7ff09fd66740 20 client.26305 trim_cache size 0 max 16384 2024-03-19T01:27:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09fd66740 1 Processor -- start 2024-03-19T01:27:57.408 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09fd66740 1 -- start start 2024-03-19T01:27:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09fd66740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92c5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09fd66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92adaf0 0x5601c92be5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09fd66740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92b2c60 0x5601c92a7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09fd66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff0981a6a00 con 0x5601c92adaf0 2024-03-19T01:27:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09fd66740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff0981a8570 con 0x5601c92b2c60 2024-03-19T01:27:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09fd66740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff0981a70d0 con 0x5601c92ad100 2024-03-19T01:27:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92adaf0 0x5601c92be5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92adaf0 0x5601c92be5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59768/0 (socket says 172.21.15.116:59768) 2024-03-19T01:27:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09659c640 1 -- 172.21.15.116:0/4279682583 learned_addr learned my addr 172.21.15.116:0/4279682583 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff096d9d640 1 --2- 172.21.15.116:0/4279682583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09759e640 1 --2- 172.21.15.116:0/4279682583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92b2c60 0x5601c92a7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09659c640 1 -- 172.21.15.116:0/4279682583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 msgr2=0x5601c92c5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:57.409 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09659c640 1 --2- 172.21.15.116:0/4279682583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92c5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09659c640 1 -- 172.21.15.116:0/4279682583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92b2c60 msgr2=0x5601c92a7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.407+0000 7ff09659c640 1 --2- 172.21.15.116:0/4279682583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92b2c60 0x5601c92a7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.408+0000 7ff09659c640 1 -- 172.21.15.116:0/4279682583 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601c8ff2620 con 0x5601c92adaf0 2024-03-19T01:27:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.408+0000 7ff09659c640 1 --2- 172.21.15.116:0/4279682583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92adaf0 0x5601c92be5e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff08c02c170 tx=0x7ff08c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c69c9f0f91050f server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.408+0000 7ff07cff9640 1 -- 172.21.15.116:0/4279682583 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c005770 con 0x5601c92adaf0 2024-03-19T01:27:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.408+0000 7ff07cff9640 1 -- 172.21.15.116:0/4279682583 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff08c041070 con 0x5601c92adaf0 2024-03-19T01:27:57.410 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.408+0000 7ff07cff9640 1 -- 172.21.15.116:0/4279682583 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08c003cc0 con 0x5601c92adaf0 2024-03-19T01:27:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 -- 172.21.15.116:0/4279682583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92adaf0 msgr2=0x5601c92be5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:57.413 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 --2- 172.21.15.116:0/4279682583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92adaf0 0x5601c92be5e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff08c02c170 tx=0x7ff08c02bc30 comp rx=0 tx=0).stop 2024-03-19T01:27:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 -- 172.21.15.116:0/4279682583 shutdown_connections 2024-03-19T01:27:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 --2- 172.21.15.116:0/4279682583 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92b2c60 0x5601c92a7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 --2- 172.21.15.116:0/4279682583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92adaf0 0x5601c92be5e0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 --2- 172.21.15.116:0/4279682583 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92c5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 -- 172.21.15.116:0/4279682583 >> 172.21.15.116:0/4279682583 conn(0x5601c92a8670 msgr2=0x5601c92a7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 -- 172.21.15.116:0/4279682583 shutdown_connections 2024-03-19T01:27:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 -- 172.21.15.116:0/4279682583 wait complete. 2024-03-19T01:27:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 -- start start 2024-03-19T01:27:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 Processor -- start 2024-03-19T01:27:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92bb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92adaf0 0x5601c92bbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92bcc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.414 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff0981a7ea0 con 0x5601c92b2c60 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff0981a77a0 con 0x5601c92adaf0 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7ff0981a4a90 con 0x5601c92ad100 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09fd66740 10 client.0 _ll_register_callbacks cb 0x7ff08c009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92bcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff096d9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92bb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92bcc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59772/0 (socket says 172.21.15.116:59772) 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.409+0000 7ff09759e640 1 -- 172.21.15.116:0/1038842449 learned_addr learned my addr 172.21.15.116:0/1038842449 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff09659c640 1 --2- 172.21.15.116:0/1038842449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92adaf0 0x5601c92bbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff09759e640 1 -- 172.21.15.116:0/1038842449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 msgr2=0x5601c92bb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff09759e640 1 --2- 172.21.15.116:0/1038842449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92bb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff09759e640 1 -- 172.21.15.116:0/1038842449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92adaf0 msgr2=0x5601c92bbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff09759e640 1 --2- 172.21.15.116:0/1038842449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92adaf0 0x5601c92bbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff09759e640 1 -- 172.21.15.116:0/1038842449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601c92bd260 con 0x5601c92b2c60 2024-03-19T01:27:57.415 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff09759e640 1 --2- 172.21.15.116:0/1038842449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92bcc70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x5601c92ae350 tx=0x7ff090010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33aac60c52b65d69 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff07aff5640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff07aff5640 1 -- 172.21.15.116:0/1038842449 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff090036020 con 0x5601c92b2c60 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff07aff5640 1 -- 172.21.15.116:0/1038842449 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7ff090004d30 con 0x5601c92b2c60 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff09fd66740 1 -- 172.21.15.116:0/1038842449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff08c041070 con 0x5601c92b2c60 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff07aff5640 1 -- 172.21.15.116:0/1038842449 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7ff090041050 con 0x5601c92b2c60 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff09fd66740 10 client.16461 fetch_fsmap learned FSMap version 50 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff09fd66740 1 -- 172.21.15.116:0/1038842449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=4+}) -- 0x5601c92be570 con 0x5601c92b2c60 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.410+0000 7ff07aff5640 1 -- 172.21.15.116:0/1038842449 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff08c041070 con 0x5601c92b2c60 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.411+0000 7ff07aff5640 1 -- 172.21.15.116:0/1038842449 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 50) ==== 8358+0+0 (secure 0 0 0) 0x7ff0900327b0 con 0x5601c92b2c60 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.411+0000 7ff09fd66740 10 client.16461 fetch_fsmap finished waiting for FSMap version 50 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.411+0000 7ff09fd66740 10 client.16461 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.411+0000 7ff09fd66740 20 client.16461 populate_metadata read hostname 'smithi116' 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.411+0000 7ff09fd66740 1 --2- 172.21.15.116:0/1038842449 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5601c9329a60 0x5601c932be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:27:57.416 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.411+0000 7ff09fd66740 4 client.16461 mds_command: new command op to 24500 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-19T01:27:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.411+0000 7ff09fd66740 1 -- 172.21.15.116:0/1038842449 --> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5601c9219470 con 0x5601c9329a60 2024-03-19T01:27:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.411+0000 7ff096d9d640 1 --2- 172.21.15.116:0/1038842449 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5601c9329a60 0x5601c932be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:27:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.411+0000 7ff07aff5640 10 client.16461 ms_handle_connect on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:57.417 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.411+0000 7ff096d9d640 1 --2- 172.21.15.116:0/1038842449 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5601c9329a60 0x5601c932be80 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5f7aead57fe0e7f server_cookie=e848aaf5d480cbdd in_seq=0 out_seq=0 2024-03-19T01:27:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff07aff5640 1 -- 172.21.15.116:0/1038842449 <== mds.0 v2:172.21.15.156:6838/1601537427 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5601c9219470 con 0x5601c9329a60 2024-03-19T01:27:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff07aff5640 10 client.16461 handle_command_reply: tid=0 2024-03-19T01:27:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff09fd66740 1 client.16461 shutdown 2024-03-19T01:27:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff09fd66740 1 -- 172.21.15.116:0/1038842449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 msgr2=0x5601c92bcc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff09fd66740 1 --2- 172.21.15.116:0/1038842449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92bcc70 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x5601c92ae350 tx=0x7ff090010f90 comp rx=0 tx=0).stop 2024-03-19T01:27:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff09fd66740 1 -- 172.21.15.116:0/1038842449 shutdown_connections 2024-03-19T01:27:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff09fd66740 1 --2- 172.21.15.116:0/1038842449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x5601c92b2c60 0x5601c92bcc70 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff09fd66740 1 --2- 172.21.15.116:0/1038842449 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5601c92adaf0 0x5601c92bbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff09fd66740 1 --2- 172.21.15.116:0/1038842449 >> [v2:172.21.15.156:6838/1601537427,v1:172.21.15.156:6839/1601537427] conn(0x5601c9329a60 0x5601c932be80 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff09fd66740 1 --2- 172.21.15.116:0/1038842449 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x5601c92ad100 0x5601c92bb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff09fd66740 1 -- 172.21.15.116:0/1038842449 >> 172.21.15.116:0/1038842449 conn(0x5601c92a8670 msgr2=0x5601c92ab8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff07aff5640 0 client.16461 ms_handle_reset on v2:172.21.15.156:6838/1601537427 2024-03-19T01:27:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff09fd66740 1 -- 172.21.15.116:0/1038842449 shutdown_connections 2024-03-19T01:27:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff09fd66740 1 -- 172.21.15.116:0/1038842449 wait complete. 2024-03-19T01:27:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.469+0000 7ff09fd66740 20 client.16461 trim_cache size 0 max 16384 2024-03-19T01:27:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:{ 2024-03-19T01:27:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "message": "", 2024-03-19T01:27:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout: "return_code": 0 2024-03-19T01:27:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stdout:} 2024-03-19T01:27:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.470+0000 7ff09d177640 1 -- 172.21.15.116:0/3838597350 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff0800a2920 msgr2=0x7ff0800a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.470+0000 7ff09d177640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff0800a2920 0x7ff0800a4de0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff09001f000 tx=0x7ff090020000 comp rx=0 tx=0).stop 2024-03-19T01:27:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.470+0000 7ff09d177640 1 -- 172.21.15.116:0/3838597350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff098101870 msgr2=0x7ff098197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:27:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.470+0000 7ff09d177640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff098101870 0x7ff098197ac0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff08800d770 tx=0x7ff08800dc30 comp rx=0 tx=0).stop 2024-03-19T01:27:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.470+0000 7ff09d177640 1 -- 172.21.15.116:0/3838597350 shutdown_connections 2024-03-19T01:27:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.470+0000 7ff09d177640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7ff098103420 0x7ff09819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.470+0000 7ff09d177640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7ff0800a2920 0x7ff0800a4de0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.470+0000 7ff09d177640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff098102a60 0x7ff098198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.470+0000 7ff09d177640 1 --2- 172.21.15.116:0/3838597350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff098101870 0x7ff098197ac0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:27:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.470+0000 7ff09d177640 1 -- 172.21.15.116:0/3838597350 >> 172.21.15.116:0/3838597350 conn(0x7ff0980fcde0 msgr2=0x7ff098105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:27:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.470+0000 7ff09d177640 1 -- 172.21.15.116:0/3838597350 shutdown_connections 2024-03-19T01:27:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi116.front.sepia.ceph.com.smithi116.stderr:2024-03-19T01:27:57.470+0000 7ff09d177640 1 -- 172.21.15.116:0/3838597350 wait complete. 2024-03-19T01:27:57.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:57 smithi162 ceph-mon[67178]: pgmap v811: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 34 MiB/s wr, 1.90k op/s 2024-03-19T01:27:57.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:57 smithi116 ceph-mon[62701]: pgmap v811: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 34 MiB/s wr, 1.90k op/s 2024-03-19T01:27:57.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:57 smithi156 ceph-mon[66799]: pgmap v811: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 34 MiB/s wr, 1.90k op/s 2024-03-19T01:27:58.649 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:27:58 smithi162 ceph-mon[67178]: pgmap v812: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 33 MiB/s wr, 1.88k op/s 2024-03-19T01:27:58.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:27:58 smithi116 ceph-mon[62701]: pgmap v812: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 33 MiB/s wr, 1.88k op/s 2024-03-19T01:27:58.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:27:58 smithi156 ceph-mon[66799]: pgmap v812: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 33 MiB/s wr, 1.88k op/s 2024-03-19T01:27:59.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2024-03-19T01:27:59.863 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/: 2024-03-19T01:27:59.863 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.863 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 . 2024-03-19T01:27:59.864 INFO:teuthology.orchestra.run.smithi116.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 .. 2024-03-19T01:27:59.864 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 data 2024-03-19T01:27:59.864 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.864 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data: 2024-03-19T01:27:59.864 INFO:teuthology.orchestra.run.smithi116.stdout:total 36 2024-03-19T01:27:59.864 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 . 2024-03-19T01:27:59.864 INFO:teuthology.orchestra.run.smithi116.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 .. 2024-03-19T01:27:59.864 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Mar 19 01:12 base 2024-03-19T01:27:59.864 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 30 Mar 19 01:12 current_logfiles 2024-03-19T01:27:59.864 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Mar 19 01:12 global 2024-03-19T01:27:59.864 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 log 2024-03-19T01:27:59.865 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_commit_ts 2024-03-19T01:27:59.865 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_dynshmem 2024-03-19T01:27:59.865 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4516 Mar 19 01:12 pg_hba.conf 2024-03-19T01:27:59.865 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Mar 19 01:12 pg_ident.conf 2024-03-19T01:27:59.865 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:27 pg_logical 2024-03-19T01:27:59.865 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 19 01:12 pg_multixact 2024-03-19T01:27:59.865 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_notify 2024-03-19T01:27:59.865 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_replslot 2024-03-19T01:27:59.865 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_serial 2024-03-19T01:27:59.865 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_snapshots 2024-03-19T01:27:59.865 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:27 pg_stat 2024-03-19T01:27:59.865 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:27 pg_stat_tmp 2024-03-19T01:27:59.866 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:27 pg_subtrans 2024-03-19T01:27:59.866 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_tblspc 2024-03-19T01:27:59.866 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pg_twophase 2024-03-19T01:27:59.866 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 PG_VERSION 2024-03-19T01:27:59.866 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Mar 19 01:27 pg_wal 2024-03-19T01:27:59.866 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:27 pg_xact 2024-03-19T01:27:59.866 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Mar 19 01:12 postgresql.auto.conf 2024-03-19T01:27:59.866 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 28082 Mar 19 01:12 postgresql.conf 2024-03-19T01:27:59.866 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 51 Mar 19 01:12 postmaster.opts 2024-03-19T01:27:59.866 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.866 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/base: 2024-03-19T01:27:59.866 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.867 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Mar 19 01:12 . 2024-03-19T01:27:59.867 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.867 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Mar 19 01:12 1 2024-03-19T01:27:59.867 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Mar 19 01:12 13493 2024-03-19T01:27:59.867 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Mar 19 01:15 13494 2024-03-19T01:27:59.867 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 pgsql_tmp 2024-03-19T01:27:59.867 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.867 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/base/1: 2024-03-19T01:27:59.867 INFO:teuthology.orchestra.run.smithi116.stdout:total 7777 2024-03-19T01:27:59.867 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Mar 19 01:12 . 2024-03-19T01:27:59.867 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Mar 19 01:12 .. 2024-03-19T01:27:59.867 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 112 2024-03-19T01:27:59.868 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 113 2024-03-19T01:27:59.868 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 19 01:12 1247 2024-03-19T01:27:59.868 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1247_fsm 2024-03-19T01:27:59.868 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1247_vm 2024-03-19T01:27:59.868 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Mar 19 01:12 1249 2024-03-19T01:27:59.868 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1249_fsm 2024-03-19T01:27:59.868 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1249_vm 2024-03-19T01:27:59.868 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Mar 19 01:12 1255 2024-03-19T01:27:59.868 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1255_fsm 2024-03-19T01:27:59.868 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1255_vm 2024-03-19T01:27:59.868 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 19 01:12 1259 2024-03-19T01:27:59.869 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1259_fsm 2024-03-19T01:27:59.869 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1259_vm 2024-03-19T01:27:59.869 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Mar 19 01:12 13344 2024-03-19T01:27:59.869 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13344_fsm 2024-03-19T01:27:59.869 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13344_vm 2024-03-19T01:27:59.869 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13346 2024-03-19T01:27:59.869 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13348 2024-03-19T01:27:59.869 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13349 2024-03-19T01:27:59.869 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13349_fsm 2024-03-19T01:27:59.869 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13349_vm 2024-03-19T01:27:59.869 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13351 2024-03-19T01:27:59.869 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13353 2024-03-19T01:27:59.869 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13354 2024-03-19T01:27:59.870 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13354_fsm 2024-03-19T01:27:59.870 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13354_vm 2024-03-19T01:27:59.870 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13356 2024-03-19T01:27:59.870 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13358 2024-03-19T01:27:59.870 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13359 2024-03-19T01:27:59.870 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13359_fsm 2024-03-19T01:27:59.870 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13359_vm 2024-03-19T01:27:59.870 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13361 2024-03-19T01:27:59.870 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13363 2024-03-19T01:27:59.870 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 1417 2024-03-19T01:27:59.870 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 1418 2024-03-19T01:27:59.870 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 174 2024-03-19T01:27:59.870 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 175 2024-03-19T01:27:59.871 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2187 2024-03-19T01:27:59.871 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2224 2024-03-19T01:27:59.871 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2328 2024-03-19T01:27:59.871 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2336 2024-03-19T01:27:59.871 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2337 2024-03-19T01:27:59.871 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2579 2024-03-19T01:27:59.871 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2600 2024-03-19T01:27:59.871 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2600_fsm 2024-03-19T01:27:59.871 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2600_vm 2024-03-19T01:27:59.871 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2601 2024-03-19T01:27:59.871 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2601_fsm 2024-03-19T01:27:59.871 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2601_vm 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2602 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2602_fsm 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2602_vm 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2603 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2603_fsm 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2603_vm 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2604 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2605 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2605_fsm 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2605_vm 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2606 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2606_fsm 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2606_vm 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2607 2024-03-19T01:27:59.872 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2607_fsm 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2607_vm 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Mar 19 01:12 2608 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2608_fsm 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2608_vm 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Mar 19 01:12 2609 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2609_fsm 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2609_vm 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2610 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2610_fsm 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2610_vm 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2611 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2612 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2612_fsm 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2612_vm 2024-03-19T01:27:59.873 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2613 2024-03-19T01:27:59.874 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2615 2024-03-19T01:27:59.874 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2615_fsm 2024-03-19T01:27:59.874 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2615_vm 2024-03-19T01:27:59.874 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2616 2024-03-19T01:27:59.874 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2616_fsm 2024-03-19T01:27:59.874 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2616_vm 2024-03-19T01:27:59.874 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Mar 19 01:12 2617 2024-03-19T01:27:59.874 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2617_fsm 2024-03-19T01:27:59.874 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2617_vm 2024-03-19T01:27:59.874 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 19 01:12 2618 2024-03-19T01:27:59.874 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2618_fsm 2024-03-19T01:27:59.874 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2618_vm 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Mar 19 01:12 2619 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2619_fsm 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2619_vm 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2620 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2650 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2651 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2652 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2653 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2654 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2655 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2656 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2657 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 19 01:12 2658 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 19 01:12 2659 2024-03-19T01:27:59.875 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2660 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2661 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2662 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2663 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2664 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2665 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2666 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2667 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2668 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2669 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2670 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Mar 19 01:12 2673 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Mar 19 01:12 2674 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Mar 19 01:12 2675 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2678 2024-03-19T01:27:59.876 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2679 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2680 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2681 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2682 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2683 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2684 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2685 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2686 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2687 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2688 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2689 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 19 01:12 2690 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Mar 19 01:12 2691 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2692 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2693 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2696 2024-03-19T01:27:59.877 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2699 2024-03-19T01:27:59.878 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2701 2024-03-19T01:27:59.878 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2702 2024-03-19T01:27:59.878 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2703 2024-03-19T01:27:59.878 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2704 2024-03-19T01:27:59.878 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2753 2024-03-19T01:27:59.878 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2753_fsm 2024-03-19T01:27:59.878 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2753_vm 2024-03-19T01:27:59.878 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2754 2024-03-19T01:27:59.878 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2755 2024-03-19T01:27:59.878 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2756 2024-03-19T01:27:59.878 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2757 2024-03-19T01:27:59.878 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2830 2024-03-19T01:27:59.879 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2831 2024-03-19T01:27:59.879 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2832 2024-03-19T01:27:59.879 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2833 2024-03-19T01:27:59.879 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2834 2024-03-19T01:27:59.879 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2835 2024-03-19T01:27:59.879 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2836 2024-03-19T01:27:59.879 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2837 2024-03-19T01:27:59.879 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Mar 19 01:12 2838 2024-03-19T01:27:59.879 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2838_fsm 2024-03-19T01:27:59.879 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2838_vm 2024-03-19T01:27:59.879 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2839 2024-03-19T01:27:59.879 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2840 2024-03-19T01:27:59.879 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2840_fsm 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2840_vm 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2841 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2995 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2996 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3079 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3079_fsm 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3079_vm 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3080 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3081 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 3085 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3118 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3119 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 19 01:12 3164 2024-03-19T01:27:59.880 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3256 2024-03-19T01:27:59.881 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3257 2024-03-19T01:27:59.881 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3258 2024-03-19T01:27:59.881 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3350 2024-03-19T01:27:59.881 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3351 2024-03-19T01:27:59.881 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3379 2024-03-19T01:27:59.881 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3380 2024-03-19T01:27:59.881 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3381 2024-03-19T01:27:59.881 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3394 2024-03-19T01:27:59.881 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3394_fsm 2024-03-19T01:27:59.881 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3394_vm 2024-03-19T01:27:59.881 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3395 2024-03-19T01:27:59.881 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3429 2024-03-19T01:27:59.882 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3430 2024-03-19T01:27:59.882 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3431 2024-03-19T01:27:59.882 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3433 2024-03-19T01:27:59.882 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3439 2024-03-19T01:27:59.882 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3440 2024-03-19T01:27:59.882 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 3455 2024-03-19T01:27:59.882 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Mar 19 01:12 3456 2024-03-19T01:27:59.882 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3456_fsm 2024-03-19T01:27:59.882 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3456_vm 2024-03-19T01:27:59.882 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3466 2024-03-19T01:27:59.882 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3467 2024-03-19T01:27:59.882 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3468 2024-03-19T01:27:59.882 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3501 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3502 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3503 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3534 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3541 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3541_fsm 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3541_vm 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3542 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3574 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3575 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3576 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3596 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3597 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3598 2024-03-19T01:27:59.883 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3599 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3600 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3600_fsm 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3600_vm 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3601 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3601_fsm 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3601_vm 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3602 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3602_fsm 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3602_vm 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3603 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3603_fsm 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3603_vm 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3604 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3605 2024-03-19T01:27:59.884 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3606 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3607 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3608 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 3609 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3712 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3764 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3764_fsm 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3764_vm 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3766 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3767 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3997 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4143 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4144 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4145 2024-03-19T01:27:59.885 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4146 2024-03-19T01:27:59.886 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4147 2024-03-19T01:27:59.886 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4148 2024-03-19T01:27:59.886 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4149 2024-03-19T01:27:59.886 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4150 2024-03-19T01:27:59.886 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4151 2024-03-19T01:27:59.886 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4152 2024-03-19T01:27:59.886 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4153 2024-03-19T01:27:59.886 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4154 2024-03-19T01:27:59.886 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4155 2024-03-19T01:27:59.886 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4156 2024-03-19T01:27:59.886 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4157 2024-03-19T01:27:59.886 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4158 2024-03-19T01:27:59.887 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4159 2024-03-19T01:27:59.887 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4160 2024-03-19T01:27:59.887 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4161 2024-03-19T01:27:59.887 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4162 2024-03-19T01:27:59.888 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4163 2024-03-19T01:27:59.888 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4164 2024-03-19T01:27:59.888 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4165 2024-03-19T01:27:59.888 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4166 2024-03-19T01:27:59.888 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4167 2024-03-19T01:27:59.888 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4168 2024-03-19T01:27:59.888 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4169 2024-03-19T01:27:59.888 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4170 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4171 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4172 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4173 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4174 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 5002 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 548 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 549 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6102 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6104 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6106 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6110 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6111 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6112 2024-03-19T01:27:59.889 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6113 2024-03-19T01:27:59.890 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6117 2024-03-19T01:27:59.890 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 826 2024-03-19T01:27:59.890 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 827 2024-03-19T01:27:59.890 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 828 2024-03-19T01:27:59.890 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 19 01:12 pg_filenode.map 2024-03-19T01:27:59.890 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 PG_VERSION 2024-03-19T01:27:59.890 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.890 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/base/13493: 2024-03-19T01:27:59.890 INFO:teuthology.orchestra.run.smithi116.stdout:total 7777 2024-03-19T01:27:59.890 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 289 Mar 19 01:12 . 2024-03-19T01:27:59.890 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Mar 19 01:12 .. 2024-03-19T01:27:59.890 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 112 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 113 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 19 01:12 1247 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1247_fsm 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1247_vm 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Mar 19 01:12 1249 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1249_fsm 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1249_vm 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Mar 19 01:12 1255 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1255_fsm 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1255_vm 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 19 01:12 1259 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1259_fsm 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1259_vm 2024-03-19T01:27:59.891 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Mar 19 01:12 13344 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13344_fsm 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13344_vm 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13346 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13348 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13349 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13349_fsm 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13349_vm 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13351 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13353 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13354 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13354_fsm 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13354_vm 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13356 2024-03-19T01:27:59.892 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13358 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13359 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13359_fsm 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13359_vm 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13361 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13363 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 1417 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 1418 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 174 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 175 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2187 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2224 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2328 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2336 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2337 2024-03-19T01:27:59.893 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2579 2024-03-19T01:27:59.894 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2600 2024-03-19T01:27:59.894 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2600_fsm 2024-03-19T01:27:59.894 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2600_vm 2024-03-19T01:27:59.894 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2601 2024-03-19T01:27:59.894 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2601_fsm 2024-03-19T01:27:59.894 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2601_vm 2024-03-19T01:27:59.894 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2602 2024-03-19T01:27:59.894 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2602_fsm 2024-03-19T01:27:59.894 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2602_vm 2024-03-19T01:27:59.894 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2603 2024-03-19T01:27:59.894 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2603_fsm 2024-03-19T01:27:59.894 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2603_vm 2024-03-19T01:27:59.894 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2604 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2605 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2605_fsm 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2605_vm 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2606 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2606_fsm 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2606_vm 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2607 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2607_fsm 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2607_vm 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Mar 19 01:12 2608 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2608_fsm 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2608_vm 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Mar 19 01:12 2609 2024-03-19T01:27:59.895 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2609_fsm 2024-03-19T01:27:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2609_vm 2024-03-19T01:27:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2610 2024-03-19T01:27:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2610_fsm 2024-03-19T01:27:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2610_vm 2024-03-19T01:27:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2611 2024-03-19T01:27:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2612 2024-03-19T01:27:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2612_fsm 2024-03-19T01:27:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2612_vm 2024-03-19T01:27:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2613 2024-03-19T01:27:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2615 2024-03-19T01:27:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2615_fsm 2024-03-19T01:27:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2615_vm 2024-03-19T01:27:59.896 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2616 2024-03-19T01:27:59.897 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2616_fsm 2024-03-19T01:27:59.897 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2616_vm 2024-03-19T01:27:59.897 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Mar 19 01:12 2617 2024-03-19T01:27:59.897 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2617_fsm 2024-03-19T01:27:59.897 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2617_vm 2024-03-19T01:27:59.897 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 19 01:12 2618 2024-03-19T01:27:59.897 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2618_fsm 2024-03-19T01:27:59.897 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2618_vm 2024-03-19T01:27:59.897 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 147456 Mar 19 01:12 2619 2024-03-19T01:27:59.897 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2619_fsm 2024-03-19T01:27:59.897 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2619_vm 2024-03-19T01:27:59.897 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2620 2024-03-19T01:27:59.897 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2650 2024-03-19T01:27:59.898 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2651 2024-03-19T01:27:59.898 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2652 2024-03-19T01:27:59.898 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2653 2024-03-19T01:27:59.898 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2654 2024-03-19T01:27:59.898 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2655 2024-03-19T01:27:59.898 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2656 2024-03-19T01:27:59.898 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2657 2024-03-19T01:27:59.898 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 19 01:12 2658 2024-03-19T01:27:59.898 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 19 01:12 2659 2024-03-19T01:27:59.898 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2660 2024-03-19T01:27:59.898 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2661 2024-03-19T01:27:59.898 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2662 2024-03-19T01:27:59.899 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2663 2024-03-19T01:27:59.899 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2664 2024-03-19T01:27:59.899 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2665 2024-03-19T01:27:59.899 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2666 2024-03-19T01:27:59.899 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2667 2024-03-19T01:27:59.899 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2668 2024-03-19T01:27:59.899 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2669 2024-03-19T01:27:59.899 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2670 2024-03-19T01:27:59.899 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Mar 19 01:12 2673 2024-03-19T01:27:59.899 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Mar 19 01:12 2674 2024-03-19T01:27:59.899 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Mar 19 01:12 2675 2024-03-19T01:27:59.899 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2678 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2679 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2680 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2681 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2682 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2683 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2684 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2685 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2686 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2687 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2688 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2689 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 19 01:12 2690 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Mar 19 01:12 2691 2024-03-19T01:27:59.900 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2692 2024-03-19T01:27:59.901 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2693 2024-03-19T01:27:59.901 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2696 2024-03-19T01:27:59.901 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2699 2024-03-19T01:27:59.901 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2701 2024-03-19T01:27:59.901 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2702 2024-03-19T01:27:59.901 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2703 2024-03-19T01:27:59.901 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2704 2024-03-19T01:27:59.901 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2753 2024-03-19T01:27:59.901 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2753_fsm 2024-03-19T01:27:59.901 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2753_vm 2024-03-19T01:27:59.901 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2754 2024-03-19T01:27:59.901 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2755 2024-03-19T01:27:59.901 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2756 2024-03-19T01:27:59.902 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2757 2024-03-19T01:27:59.902 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2830 2024-03-19T01:27:59.902 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2831 2024-03-19T01:27:59.902 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2832 2024-03-19T01:27:59.902 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2833 2024-03-19T01:27:59.902 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2834 2024-03-19T01:27:59.902 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2835 2024-03-19T01:27:59.902 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2836 2024-03-19T01:27:59.902 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2837 2024-03-19T01:27:59.902 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Mar 19 01:12 2838 2024-03-19T01:27:59.902 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2838_fsm 2024-03-19T01:27:59.902 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2838_vm 2024-03-19T01:27:59.903 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2839 2024-03-19T01:27:59.903 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2840 2024-03-19T01:27:59.903 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2840_fsm 2024-03-19T01:27:59.903 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2840_vm 2024-03-19T01:27:59.903 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2841 2024-03-19T01:27:59.903 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2995 2024-03-19T01:27:59.903 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2996 2024-03-19T01:27:59.903 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3079 2024-03-19T01:27:59.903 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3079_fsm 2024-03-19T01:27:59.903 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3079_vm 2024-03-19T01:27:59.903 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3080 2024-03-19T01:27:59.903 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3081 2024-03-19T01:27:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 3085 2024-03-19T01:27:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3118 2024-03-19T01:27:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3119 2024-03-19T01:27:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 19 01:12 3164 2024-03-19T01:27:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3256 2024-03-19T01:27:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3257 2024-03-19T01:27:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3258 2024-03-19T01:27:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3350 2024-03-19T01:27:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3351 2024-03-19T01:27:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3379 2024-03-19T01:27:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3380 2024-03-19T01:27:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3381 2024-03-19T01:27:59.904 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3394 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3394_fsm 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3394_vm 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3395 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3429 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3430 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3431 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3433 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3439 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3440 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 3455 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Mar 19 01:12 3456 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3456_fsm 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3456_vm 2024-03-19T01:27:59.905 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3466 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3467 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3468 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3501 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3502 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3503 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3534 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3541 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3541_fsm 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3541_vm 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3542 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3574 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3575 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3576 2024-03-19T01:27:59.906 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3596 2024-03-19T01:27:59.907 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3597 2024-03-19T01:27:59.907 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3598 2024-03-19T01:27:59.907 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3599 2024-03-19T01:27:59.907 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3600 2024-03-19T01:27:59.907 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3600_fsm 2024-03-19T01:27:59.907 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3600_vm 2024-03-19T01:27:59.907 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3601 2024-03-19T01:27:59.907 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3601_fsm 2024-03-19T01:27:59.907 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3601_vm 2024-03-19T01:27:59.907 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3602 2024-03-19T01:27:59.907 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3602_fsm 2024-03-19T01:27:59.907 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3602_vm 2024-03-19T01:27:59.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3603 2024-03-19T01:27:59.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3603_fsm 2024-03-19T01:27:59.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3603_vm 2024-03-19T01:27:59.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3604 2024-03-19T01:27:59.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3605 2024-03-19T01:27:59.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3606 2024-03-19T01:27:59.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3607 2024-03-19T01:27:59.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3608 2024-03-19T01:27:59.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 3609 2024-03-19T01:27:59.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3712 2024-03-19T01:27:59.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3764 2024-03-19T01:27:59.908 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3764_fsm 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3764_vm 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3766 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3767 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3997 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4143 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4144 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4145 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4146 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4147 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4148 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4149 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4150 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4151 2024-03-19T01:27:59.909 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4152 2024-03-19T01:27:59.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4153 2024-03-19T01:27:59.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4154 2024-03-19T01:27:59.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4155 2024-03-19T01:27:59.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4156 2024-03-19T01:27:59.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4157 2024-03-19T01:27:59.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4158 2024-03-19T01:27:59.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4159 2024-03-19T01:27:59.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4160 2024-03-19T01:27:59.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4161 2024-03-19T01:27:59.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4162 2024-03-19T01:27:59.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4163 2024-03-19T01:27:59.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4164 2024-03-19T01:27:59.910 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4165 2024-03-19T01:27:59.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4166 2024-03-19T01:27:59.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4167 2024-03-19T01:27:59.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4168 2024-03-19T01:27:59.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4169 2024-03-19T01:27:59.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4170 2024-03-19T01:27:59.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4171 2024-03-19T01:27:59.911 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4172 2024-03-19T01:27:59.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4173 2024-03-19T01:27:59.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4174 2024-03-19T01:27:59.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 5002 2024-03-19T01:27:59.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 548 2024-03-19T01:27:59.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 549 2024-03-19T01:27:59.913 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6102 2024-03-19T01:27:59.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6104 2024-03-19T01:27:59.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6106 2024-03-19T01:27:59.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6110 2024-03-19T01:27:59.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6111 2024-03-19T01:27:59.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6112 2024-03-19T01:27:59.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6113 2024-03-19T01:27:59.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6117 2024-03-19T01:27:59.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 826 2024-03-19T01:27:59.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 827 2024-03-19T01:27:59.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 828 2024-03-19T01:27:59.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 19 01:12 pg_filenode.map 2024-03-19T01:27:59.914 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 PG_VERSION 2024-03-19T01:27:59.914 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.915 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/base/13494: 2024-03-19T01:27:59.915 INFO:teuthology.orchestra.run.smithi116.stdout:total 624318 2024-03-19T01:27:59.915 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 305 Mar 19 01:15 . 2024-03-19T01:27:59.915 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Mar 19 01:12 .. 2024-03-19T01:27:59.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 112 2024-03-19T01:27:59.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 113 2024-03-19T01:27:59.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 19 01:12 1247 2024-03-19T01:27:59.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1247_fsm 2024-03-19T01:27:59.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1247_vm 2024-03-19T01:27:59.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 434176 Mar 19 01:12 1249 2024-03-19T01:27:59.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1249_fsm 2024-03-19T01:27:59.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1249_vm 2024-03-19T01:27:59.915 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 663552 Mar 19 01:12 1255 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1255_fsm 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1255_vm 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 19 01:27 1259 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1259_fsm 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1259_vm 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 65536 Mar 19 01:12 13344 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13344_fsm 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13344_vm 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13346 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13348 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13349 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13349_fsm 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13349_vm 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13351 2024-03-19T01:27:59.916 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13353 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13354 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13354_fsm 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13354_vm 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13356 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13358 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13359 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 13359_fsm 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13359_vm 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 13361 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 13363 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 1417 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 1418 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 445980672 Mar 19 01:27 16396 2024-03-19T01:27:59.917 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Mar 19 01:27 16396_fsm 2024-03-19T01:27:59.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:15 16396_vm 2024-03-19T01:27:59.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1294336 Mar 19 01:27 16397 2024-03-19T01:27:59.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:27 16397_fsm 2024-03-19T01:27:59.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:27 16397_vm 2024-03-19T01:27:59.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1449984 Mar 19 01:27 16399 2024-03-19T01:27:59.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:27 16399_fsm 2024-03-19T01:27:59.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:27 16399_vm 2024-03-19T01:27:59.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:27 16400 2024-03-19T01:27:59.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 19 01:27 16402 2024-03-19T01:27:59.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 71901184 Mar 19 01:27 16404 2024-03-19T01:27:59.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 110174208 Mar 19 01:27 16406 2024-03-19T01:27:59.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:27 16406_fsm 2024-03-19T01:27:59.918 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:23 16406_vm 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 174 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 175 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2187 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2224 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2328 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2336 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2337 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2579 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2600 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2600_fsm 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2600_vm 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2601 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2601_fsm 2024-03-19T01:27:59.919 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2601_vm 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2602 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2602_fsm 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2602_vm 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2603 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2603_fsm 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2603_vm 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2604 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2605 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2605_fsm 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2605_vm 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2606 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2606_fsm 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2606_vm 2024-03-19T01:27:59.920 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2607 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2607_fsm 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2607_vm 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 466944 Mar 19 01:12 2608 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2608_fsm 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2608_vm 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 344064 Mar 19 01:12 2609 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2609_fsm 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2609_vm 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2610 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2610_fsm 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2610_vm 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2611 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2612 2024-03-19T01:27:59.921 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2612_fsm 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2612_vm 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2613 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2615 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2615_fsm 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2615_vm 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2616 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2616_fsm 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2616_vm 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 114688 Mar 19 01:12 2617 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2617_fsm 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2617_vm 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 19 01:12 2618 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2618_fsm 2024-03-19T01:27:59.922 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2618_vm 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 172032 Mar 19 01:27 2619 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:14 2619_fsm 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2619_vm 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2620 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2650 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2651 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2652 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2653 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2654 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2655 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2656 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2657 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 19 01:12 2658 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 19 01:12 2659 2024-03-19T01:27:59.923 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2660 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2661 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2662 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2663 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2664 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2665 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2666 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2667 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2668 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2669 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2670 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 278528 Mar 19 01:12 2673 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 352256 Mar 19 01:12 2674 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 196608 Mar 19 01:12 2675 2024-03-19T01:27:59.924 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2678 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2679 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2680 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2681 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2682 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2683 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2684 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2685 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2686 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2687 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2688 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 19 01:12 2689 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 19 01:12 2690 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 253952 Mar 19 01:12 2691 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2692 2024-03-19T01:27:59.925 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2693 2024-03-19T01:27:59.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:17 2696 2024-03-19T01:27:59.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2699 2024-03-19T01:27:59.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2701 2024-03-19T01:27:59.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2702 2024-03-19T01:27:59.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2703 2024-03-19T01:27:59.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 2704 2024-03-19T01:27:59.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2753 2024-03-19T01:27:59.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2753_fsm 2024-03-19T01:27:59.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2753_vm 2024-03-19T01:27:59.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2754 2024-03-19T01:27:59.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2755 2024-03-19T01:27:59.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2756 2024-03-19T01:27:59.926 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 2757 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2830 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2831 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2832 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2833 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2834 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2835 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2836 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2837 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 491520 Mar 19 01:12 2838 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2838_fsm 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2838_vm 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2839 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2840 2024-03-19T01:27:59.927 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2840_fsm 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2840_vm 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2841 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2995 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2996 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3079 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3079_fsm 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3079_vm 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3080 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3081 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 3085 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3118 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3119 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 19 01:12 3164 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3256 2024-03-19T01:27:59.928 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3257 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3258 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3350 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3351 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3379 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3380 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3381 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3394 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3394_fsm 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3394_vm 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3395 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3429 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3430 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3431 2024-03-19T01:27:59.929 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3433 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3439 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3440 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:12 3455 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 221184 Mar 19 01:12 3456 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3456_fsm 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3456_vm 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3466 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3467 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3468 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3501 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3502 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3503 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3534 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3541 2024-03-19T01:27:59.930 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3541_fsm 2024-03-19T01:27:59.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3541_vm 2024-03-19T01:27:59.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3542 2024-03-19T01:27:59.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3574 2024-03-19T01:27:59.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3575 2024-03-19T01:27:59.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3576 2024-03-19T01:27:59.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3596 2024-03-19T01:27:59.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3597 2024-03-19T01:27:59.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3598 2024-03-19T01:27:59.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3599 2024-03-19T01:27:59.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3600 2024-03-19T01:27:59.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3600_fsm 2024-03-19T01:27:59.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3600_vm 2024-03-19T01:27:59.931 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3601 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3601_fsm 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3601_vm 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3602 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3602_fsm 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3602_vm 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3603 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3603_fsm 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3603_vm 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3604 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3605 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3606 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3607 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3608 2024-03-19T01:27:59.932 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 19 01:12 3609 2024-03-19T01:27:59.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3712 2024-03-19T01:27:59.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3764 2024-03-19T01:27:59.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 3764_fsm 2024-03-19T01:27:59.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3764_vm 2024-03-19T01:27:59.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3766 2024-03-19T01:27:59.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 3767 2024-03-19T01:27:59.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3997 2024-03-19T01:27:59.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4143 2024-03-19T01:27:59.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4144 2024-03-19T01:27:59.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4145 2024-03-19T01:27:59.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4146 2024-03-19T01:27:59.933 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4147 2024-03-19T01:27:59.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4148 2024-03-19T01:27:59.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4149 2024-03-19T01:27:59.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4150 2024-03-19T01:27:59.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4151 2024-03-19T01:27:59.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4152 2024-03-19T01:27:59.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4153 2024-03-19T01:27:59.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4154 2024-03-19T01:27:59.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4155 2024-03-19T01:27:59.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4156 2024-03-19T01:27:59.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4157 2024-03-19T01:27:59.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4158 2024-03-19T01:27:59.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4159 2024-03-19T01:27:59.934 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4160 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4161 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4162 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4163 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4164 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4165 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4166 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4167 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4168 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4169 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4170 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4171 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4172 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4173 2024-03-19T01:27:59.935 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4174 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 5002 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 548 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 549 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6102 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6104 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6106 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6110 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6111 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6112 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6113 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6117 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 826 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 827 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 828 2024-03-19T01:27:59.936 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 19 01:12 pg_filenode.map 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 151596 Mar 19 01:12 pg_internal.init 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:12 PG_VERSION 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/base/pgsql_tmp: 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 6 postgres postgres system_u:object_r:postgresql_db_t:s0 4 Mar 19 01:12 .. 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/global: 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout:total 560 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Mar 19 01:12 . 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1213 2024-03-19T01:27:59.937 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1213_fsm 2024-03-19T01:27:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1213_vm 2024-03-19T01:27:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1214 2024-03-19T01:27:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1214_fsm 2024-03-19T01:27:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1214_vm 2024-03-19T01:27:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 1232 2024-03-19T01:27:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 1233 2024-03-19T01:27:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1260 2024-03-19T01:27:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1260_fsm 2024-03-19T01:27:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1260_vm 2024-03-19T01:27:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1261 2024-03-19T01:27:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1261_fsm 2024-03-19T01:27:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1261_vm 2024-03-19T01:27:59.938 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1262 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 1262_fsm 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 1262_vm 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2396 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 19 01:12 2396_fsm 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2396_vm 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2397 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2671 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2672 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2676 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2677 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2694 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2695 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2697 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 19 01:12 2698 2024-03-19T01:27:59.939 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2846 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2847 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2964 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2965 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 2966 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 2967 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 3592 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 3593 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4060 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4061 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4175 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4176 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4177 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4178 2024-03-19T01:27:59.940 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4181 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4182 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4183 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4184 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 4185 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 4186 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6000 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6001 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6002 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 6100 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6114 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 6115 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:27 pg_control 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 19 01:12 pg_filenode.map 2024-03-19T01:27:59.941 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 23144 Mar 19 01:12 pg_internal.init 2024-03-19T01:27:59.942 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.942 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/log: 2024-03-19T01:27:59.942 INFO:teuthology.orchestra.run.smithi116.stdout:total 2 2024-03-19T01:27:59.942 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 . 2024-03-19T01:27:59.942 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.942 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1131 Mar 19 01:27 postgresql-Tue.log 2024-03-19T01:27:59.942 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.942 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2024-03-19T01:27:59.942 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.942 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:27:59.942 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.942 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.942 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_logical: 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout:total 1 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:27 . 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 mappings 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Mar 19 01:27 replorigin_checkpoint 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 snapshots 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2024-03-19T01:27:59.943 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:27 .. 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:27 .. 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 19 01:12 . 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 members 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 offsets 2024-03-19T01:27:59.944 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout:total 8 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 . 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 19 01:12 .. 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:12 0000 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout:total 8 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:12 . 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 19 01:12 .. 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:15 0000 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_notify: 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.945 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:27:59.946 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.946 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.946 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2024-03-19T01:27:59.946 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.946 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:27:59.946 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.946 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.946 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_serial: 2024-03-19T01:27:59.946 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.946 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:27:59.946 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.946 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.946 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2024-03-19T01:27:59.947 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.947 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:27:59.947 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.947 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.947 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_stat: 2024-03-19T01:27:59.947 INFO:teuthology.orchestra.run.smithi116.stdout:total 15 2024-03-19T01:27:59.947 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:27 . 2024-03-19T01:27:59.947 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.947 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1952 Mar 19 01:27 db_0.stat 2024-03-19T01:27:59.947 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 10979 Mar 19 01:27 db_13494.stat 2024-03-19T01:27:59.947 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1151 Mar 19 01:27 global.stat 2024-03-19T01:27:59.947 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.947 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:27 . 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout:total 40 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 19 01:27 . 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 19 01:27 0020 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:27:59.948 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_wal: 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout:total 1048576 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Mar 19 01:27 . 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000BA 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:21 0000000100000000000000BB 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:21 0000000100000000000000BC 2024-03-19T01:27:59.949 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:20 0000000100000000000000BD 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:18 0000000100000000000000BE 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000BF 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000C0 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000C1 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:26 0000000100000000000000C2 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:24 0000000100000000000000C3 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000C4 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:23 0000000100000000000000C5 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:24 0000000100000000000000C6 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:23 0000000100000000000000C7 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000C8 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000C9 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:23 0000000100000000000000CA 2024-03-19T01:27:59.950 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:25 0000000100000000000000CB 2024-03-19T01:27:59.951 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000CC 2024-03-19T01:27:59.951 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000CD 2024-03-19T01:27:59.951 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000CE 2024-03-19T01:27:59.951 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:24 0000000100000000000000CF 2024-03-19T01:27:59.951 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:25 0000000100000000000000D0 2024-03-19T01:27:59.951 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:26 0000000100000000000000D1 2024-03-19T01:27:59.951 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000D2 2024-03-19T01:27:59.951 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:23 0000000100000000000000D3 2024-03-19T01:27:59.951 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000D4 2024-03-19T01:27:59.951 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:23 0000000100000000000000D5 2024-03-19T01:27:59.951 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:23 0000000100000000000000D6 2024-03-19T01:27:59.951 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:24 0000000100000000000000D7 2024-03-19T01:27:59.951 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000D8 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:25 0000000100000000000000D9 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000DA 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000DB 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000DC 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:23 0000000100000000000000DD 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000DE 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000DF 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:23 0000000100000000000000E0 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000E1 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:23 0000000100000000000000E2 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:24 0000000100000000000000E3 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000E4 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000E5 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:23 0000000100000000000000E6 2024-03-19T01:27:59.952 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:22 0000000100000000000000E7 2024-03-19T01:27:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:26 0000000100000000000000E8 2024-03-19T01:27:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000E9 2024-03-19T01:27:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000EA 2024-03-19T01:27:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000EB 2024-03-19T01:27:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000EC 2024-03-19T01:27:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000ED 2024-03-19T01:27:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000EE 2024-03-19T01:27:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000EF 2024-03-19T01:27:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000F0 2024-03-19T01:27:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000F1 2024-03-19T01:27:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000F2 2024-03-19T01:27:59.953 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000F3 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000F4 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000F5 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000F6 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000F7 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000F8 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 19 01:27 0000000100000000000000F9 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 archive_status 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout:total 0 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 19 01:12 . 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 65 Mar 19 01:27 .. 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:27:59.954 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/cephfs/postgres/data/pg_xact: 2024-03-19T01:27:59.955 INFO:teuthology.orchestra.run.smithi116.stdout:total 520 2024-03-19T01:27:59.955 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 19 01:27 . 2024-03-19T01:27:59.955 INFO:teuthology.orchestra.run.smithi116.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 25 Mar 19 01:27 .. 2024-03-19T01:27:59.955 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 262144 Mar 19 01:18 0000 2024-03-19T01:27:59.955 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 262144 Mar 19 01:27 0001 2024-03-19T01:27:59.955 INFO:teuthology.orchestra.run.smithi116.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 19 01:27 0002 2024-03-19T01:27:59.956 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-19T01:27:59.971 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T01:27:59.980 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-19T01:27:59.989 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:28:00.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.123+0000 7f11b759d640 1 -- 172.21.15.116:0/2753473754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11b0101690 msgr2=0x7f11b0101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.123+0000 7f11b759d640 1 --2- 172.21.15.116:0/2753473754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11b0101690 0x7f11b0101ab0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f11ac009900 tx=0x7f11ac02bc30 comp rx=0 tx=0).stop 2024-03-19T01:28:00.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.123+0000 7f11b759d640 1 -- 172.21.15.116:0/2753473754 shutdown_connections 2024-03-19T01:28:00.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.123+0000 7f11b759d640 1 --2- 172.21.15.116:0/2753473754 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11b0103240 0x7f11b0109ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.123+0000 7f11b759d640 1 --2- 172.21.15.116:0/2753473754 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11b0102880 0x7f11b0102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.123+0000 7f11b759d640 1 --2- 172.21.15.116:0/2753473754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11b0101690 0x7f11b0101ab0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.123+0000 7f11b759d640 1 -- 172.21.15.116:0/2753473754 >> 172.21.15.116:0/2753473754 conn(0x7f11b00fcc00 msgr2=0x7f11b00ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:00.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.123+0000 7f11b759d640 1 -- 172.21.15.116:0/2753473754 shutdown_connections 2024-03-19T01:28:00.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.123+0000 7f11b759d640 1 -- 172.21.15.116:0/2753473754 wait complete. 2024-03-19T01:28:00.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.123+0000 7f11b759d640 1 Processor -- start 2024-03-19T01:28:00.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.123+0000 7f11b759d640 1 -- start start 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b759d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11b0101690 0x7f11b0197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b759d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11b0102880 0x7f11b0197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b759d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11b0103240 0x7f11b019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b759d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11b010d310 con 0x7f11b0101690 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b759d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f11b010d190 con 0x7f11b0102880 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b759d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f11b010d490 con 0x7f11b0103240 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b659b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11b0101690 0x7f11b0197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b659b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11b0101690 0x7f11b0197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59792/0 (socket says 172.21.15.116:59792) 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b659b640 1 -- 172.21.15.116:0/3201220466 learned_addr learned my addr 172.21.15.116:0/3201220466 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b5d9a640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11b0102880 0x7f11b0197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b6d9c640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11b0103240 0x7f11b019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b659b640 1 -- 172.21.15.116:0/3201220466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11b0103240 msgr2=0x7f11b019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b659b640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11b0103240 0x7f11b019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b659b640 1 -- 172.21.15.116:0/3201220466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11b0102880 msgr2=0x7f11b0197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b659b640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11b0102880 0x7f11b0197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b659b640 1 -- 172.21.15.116:0/3201220466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b019f5a0 con 0x7f11b0101690 2024-03-19T01:28:00.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b5d9a640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11b0102880 0x7f11b0197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:28:00.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b659b640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11b0101690 0x7f11b0197980 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f11ac000c00 tx=0x7f11ac02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:00.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11a77fe640 1 -- 172.21.15.116:0/3201220466 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac041070 con 0x7f11b0101690 2024-03-19T01:28:00.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11a77fe640 1 -- 172.21.15.116:0/3201220466 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11ac004450 con 0x7f11b0101690 2024-03-19T01:28:00.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b759d640 1 -- 172.21.15.116:0/3201220466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11b019f830 con 0x7f11b0101690 2024-03-19T01:28:00.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11a77fe640 1 -- 172.21.15.116:0/3201220466 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11ac034440 con 0x7f11b0101690 2024-03-19T01:28:00.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.124+0000 7f11b759d640 1 -- 172.21.15.116:0/3201220466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11b019fd20 con 0x7f11b0101690 2024-03-19T01:28:00.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.126+0000 7f11b759d640 1 -- 172.21.15.116:0/3201220466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11b01089b0 con 0x7f11b0101690 2024-03-19T01:28:00.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.127+0000 7f11a77fe640 1 -- 172.21.15.116:0/3201220466 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f11ac044070 con 0x7f11b0101690 2024-03-19T01:28:00.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.127+0000 7f11a77fe640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f119c0a2800 0x7f119c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:00.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.127+0000 7f11b5d9a640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f119c0a2800 0x7f119c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:00.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.128+0000 7f11a77fe640 1 -- 172.21.15.116:0/3201220466 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f11ac0f0540 con 0x7f11b0101690 2024-03-19T01:28:00.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.128+0000 7f11b5d9a640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f119c0a2800 0x7f119c0a4cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f11b0198e90 tx=0x7f11a0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:00.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.132+0000 7f11a77fe640 1 -- 172.21.15.116:0/3201220466 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f11ac0bd000 con 0x7f11b0101690 2024-03-19T01:28:00.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.329+0000 7f11b759d640 1 -- 172.21.15.116:0/3201220466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11b002d410 con 0x7f11b0101690 2024-03-19T01:28:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.330+0000 7f11a77fe640 1 -- 172.21.15.116:0/3201220466 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22804 (secure 0 0 0) 0x7f11ac0c10a0 con 0x7f11b0101690 2024-03-19T01:28:00.331 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:28:00.331 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:11:28.144841+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:28:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.333+0000 7f11b759d640 1 -- 172.21.15.116:0/3201220466 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f119c0a2800 msgr2=0x7f119c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.333+0000 7f11b759d640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f119c0a2800 0x7f119c0a4cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f11b0198e90 tx=0x7f11a0005f90 comp rx=0 tx=0).stop 2024-03-19T01:28:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.333+0000 7f11b759d640 1 -- 172.21.15.116:0/3201220466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11b0101690 msgr2=0x7f11b0197980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.333+0000 7f11b759d640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11b0101690 0x7f11b0197980 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f11ac000c00 tx=0x7f11ac02c640 comp rx=0 tx=0).stop 2024-03-19T01:28:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.333+0000 7f11b759d640 1 -- 172.21.15.116:0/3201220466 shutdown_connections 2024-03-19T01:28:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.333+0000 7f11b759d640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f11b0103240 0x7f11b019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.333+0000 7f11b759d640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f119c0a2800 0x7f119c0a4cc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.333+0000 7f11b759d640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11b0102880 0x7f11b0197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.333+0000 7f11b759d640 1 --2- 172.21.15.116:0/3201220466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11b0101690 0x7f11b0197980 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.333+0000 7f11b759d640 1 -- 172.21.15.116:0/3201220466 >> 172.21.15.116:0/3201220466 conn(0x7f11b00fcc00 msgr2=0x7f11b0105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.333+0000 7f11b759d640 1 -- 172.21.15.116:0/3201220466 shutdown_connections 2024-03-19T01:28:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.333+0000 7f11b759d640 1 -- 172.21.15.116:0/3201220466 wait complete. 2024-03-19T01:28:00.345 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:05:53.301933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:10:31.028304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:10:31.704469+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:28:00.345 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:28:00.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.452+0000 7f6459f9d640 1 -- 172.21.15.116:0/1193000298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64540ffae0 msgr2=0x7f64540fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.452+0000 7f6459f9d640 1 --2- 172.21.15.116:0/1193000298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64540ffae0 0x7f64540fff00 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6444009930 tx=0x7f644402bc30 comp rx=0 tx=0).stop 2024-03-19T01:28:00.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.453+0000 7f6459f9d640 1 -- 172.21.15.116:0/1193000298 shutdown_connections 2024-03-19T01:28:00.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.453+0000 7f6459f9d640 1 --2- 172.21.15.116:0/1193000298 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6454103470 0x7f6454109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.453+0000 7f6459f9d640 1 --2- 172.21.15.116:0/1193000298 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6454102ab0 0x7f6454102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.453+0000 7f6459f9d640 1 --2- 172.21.15.116:0/1193000298 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64540ffae0 0x7f64540fff00 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.453+0000 7f6459f9d640 1 -- 172.21.15.116:0/1193000298 >> 172.21.15.116:0/1193000298 conn(0x7f64540fb410 msgr2=0x7f64540fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:00.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.453+0000 7f6459f9d640 1 -- 172.21.15.116:0/1193000298 shutdown_connections 2024-03-19T01:28:00.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.453+0000 7f6459f9d640 1 -- 172.21.15.116:0/1193000298 wait complete. 2024-03-19T01:28:00.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.453+0000 7f6459f9d640 1 Processor -- start 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.453+0000 7f6459f9d640 1 -- start start 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.453+0000 7f6459f9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64540ffae0 0x7f6454197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.453+0000 7f6459f9d640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6454102ab0 0x7f6454198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.453+0000 7f6459f9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454103470 0x7f645419f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f6459f9d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f645410d490 con 0x7f6454103470 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f6459f9d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f645410d310 con 0x7f64540ffae0 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f6459f9d640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f645410d610 con 0x7f6454102ab0 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f6453fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454103470 0x7f645419f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f64537fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64540ffae0 0x7f6454197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f6453fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454103470 0x7f645419f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59818/0 (socket says 172.21.15.116:59818) 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f6453fff640 1 -- 172.21.15.116:0/81282286 learned_addr learned my addr 172.21.15.116:0/81282286 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f64537fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64540ffae0 0x7f6454197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:52210/0 (socket says 172.21.15.116:52210) 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f6452ffd640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6454102ab0 0x7f6454198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:00.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f64537fe640 1 -- 172.21.15.116:0/81282286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6454102ab0 msgr2=0x7f6454198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f64537fe640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6454102ab0 0x7f6454198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f64537fe640 1 -- 172.21.15.116:0/81282286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454103470 msgr2=0x7f645419f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f64537fe640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454103470 0x7f645419f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f64537fe640 1 -- 172.21.15.116:0/81282286 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f645419f810 con 0x7f64540ffae0 2024-03-19T01:28:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f6453fff640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454103470 0x7f645419f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:28:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f64537fe640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64540ffae0 0x7f6454197bf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6444036040 tx=0x7f644402c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f6450ff9640 1 -- 172.21.15.116:0/81282286 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6444041070 con 0x7f64540ffae0 2024-03-19T01:28:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f6450ff9640 1 -- 172.21.15.116:0/81282286 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64440054e0 con 0x7f64540ffae0 2024-03-19T01:28:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f6459f9d640 1 -- 172.21.15.116:0/81282286 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f645419faa0 con 0x7f64540ffae0 2024-03-19T01:28:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f6450ff9640 1 -- 172.21.15.116:0/81282286 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6444034440 con 0x7f64540ffae0 2024-03-19T01:28:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.454+0000 7f6459f9d640 1 -- 172.21.15.116:0/81282286 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f645419ff60 con 0x7f64540ffae0 2024-03-19T01:28:00.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.455+0000 7f6459f9d640 1 -- 172.21.15.116:0/81282286 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6454108d50 con 0x7f64540ffae0 2024-03-19T01:28:00.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.456+0000 7f6450ff9640 1 -- 172.21.15.116:0/81282286 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6444044070 con 0x7f64540ffae0 2024-03-19T01:28:00.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.457+0000 7f6450ff9640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f643c0a2800 0x7f643c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:00.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.457+0000 7f6452ffd640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f643c0a2800 0x7f643c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:00.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.457+0000 7f6450ff9640 1 -- 172.21.15.116:0/81282286 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f64440f0550 con 0x7f64540ffae0 2024-03-19T01:28:00.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.457+0000 7f6452ffd640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f643c0a2800 0x7f643c0a4cc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6454199100 tx=0x7f6448005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:00.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.466+0000 7f6450ff9640 1 -- 172.21.15.116:0/81282286 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f64440bcf80 con 0x7f64540ffae0 2024-03-19T01:28:00.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.653+0000 7f6459f9d640 1 -- 172.21.15.116:0/81282286 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6454062420 con 0x7f64540ffae0 2024-03-19T01:28:00.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.653+0000 7f6450ff9640 1 -- 172.21.15.116:0/81282286 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f64440c1020 con 0x7f64540ffae0 2024-03-19T01:28:00.654 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T01:28:00.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:00 smithi116 ceph-mon[62701]: pgmap v813: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 32 MiB/s wr, 1.92k op/s 2024-03-19T01:28:00.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:00 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3201220466' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:28:00.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.656+0000 7f6459f9d640 1 -- 172.21.15.116:0/81282286 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f643c0a2800 msgr2=0x7f643c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.656+0000 7f6459f9d640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f643c0a2800 0x7f643c0a4cc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6454199100 tx=0x7f6448005f90 comp rx=0 tx=0).stop 2024-03-19T01:28:00.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.656+0000 7f6459f9d640 1 -- 172.21.15.116:0/81282286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64540ffae0 msgr2=0x7f6454197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.656+0000 7f6459f9d640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64540ffae0 0x7f6454197bf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6444036040 tx=0x7f644402c6d0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.656+0000 7f6459f9d640 1 -- 172.21.15.116:0/81282286 shutdown_connections 2024-03-19T01:28:00.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.656+0000 7f6459f9d640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6454103470 0x7f645419f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.656+0000 7f6459f9d640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6454102ab0 0x7f6454198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.656+0000 7f6459f9d640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f643c0a2800 0x7f643c0a4cc0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.656+0000 7f6459f9d640 1 --2- 172.21.15.116:0/81282286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64540ffae0 0x7f6454197bf0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.656+0000 7f6459f9d640 1 -- 172.21.15.116:0/81282286 >> 172.21.15.116:0/81282286 conn(0x7f64540fb410 msgr2=0x7f6454105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:00.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.656+0000 7f6459f9d640 1 -- 172.21.15.116:0/81282286 shutdown_connections 2024-03-19T01:28:00.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.656+0000 7f6459f9d640 1 -- 172.21.15.116:0/81282286 wait complete. 2024-03-19T01:28:00.667 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:28:00.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:00 smithi156 ceph-mon[66799]: pgmap v813: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 32 MiB/s wr, 1.92k op/s 2024-03-19T01:28:00.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:00 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3201220466' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:28:00.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.778+0000 7f934ae20640 1 -- 172.21.15.116:0/987632407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9344102a20 msgr2=0x7f9344102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.778+0000 7f934ae20640 1 --2- 172.21.15.116:0/987632407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9344102a20 0x7f9344102ea0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9330009900 tx=0x7f933002bc30 comp rx=0 tx=0).stop 2024-03-19T01:28:00.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.778+0000 7f934ae20640 1 -- 172.21.15.116:0/987632407 shutdown_connections 2024-03-19T01:28:00.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.778+0000 7f934ae20640 1 --2- 172.21.15.116:0/987632407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93441033e0 0x7f9344109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.778+0000 7f934ae20640 1 --2- 172.21.15.116:0/987632407 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9344102a20 0x7f9344102ea0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.778+0000 7f934ae20640 1 --2- 172.21.15.116:0/987632407 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9344101830 0x7f9344101c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.778+0000 7f934ae20640 1 -- 172.21.15.116:0/987632407 >> 172.21.15.116:0/987632407 conn(0x7f93440fcdc0 msgr2=0x7f93440ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.778+0000 7f934ae20640 1 -- 172.21.15.116:0/987632407 shutdown_connections 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.778+0000 7f934ae20640 1 -- 172.21.15.116:0/987632407 wait complete. 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f934ae20640 1 Processor -- start 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f934ae20640 1 -- start start 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f934ae20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9344101830 0x7f93440755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f934ae20640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9344102a20 0x7f9344073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f934ae20640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93441033e0 0x7f9344074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f934ae20640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f934410d4b0 con 0x7f9344101830 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f934ae20640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f934410d330 con 0x7f93441033e0 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f934ae20640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f934410d630 con 0x7f9344102a20 2024-03-19T01:28:00.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f9348bbc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9344101830 0x7f93440755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f9348bbc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9344101830 0x7f93440755d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59838/0 (socket says 172.21.15.116:59838) 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f9348bbc640 1 -- 172.21.15.116:0/1648003344 learned_addr learned my addr 172.21.15.116:0/1648003344 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f9343fff640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9344102a20 0x7f9344073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f93493bd640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93441033e0 0x7f9344074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f9348bbc640 1 -- 172.21.15.116:0/1648003344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9344102a20 msgr2=0x7f9344073c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f9348bbc640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9344102a20 0x7f9344073c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f9348bbc640 1 -- 172.21.15.116:0/1648003344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93441033e0 msgr2=0x7f9344074150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f9348bbc640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93441033e0 0x7f9344074150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.779+0000 7f9348bbc640 1 -- 172.21.15.116:0/1648003344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93440749e0 con 0x7f9344101830 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.780+0000 7f9348bbc640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9344101830 0x7f93440755d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f933800e9c0 tx=0x7f933800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.780+0000 7f9341ffb640 1 -- 172.21.15.116:0/1648003344 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933800cd80 con 0x7f9344101830 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.780+0000 7f934ae20640 1 -- 172.21.15.116:0/1648003344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93441a1c70 con 0x7f9344101830 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.780+0000 7f9341ffb640 1 -- 172.21.15.116:0/1648003344 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93380035f0 con 0x7f9344101830 2024-03-19T01:28:00.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.780+0000 7f9341ffb640 1 -- 172.21.15.116:0/1648003344 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9338010690 con 0x7f9344101830 2024-03-19T01:28:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.780+0000 7f934ae20640 1 -- 172.21.15.116:0/1648003344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93441a2140 con 0x7f9344101830 2024-03-19T01:28:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.781+0000 7f934ae20640 1 -- 172.21.15.116:0/1648003344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9344068a60 con 0x7f9344101830 2024-03-19T01:28:00.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.782+0000 7f9341ffb640 1 -- 172.21.15.116:0/1648003344 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f93380037a0 con 0x7f9344101830 2024-03-19T01:28:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.782+0000 7f9341ffb640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f93340a2800 0x7f93340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.782+0000 7f9343fff640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f93340a2800 0x7f93340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.782+0000 7f9341ffb640 1 -- 172.21.15.116:0/1648003344 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f93380cee60 con 0x7f9344101830 2024-03-19T01:28:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.783+0000 7f9343fff640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f93340a2800 0x7f93340a4cc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f933002c7b0 tx=0x7f9330005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:00.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.787+0000 7f9341ffb640 1 -- 172.21.15.116:0/1648003344 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f933809bee0 con 0x7f9344101830 2024-03-19T01:28:00.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:00 smithi162 ceph-mon[67178]: pgmap v813: 97 pgs: 97 active+clean; 1.6 GiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 32 MiB/s wr, 1.92k op/s 2024-03-19T01:28:00.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:00 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3201220466' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:28:00.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.969+0000 7f934ae20640 1 -- 172.21.15.116:0/1648003344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9344108bd0 con 0x7f9344101830 2024-03-19T01:28:00.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.969+0000 7f9341ffb640 1 -- 172.21.15.116:0/1648003344 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f9344108bd0 con 0x7f9344101830 2024-03-19T01:28:00.970 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T01:28:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.971+0000 7f934ae20640 1 -- 172.21.15.116:0/1648003344 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f93340a2800 msgr2=0x7f93340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.971+0000 7f934ae20640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f93340a2800 0x7f93340a4cc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f933002c7b0 tx=0x7f9330005fd0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.971+0000 7f934ae20640 1 -- 172.21.15.116:0/1648003344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9344101830 msgr2=0x7f93440755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.971+0000 7f934ae20640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9344101830 0x7f93440755d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f933800e9c0 tx=0x7f933800ee80 comp rx=0 tx=0).stop 2024-03-19T01:28:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.971+0000 7f934ae20640 1 -- 172.21.15.116:0/1648003344 shutdown_connections 2024-03-19T01:28:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.971+0000 7f934ae20640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f93340a2800 0x7f93340a4cc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.971+0000 7f934ae20640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93441033e0 0x7f9344074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.971+0000 7f934ae20640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f9344102a20 0x7f9344073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.971+0000 7f934ae20640 1 --2- 172.21.15.116:0/1648003344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9344101830 0x7f93440755d0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:00.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.971+0000 7f934ae20640 1 -- 172.21.15.116:0/1648003344 >> 172.21.15.116:0/1648003344 conn(0x7f93440fcdc0 msgr2=0x7f9344105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:00.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.971+0000 7f934ae20640 1 -- 172.21.15.116:0/1648003344 shutdown_connections 2024-03-19T01:28:00.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:00.971+0000 7f934ae20640 1 -- 172.21.15.116:0/1648003344 wait complete. 2024-03-19T01:28:00.983 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:28:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.094+0000 7fda5b2ea640 1 -- 172.21.15.116:0/1890778753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda54101a10 msgr2=0x7fda54108330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.094+0000 7fda5b2ea640 1 --2- 172.21.15.116:0/1890778753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda54101a10 0x7fda54108330 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fda50009900 tx=0x7fda5002bc30 comp rx=0 tx=0).stop 2024-03-19T01:28:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.094+0000 7fda5b2ea640 1 -- 172.21.15.116:0/1890778753 shutdown_connections 2024-03-19T01:28:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.094+0000 7fda5b2ea640 1 --2- 172.21.15.116:0/1890778753 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda54101a10 0x7fda54108330 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.094+0000 7fda5b2ea640 1 --2- 172.21.15.116:0/1890778753 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda54101050 0x7fda541014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.094+0000 7fda5b2ea640 1 --2- 172.21.15.116:0/1890778753 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda540ffe70 0x7fda54100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.094+0000 7fda5b2ea640 1 -- 172.21.15.116:0/1890778753 >> 172.21.15.116:0/1890778753 conn(0x7fda540fb450 msgr2=0x7fda540fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.094+0000 7fda5b2ea640 1 -- 172.21.15.116:0/1890778753 shutdown_connections 2024-03-19T01:28:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.094+0000 7fda5b2ea640 1 -- 172.21.15.116:0/1890778753 wait complete. 2024-03-19T01:28:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda5b2ea640 1 Processor -- start 2024-03-19T01:28:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda5b2ea640 1 -- start start 2024-03-19T01:28:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda5b2ea640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda540ffe70 0x7fda54197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda5b2ea640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda54101050 0x7fda54198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda5b2ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda54101a10 0x7fda5419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda5b2ea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fda5410b2e0 con 0x7fda54101a10 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda5b2ea640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fda5410b160 con 0x7fda540ffe70 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda5b2ea640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fda5410b460 con 0x7fda54101050 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda59887640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda54101a10 0x7fda5419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda59887640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda54101a10 0x7fda5419f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59854/0 (socket says 172.21.15.116:59854) 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda59887640 1 -- 172.21.15.116:0/3325227269 learned_addr learned my addr 172.21.15.116:0/3325227269 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda59086640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda540ffe70 0x7fda54197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda58885640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda54101050 0x7fda54198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda59887640 1 -- 172.21.15.116:0/3325227269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda54101050 msgr2=0x7fda54198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda59887640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda54101050 0x7fda54198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda59887640 1 -- 172.21.15.116:0/3325227269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda540ffe70 msgr2=0x7fda54197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda59887640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda540ffe70 0x7fda54197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda59887640 1 -- 172.21.15.116:0/3325227269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda5419f720 con 0x7fda54101a10 2024-03-19T01:28:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.095+0000 7fda59086640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda540ffe70 0x7fda54197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:28:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.096+0000 7fda59887640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda54101a10 0x7fda5419f0b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fda50005ab0 tx=0x7fda5002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.096+0000 7fda4a7fc640 1 -- 172.21.15.116:0/3325227269 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50041070 con 0x7fda54101a10 2024-03-19T01:28:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.096+0000 7fda4a7fc640 1 -- 172.21.15.116:0/3325227269 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda50004450 con 0x7fda54101a10 2024-03-19T01:28:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.096+0000 7fda5b2ea640 1 -- 172.21.15.116:0/3325227269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda5419f9b0 con 0x7fda54101a10 2024-03-19T01:28:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.096+0000 7fda4a7fc640 1 -- 172.21.15.116:0/3325227269 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50003e60 con 0x7fda54101a10 2024-03-19T01:28:01.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.096+0000 7fda5b2ea640 1 -- 172.21.15.116:0/3325227269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda5419fef0 con 0x7fda54101a10 2024-03-19T01:28:01.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.097+0000 7fda5b2ea640 1 -- 172.21.15.116:0/3325227269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda541071b0 con 0x7fda54101a10 2024-03-19T01:28:01.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.098+0000 7fda4a7fc640 1 -- 172.21.15.116:0/3325227269 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fda50044070 con 0x7fda54101a10 2024-03-19T01:28:01.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.099+0000 7fda4a7fc640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fda400a2900 0x7fda400a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:01.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.099+0000 7fda59086640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fda400a2900 0x7fda400a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:01.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.099+0000 7fda4a7fc640 1 -- 172.21.15.116:0/3325227269 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7fda500f0a20 con 0x7fda54101a10 2024-03-19T01:28:01.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.099+0000 7fda59086640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fda400a2900 0x7fda400a4dc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0099a0 tx=0x7fda4c009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:01.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.102+0000 7fda4a7fc640 1 -- 172.21.15.116:0/3325227269 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fda500bd3f0 con 0x7fda54101a10 2024-03-19T01:28:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.283+0000 7fda5b2ea640 1 -- 172.21.15.116:0/3325227269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fda5402d1f0 con 0x7fda54101a10 2024-03-19T01:28:01.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.284+0000 7fda4a7fc640 1 -- 172.21.15.116:0/3325227269 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7fda500c1490 con 0x7fda54101a10 2024-03-19T01:28:01.284 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T01:28:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.285+0000 7fda5b2ea640 1 -- 172.21.15.116:0/3325227269 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fda400a2900 msgr2=0x7fda400a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.285+0000 7fda5b2ea640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fda400a2900 0x7fda400a4dc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0099a0 tx=0x7fda4c009500 comp rx=0 tx=0).stop 2024-03-19T01:28:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.285+0000 7fda5b2ea640 1 -- 172.21.15.116:0/3325227269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda54101a10 msgr2=0x7fda5419f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:01.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.285+0000 7fda5b2ea640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda54101a10 0x7fda5419f0b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fda50005ab0 tx=0x7fda5002c6c0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.286+0000 7fda5b2ea640 1 -- 172.21.15.116:0/3325227269 shutdown_connections 2024-03-19T01:28:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.286+0000 7fda5b2ea640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fda54101a10 0x7fda5419f0b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.286+0000 7fda5b2ea640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fda54101050 0x7fda54198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.286+0000 7fda5b2ea640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fda400a2900 0x7fda400a4dc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.286+0000 7fda5b2ea640 1 --2- 172.21.15.116:0/3325227269 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda540ffe70 0x7fda54197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.286+0000 7fda5b2ea640 1 -- 172.21.15.116:0/3325227269 >> 172.21.15.116:0/3325227269 conn(0x7fda540fb450 msgr2=0x7fda54104380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.286+0000 7fda5b2ea640 1 -- 172.21.15.116:0/3325227269 shutdown_connections 2024-03-19T01:28:01.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.286+0000 7fda5b2ea640 1 -- 172.21.15.116:0/3325227269 wait complete. 2024-03-19T01:28:01.297 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-19T01:28:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.416+0000 7f89ce259640 1 -- 172.21.15.116:0/3532136486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c81033f0 msgr2=0x7f89c8109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.416+0000 7f89ce259640 1 --2- 172.21.15.116:0/3532136486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c81033f0 0x7f89c8109c80 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009900 tx=0x7f89c002bc30 comp rx=0 tx=0).stop 2024-03-19T01:28:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.416+0000 7f89ce259640 1 -- 172.21.15.116:0/3532136486 shutdown_connections 2024-03-19T01:28:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.416+0000 7f89ce259640 1 --2- 172.21.15.116:0/3532136486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c81033f0 0x7f89c8109c80 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.416+0000 7f89ce259640 1 --2- 172.21.15.116:0/3532136486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89c8102a30 0x7f89c8102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.416+0000 7f89ce259640 1 --2- 172.21.15.116:0/3532136486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89c8101840 0x7f89c8101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.416+0000 7f89ce259640 1 -- 172.21.15.116:0/3532136486 >> 172.21.15.116:0/3532136486 conn(0x7f89c80fcdd0 msgr2=0x7f89c80ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.416+0000 7f89ce259640 1 -- 172.21.15.116:0/3532136486 shutdown_connections 2024-03-19T01:28:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.416+0000 7f89ce259640 1 -- 172.21.15.116:0/3532136486 wait complete. 2024-03-19T01:28:01.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.416+0000 7f89ce259640 1 Processor -- start 2024-03-19T01:28:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.416+0000 7f89ce259640 1 -- start start 2024-03-19T01:28:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89ce259640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8101840 0x7f89c8197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89ce259640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89c8102a30 0x7f89c81980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89ce259640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89c81033f0 0x7f89c819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89ce259640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f89c810d4c0 con 0x7f89c8101840 2024-03-19T01:28:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89ce259640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f89c810d340 con 0x7f89c81033f0 2024-03-19T01:28:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89ce259640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f89c810d640 con 0x7f89c8102a30 2024-03-19T01:28:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8101840 0x7f89c8197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8101840 0x7f89c8197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59878/0 (socket says 172.21.15.116:59878) 2024-03-19T01:28:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c77fe640 1 -- 172.21.15.116:0/2639695340 learned_addr learned my addr 172.21.15.116:0/2639695340 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c6ffd640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89c8102a30 0x7f89c81980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c7fff640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89c81033f0 0x7f89c819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c77fe640 1 -- 172.21.15.116:0/2639695340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89c8102a30 msgr2=0x7f89c81980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c77fe640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89c8102a30 0x7f89c81980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c77fe640 1 -- 172.21.15.116:0/2639695340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89c81033f0 msgr2=0x7f89c819f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c77fe640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89c81033f0 0x7f89c819f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c77fe640 1 -- 172.21.15.116:0/2639695340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c819f790 con 0x7f89c8101840 2024-03-19T01:28:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c77fe640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8101840 0x7f89c8197b70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f89b800ebc0 tx=0x7f89b800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c4ff9640 1 -- 172.21.15.116:0/2639695340 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8003b70 con 0x7f89c8101840 2024-03-19T01:28:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c4ff9640 1 -- 172.21.15.116:0/2639695340 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89b8003d10 con 0x7f89c8101840 2024-03-19T01:28:01.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89ce259640 1 -- 172.21.15.116:0/2639695340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c819fa80 con 0x7f89c8101840 2024-03-19T01:28:01.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89c4ff9640 1 -- 172.21.15.116:0/2639695340 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8019980 con 0x7f89c8101840 2024-03-19T01:28:01.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.417+0000 7f89ce259640 1 -- 172.21.15.116:0/2639695340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c819ffd0 con 0x7f89c8101840 2024-03-19T01:28:01.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.418+0000 7f89ce259640 1 -- 172.21.15.116:0/2639695340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c8108b00 con 0x7f89c8101840 2024-03-19T01:28:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.420+0000 7f89c4ff9640 1 -- 172.21.15.116:0/2639695340 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f89b8019b20 con 0x7f89c8101840 2024-03-19T01:28:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.420+0000 7f89c4ff9640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f89b40a2800 0x7f89b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.420+0000 7f89c6ffd640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f89b40a2800 0x7f89b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.420+0000 7f89c4ff9640 1 -- 172.21.15.116:0/2639695340 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f89b80cf260 con 0x7f89c8101840 2024-03-19T01:28:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.421+0000 7f89c6ffd640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f89b40a2800 0x7f89b40a4cc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f89c8199080 tx=0x7f89b0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:01.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.425+0000 7f89c4ff9640 1 -- 172.21.15.116:0/2639695340 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f89b809c2e0 con 0x7f89c8101840 2024-03-19T01:28:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:01 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:01 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/81282286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:28:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:01 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1648003344' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:28:01.655 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:01 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3325227269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:28:01.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.668+0000 7f89ce259640 1 -- 172.21.15.116:0/2639695340 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f89c8062420 con 0x7f89c8101840 2024-03-19T01:28:01.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.676+0000 7f89c4ff9640 1 -- 172.21.15.116:0/2639695340 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) ==== 81+0+739839 (secure 0 0 0) 0x7f89b809bbe0 con 0x7f89c8101840 2024-03-19T01:28:01.687 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:28:01.687 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:28:01.687 INFO:teuthology.orchestra.run.smithi116.stdout: "epoch": 18, 2024-03-19T01:28:01.687 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.687 INFO:teuthology.orchestra.run.smithi116.stdout: "active_gid": 14150, 2024-03-19T01:28:01.687 INFO:teuthology.orchestra.run.smithi116.stdout: "active_name": "x", 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: "active_addrs": { 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6800", 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2389438183 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:6801", 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 2389438183 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T01:28:01.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.689 INFO:teuthology.orchestra.run.smithi116.stdout: "active_addr": "172.21.15.116:6801/2389438183", 2024-03-19T01:28:01.689 INFO:teuthology.orchestra.run.smithi116.stdout: "active_change": "2024-03-19T01:03:26.974997+0000", 2024-03-19T01:28:01.689 INFO:teuthology.orchestra.run.smithi116.stdout: "active_mgr_features": 4540701547738038271, 2024-03-19T01:28:01.689 INFO:teuthology.orchestra.run.smithi116.stdout: "available": true, 2024-03-19T01:28:01.689 INFO:teuthology.orchestra.run.smithi116.stdout: "standbys": [ 2024-03-19T01:28:01.689 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.689 INFO:teuthology.orchestra.run.smithi116.stdout: "gid": 24109, 2024-03-19T01:28:01.689 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "z", 2024-03-19T01:28:01.689 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr_features": 4540701547738038271, 2024-03-19T01:28:01.689 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-19T01:28:01.689 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.689 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-19T01:28:01.689 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.690 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.690 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.690 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:01.690 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:01.690 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.690 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.690 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.690 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:01.690 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.690 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.690 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.690 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-19T01:28:01.690 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.691 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.692 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.693 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.694 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.694 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.694 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.694 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.694 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.694 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.694 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.694 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.694 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.694 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.695 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.696 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.697 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-19T01:28:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-19T01:28:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-19T01:28:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.699 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.700 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-19T01:28:01.701 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.701 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.701 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.701 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-19T01:28:01.701 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-19T01:28:01.701 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.701 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.701 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.701 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.701 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.702 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.702 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.702 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-19T01:28:01.702 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.702 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.702 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.702 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-19T01:28:01.702 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-19T01:28:01.702 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-19T01:28:01.703 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.704 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.704 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.704 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.704 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.704 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.704 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.704 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-19T01:28:01.704 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.704 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.704 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.704 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.705 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.705 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-19T01:28:01.705 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.705 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.705 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.705 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-19T01:28:01.705 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-19T01:28:01.705 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.705 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.705 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.705 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.705 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-19T01:28:01.706 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.707 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.707 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.707 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-19T01:28:01.707 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T01:28:01.707 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.707 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.707 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-19T01:28:01.707 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-19T01:28:01.707 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:01.707 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.708 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.708 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:01.708 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-19T01:28:01.708 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-19T01:28:01.708 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.708 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-19T01:28:01.708 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T01:28:01.708 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.708 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.709 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-19T01:28:01.709 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-19T01:28:01.709 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:01.709 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.709 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.709 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-19T01:28:01.709 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T01:28:01.710 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-19T01:28:01.710 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.710 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-19T01:28:01.710 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.710 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.710 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.710 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-19T01:28:01.710 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-19T01:28:01.711 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.711 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.711 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.711 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-19T01:28:01.711 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.711 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.711 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.711 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-19T01:28:01.712 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-19T01:28:01.712 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.712 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.712 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.712 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-19T01:28:01.712 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-19T01:28:01.712 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:01.712 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.713 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.713 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-19T01:28:01.713 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-19T01:28:01.713 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-19T01:28:01.713 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.713 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-19T01:28:01.713 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-19T01:28:01.713 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.714 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.714 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.714 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-19T01:28:01.714 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-19T01:28:01.714 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.714 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.714 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.714 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-19T01:28:01.715 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.715 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.715 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.715 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-19T01:28:01.715 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T01:28:01.715 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.715 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.716 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-19T01:28:01.716 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-19T01:28:01.716 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:01.716 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.716 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.716 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:01.716 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-19T01:28:01.716 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-19T01:28:01.716 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.717 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-19T01:28:01.717 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T01:28:01.717 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.717 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.717 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.717 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.717 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.718 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.718 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.718 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.718 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.718 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.718 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.718 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.718 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.719 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.719 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.719 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.719 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.719 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.719 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.719 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.719 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.719 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.720 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.720 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.720 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.720 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.720 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.720 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.721 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.721 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.721 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.721 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.721 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.721 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.721 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.721 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.721 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.721 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.721 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.723 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.723 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.723 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.723 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.723 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.723 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.723 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.723 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.723 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.723 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.723 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.723 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.724 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.724 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.724 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.724 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.724 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-19T01:28:01.724 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-19T01:28:01.725 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.726 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.726 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.726 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-19T01:28:01.726 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.726 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.726 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.726 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-19T01:28:01.726 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-19T01:28:01.727 INFO:teuthology.orchestra.run.smithi116.stdout: "read", 2024-03-19T01:28:01.727 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap", 2024-03-19T01:28:01.727 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap-read" 2024-03-19T01:28:01.727 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.727 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-19T01:28:01.728 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.728 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.728 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.728 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-19T01:28:01.728 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-19T01:28:01.729 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.729 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.729 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.729 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.729 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.729 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.730 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.730 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-19T01:28:01.730 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.730 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.730 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.730 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:01.731 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:01.731 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.731 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.731 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.731 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:01.731 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.731 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.731 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.731 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-19T01:28:01.731 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.731 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.731 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.732 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.732 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-19T01:28:01.732 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-19T01:28:01.732 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.732 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.732 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.732 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T01:28:01.732 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T01:28:01.732 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.732 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.732 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-19T01:28:01.733 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-19T01:28:01.733 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.733 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.733 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-19T01:28:01.733 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-19T01:28:01.733 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:01.733 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.733 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.733 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:01.733 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.733 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.733 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.734 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-19T01:28:01.734 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.734 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.734 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.734 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.734 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.734 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-19T01:28:01.734 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.734 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.734 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.734 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-19T01:28:01.735 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-19T01:28:01.735 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:01.735 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.735 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.735 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-19T01:28:01.735 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.735 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.736 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.736 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-19T01:28:01.736 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.736 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.736 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.736 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-19T01:28:01.736 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-19T01:28:01.736 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.736 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.736 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.736 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-19T01:28:01.736 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.737 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.737 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.737 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-19T01:28:01.737 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.737 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.737 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.737 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-19T01:28:01.737 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-19T01:28:01.737 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.737 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.737 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.738 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-19T01:28:01.738 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.738 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.738 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.738 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-19T01:28:01.738 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.738 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.738 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.738 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-19T01:28:01.738 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-19T01:28:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-19T01:28:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-19T01:28:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.739 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-19T01:28:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-19T01:28:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-19T01:28:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.740 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-19T01:28:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-19T01:28:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:01.741 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-19T01:28:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.742 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-19T01:28:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm_log_destination": { 2024-03-19T01:28:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm_log_destination", 2024-03-19T01:28:01.743 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "file", 2024-03-19T01:28:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "file,syslog", 2024-03-19T01:28:01.744 INFO:teuthology.orchestra.run.smithi116.stdout: "syslog" 2024-03-19T01:28:01.745 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.745 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-19T01:28:01.745 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.745 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.745 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.746 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-19T01:28:01.746 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-19T01:28:01.746 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.746 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.746 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.746 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.746 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.746 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.746 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.746 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-19T01:28:01.746 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.747 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.747 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.747 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-19T01:28:01.747 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-19T01:28:01.747 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.748 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.748 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.748 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.748 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.748 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.749 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-19T01:28:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-19T01:28:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-19T01:28:01.750 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.751 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.752 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.752 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-19T01:28:01.752 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.752 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.752 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.752 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-19T01:28:01.752 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.752 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.752 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.752 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.752 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-19T01:28:01.752 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-19T01:28:01.752 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.753 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.753 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.753 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-19T01:28:01.753 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.753 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.753 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.753 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-19T01:28:01.753 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.753 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.753 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.753 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.754 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-19T01:28:01.754 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-19T01:28:01.754 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.754 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.754 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.754 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-19T01:28:01.754 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.754 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.754 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.755 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T01:28:01.755 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.755 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.755 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.755 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-19T01:28:01.755 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-19T01:28:01.755 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.755 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.755 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.756 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-19T01:28:01.756 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.756 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.756 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.756 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-19T01:28:01.756 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.757 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.757 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.757 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.758 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-19T01:28:01.758 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-19T01:28:01.758 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.758 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.759 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.759 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-19T01:28:01.759 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.759 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.759 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.759 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-19T01:28:01.759 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.759 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.759 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-19T01:28:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-19T01:28:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-19T01:28:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-19T01:28:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.760 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.761 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-19T01:28:01.761 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-19T01:28:01.761 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.761 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.761 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.761 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-19T01:28:01.761 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.761 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.762 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.762 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-19T01:28:01.762 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.762 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.762 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.762 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-19T01:28:01.762 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-19T01:28:01.763 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-19T01:28:01.764 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.765 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.765 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.765 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-19T01:28:01.765 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.765 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.765 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.765 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T01:28:01.765 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.765 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.766 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.766 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-19T01:28:01.766 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-19T01:28:01.766 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.766 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.766 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.766 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-19T01:28:01.766 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.766 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.766 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.767 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-19T01:28:01.767 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.767 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.767 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.767 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-19T01:28:01.767 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-19T01:28:01.767 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.767 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.767 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.767 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-19T01:28:01.767 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.767 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.768 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.768 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T01:28:01.768 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.768 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.768 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.768 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.768 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-19T01:28:01.768 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-19T01:28:01.769 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.769 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.769 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.769 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-19T01:28:01.769 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.769 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.770 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.770 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-19T01:28:01.770 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.770 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.770 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.771 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-19T01:28:01.771 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-19T01:28:01.771 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.771 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.771 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.772 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-19T01:28:01.772 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.772 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.772 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.772 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-19T01:28:01.772 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.773 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.773 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.773 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-19T01:28:01.773 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-19T01:28:01.773 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.774 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.774 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.774 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.774 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.774 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.774 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.775 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-19T01:28:01.775 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.775 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.775 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.776 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-19T01:28:01.776 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-19T01:28:01.776 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.776 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.776 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.777 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:01.777 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.777 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.777 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.777 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-19T01:28:01.778 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.778 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.778 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.778 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.778 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-19T01:28:01.778 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-19T01:28:01.778 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.779 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.779 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.779 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-19T01:28:01.779 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.779 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.779 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.779 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-19T01:28:01.780 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.780 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.780 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.780 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.780 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-19T01:28:01.780 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-19T01:28:01.780 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.781 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.781 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.781 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-19T01:28:01.781 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.781 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.781 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.781 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-19T01:28:01.781 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.782 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.782 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.782 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-19T01:28:01.782 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-19T01:28:01.782 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-19T01:28:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-19T01:28:01.783 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-19T01:28:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-19T01:28:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.784 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-19T01:28:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.785 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-19T01:28:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-19T01:28:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.786 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-19T01:28:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.787 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.788 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.788 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.788 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-19T01:28:01.788 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-19T01:28:01.788 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.788 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.789 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.789 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:01.789 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.789 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.789 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.789 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-19T01:28:01.790 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.790 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.790 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.790 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.790 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-19T01:28:01.790 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-19T01:28:01.791 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.791 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.791 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.791 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.791 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.791 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.791 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.792 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-19T01:28:01.792 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.792 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.792 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.792 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.792 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-19T01:28:01.792 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-19T01:28:01.793 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.793 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.793 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.793 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.793 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.793 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.793 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.793 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-19T01:28:01.793 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.794 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.794 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.794 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.794 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.794 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.794 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.794 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.795 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.795 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.795 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.795 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.795 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.795 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.795 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.796 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.796 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.796 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.796 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.796 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.796 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.797 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.797 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.797 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.797 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-19T01:28:01.797 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-19T01:28:01.798 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.798 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.798 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.798 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.798 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.798 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.799 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.799 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-19T01:28:01.799 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.799 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.799 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.800 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.800 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.800 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.800 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.800 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.801 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.801 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.801 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.801 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.801 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-19T01:28:01.801 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.801 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.802 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.802 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.802 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.802 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.802 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.802 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.802 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.802 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.802 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.802 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.802 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.803 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.804 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.805 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.806 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.806 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-19T01:28:01.806 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.806 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.806 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.806 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-19T01:28:01.806 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.806 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.806 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.806 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.806 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-19T01:28:01.806 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-19T01:28:01.806 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.807 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.808 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.809 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-19T01:28:01.810 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.810 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.810 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.810 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-19T01:28:01.810 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-19T01:28:01.810 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.810 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-19T01:28:01.810 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.810 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.810 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.810 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-19T01:28:01.810 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-19T01:28:01.811 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.812 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.812 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.812 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-19T01:28:01.812 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.812 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.812 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.813 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-19T01:28:01.813 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.813 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.813 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.813 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-19T01:28:01.814 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-19T01:28:01.814 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.814 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.818 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.818 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.818 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.818 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.819 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.819 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-19T01:28:01.819 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.819 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.819 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.819 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.819 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.820 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-19T01:28:01.821 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-19T01:28:01.821 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.821 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.821 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.821 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.821 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.821 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.821 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.821 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-19T01:28:01.821 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.821 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.821 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.822 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.822 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-19T01:28:01.822 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-19T01:28:01.822 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.822 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.822 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.822 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.822 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.822 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.822 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.822 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-19T01:28:01.822 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.822 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.823 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.824 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.825 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.826 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-19T01:28:01.826 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-19T01:28:01.826 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.826 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.826 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.826 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.826 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.826 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.826 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.826 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-19T01:28:01.826 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.826 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.827 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.829 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-19T01:28:01.830 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-19T01:28:01.830 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.830 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.830 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.830 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.830 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.830 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.830 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.830 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-19T01:28:01.830 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.830 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.830 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.830 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.831 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.832 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.833 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.834 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.835 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.835 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.835 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.835 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.835 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.835 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.835 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.835 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.835 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.835 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.835 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.835 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.836 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.837 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.838 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.838 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.838 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.838 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-19T01:28:01.838 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.838 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.838 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.838 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-19T01:28:01.838 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-19T01:28:01.838 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.839 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.839 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.839 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:01.839 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.839 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.839 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.839 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.839 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.839 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.839 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.839 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-19T01:28:01.839 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-19T01:28:01.840 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-19T01:28:01.841 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.842 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.842 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.842 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.842 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.842 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.842 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.842 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.842 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.842 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.842 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.842 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.842 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-19T01:28:01.842 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-19T01:28:01.843 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-19T01:28:01.844 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-19T01:28:01.845 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.846 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-19T01:28:01.847 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-19T01:28:01.848 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-19T01:28:01.849 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.850 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-19T01:28:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-19T01:28:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.851 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-19T01:28:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-19T01:28:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.852 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-19T01:28:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-19T01:28:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T01:28:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.853 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.854 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-19T01:28:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-19T01:28:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.855 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-19T01:28:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-19T01:28:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T01:28:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.856 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-19T01:28:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-19T01:28:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.857 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-19T01:28:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-19T01:28:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.858 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.859 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-19T01:28:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-19T01:28:01.860 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "MULTICLUSTER_CONFIG": { 2024-03-19T01:28:01.861 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-19T01:28:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "{}", 2024-03-19T01:28:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.862 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-19T01:28:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-19T01:28:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.864 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.865 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-19T01:28:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-19T01:28:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.866 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.867 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.868 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-19T01:28:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-19T01:28:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.869 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.870 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.871 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-19T01:28:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-19T01:28:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.872 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.873 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-19T01:28:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-19T01:28:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.874 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.875 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.876 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-19T01:28:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-19T01:28:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.877 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.878 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-19T01:28:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-19T01:28:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.879 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.880 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "UNSAFE_TLS_v1_2": { 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.881 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.882 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.883 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.884 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.885 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.886 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-19T01:28:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-19T01:28:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-19T01:28:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.887 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-19T01:28:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-19T01:28:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-19T01:28:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.888 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.889 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.890 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.891 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.892 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.893 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.894 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-19T01:28:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-19T01:28:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-19T01:28:01.895 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-19T01:28:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-19T01:28:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.896 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T01:28:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T01:28:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-19T01:28:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.897 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T01:28:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T01:28:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-19T01:28:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.898 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.899 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.900 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-19T01:28:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-19T01:28:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-19T01:28:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.901 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-19T01:28:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-19T01:28:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-19T01:28:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.902 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.903 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.904 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.905 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.906 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.909 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.910 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.911 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.912 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.913 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-19T01:28:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-19T01:28:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-19T01:28:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-19T01:28:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.914 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-19T01:28:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-19T01:28:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-19T01:28:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.915 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-19T01:28:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.916 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.917 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-19T01:28:01.918 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.919 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.920 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.921 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.922 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.923 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.924 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.925 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.926 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-19T01:28:01.927 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-19T01:28:01.928 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-19T01:28:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-19T01:28:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.929 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.930 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-19T01:28:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-19T01:28:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-19T01:28:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.933 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.933 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.933 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.933 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.933 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.933 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.933 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.933 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.933 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.933 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.933 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.933 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.933 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.934 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.934 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.934 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.934 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.934 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.934 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.934 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.934 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.934 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.934 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.934 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.934 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.934 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.935 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.936 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.937 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.938 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.940 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.941 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-19T01:28:01.942 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-19T01:28:01.942 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.942 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.942 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.942 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-19T01:28:01.942 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.942 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.942 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.942 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T01:28:01.942 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.942 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.942 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.943 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.946 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.946 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.946 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.946 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.946 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.946 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.946 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.946 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.946 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.946 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.946 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.946 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.946 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.947 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.947 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.947 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.947 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.947 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.947 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.947 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.947 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.947 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.947 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.947 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.947 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.947 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.948 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.948 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.948 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.948 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.948 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.948 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.948 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.948 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.948 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.948 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.948 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.948 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-19T01:28:01.949 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.949 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.949 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.949 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.949 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.949 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.949 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.949 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.949 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.950 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.951 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.951 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.951 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.951 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.951 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.951 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.951 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.951 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.951 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.951 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.951 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.953 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.954 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.954 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.954 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.954 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.954 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.954 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.954 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.954 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.954 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.954 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.954 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.954 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.955 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-19T01:28:01.956 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.957 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.957 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-19T01:28:01.957 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.957 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.957 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.957 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.957 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.957 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.957 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.957 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.957 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.957 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.958 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.958 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.958 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.958 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.958 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.958 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.958 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.958 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.958 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.958 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.958 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.958 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.959 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.960 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.961 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.961 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.961 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.961 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.961 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.961 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.961 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.961 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.961 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.961 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.961 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.962 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.963 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.964 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.965 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.966 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.967 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.968 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.969 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-19T01:28:01.970 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-19T01:28:01.970 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.970 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.970 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.970 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-19T01:28:01.970 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.970 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.970 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.970 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-19T01:28:01.970 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.970 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.970 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-19T01:28:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-19T01:28:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-19T01:28:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-19T01:28:01.971 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-19T01:28:01.972 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-19T01:28:01.973 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-19T01:28:01.974 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.974 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.974 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.974 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.974 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.974 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-19T01:28:01.974 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.974 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.974 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.974 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.974 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.975 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.976 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.977 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.978 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.979 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.979 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.979 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.979 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.979 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.979 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.979 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.979 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.979 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.979 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.979 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.979 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.980 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.981 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.982 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.982 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.982 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.982 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.982 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.982 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.982 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.982 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.982 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.982 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.982 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.982 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.983 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.984 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.985 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.985 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.985 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.985 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.985 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.985 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.985 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.985 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.985 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-19T01:28:01.985 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.985 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.985 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.986 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.987 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.988 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.989 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.991 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.992 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.993 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.994 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.995 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.996 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-19T01:28:01.997 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:01.998 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:01.999 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.000 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.002 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.002 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.002 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.002 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.002 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.002 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.002 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.002 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.002 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.002 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.002 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.002 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.002 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.003 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.004 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.005 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.006 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.007 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.009 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.010 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.011 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.012 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.013 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:02.014 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.015 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.015 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.015 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:02.015 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.015 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.015 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.015 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.015 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.015 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.015 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.015 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-19T01:28:02.015 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.016 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.017 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.018 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.019 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.019 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.019 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.019 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.019 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.019 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.019 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.019 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.019 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.019 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.019 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.019 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.020 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.020 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.020 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.020 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.020 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.020 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.020 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.020 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.020 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.020 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.020 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.020 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.021 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.022 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.023 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.023 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.025 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.026 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.026 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.026 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.026 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.026 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.026 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.026 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.026 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.026 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.026 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.026 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.026 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.026 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-19T01:28:02.027 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.028 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.029 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.030 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.031 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.032 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.032 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.032 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.032 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.032 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.032 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.032 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.032 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.032 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.032 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.032 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.032 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.032 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.033 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.034 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.034 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.034 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.034 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.034 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.034 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.034 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.034 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.034 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.034 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.034 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.034 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.035 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-19T01:28:02.036 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-19T01:28:02.037 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.038 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.039 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.040 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.041 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-19T01:28:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-19T01:28:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-19T01:28:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.042 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.043 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-19T01:28:02.043 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.043 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.043 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.043 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-19T01:28:02.043 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-19T01:28:02.043 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.043 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.043 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.043 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.043 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.043 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.044 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.045 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.046 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.047 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.048 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.049 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.050 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.051 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.052 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.053 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.054 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.055 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.055 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.055 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.055 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.055 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.055 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.055 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.055 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.055 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.055 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.056 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.056 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.056 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.056 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.056 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.056 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.056 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.056 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.056 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.056 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.056 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.056 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.056 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.057 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.058 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.059 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.059 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.059 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.059 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.059 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.059 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.059 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.059 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.059 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.059 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.059 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.059 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.059 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.060 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.061 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.062 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.063 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.064 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.065 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.066 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.067 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.067 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.067 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.067 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.067 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.067 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.067 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.067 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.067 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.067 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.067 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.067 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.067 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.068 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.069 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.070 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.071 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.072 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.072 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.072 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.072 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.072 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.072 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.072 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.072 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.072 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.072 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.072 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.072 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.073 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.074 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.075 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.076 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.077 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.078 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.079 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.079 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.079 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.079 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.079 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.079 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.079 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.079 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.079 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.079 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.079 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.079 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.079 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.080 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.081 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.082 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-19T01:28:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-19T01:28:02.083 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.084 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.085 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.086 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.087 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.088 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-19T01:28:02.089 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-19T01:28:02.090 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.090 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.090 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.090 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.090 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.090 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.090 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.090 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.090 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.090 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.090 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.090 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-19T01:28:02.091 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-19T01:28:02.093 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.094 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.094 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.094 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-19T01:28:02.094 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-19T01:28:02.094 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.094 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.094 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.094 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.094 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.094 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.094 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.095 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.096 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.096 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.096 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.096 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.096 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.096 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.096 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.096 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.096 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.096 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.096 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.096 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.096 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.097 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.098 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.098 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.098 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.098 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.098 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.098 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.098 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.098 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.098 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.098 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.098 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.098 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.099 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.099 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.099 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.099 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.099 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.099 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.099 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.099 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.099 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.099 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.099 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.099 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.100 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.101 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.102 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.103 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.104 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.104 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.104 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.104 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.104 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.104 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.104 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.104 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.104 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.104 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.104 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.104 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.104 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.105 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.106 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.106 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.106 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.106 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-19T01:28:02.106 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.106 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.106 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.106 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.106 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.106 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.106 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.106 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.107 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.108 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.109 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.110 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.111 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-19T01:28:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-19T01:28:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-19T01:28:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.112 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-19T01:28:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.113 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.114 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.115 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.116 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.117 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.118 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.119 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.120 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.121 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.122 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.123 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.124 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.125 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.126 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.127 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.128 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.129 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.129 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.129 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.129 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.129 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.129 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-19T01:28:02.129 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-19T01:28:02.129 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.129 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.129 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.130 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.131 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.132 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.133 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.134 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.135 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.136 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.137 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.138 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.139 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.139 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.139 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.139 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.139 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.139 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.139 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-19T01:28:02.139 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-19T01:28:02.139 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.139 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.139 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.139 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-19T01:28:02.139 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.140 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.140 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.140 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.140 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.140 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.140 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.140 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.140 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.140 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-19T01:28:02.140 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.140 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.141 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.142 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.142 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.142 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.142 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.142 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.142 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.142 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.142 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.142 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.142 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.142 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.142 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.142 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.143 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.144 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.144 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.144 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.144 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.144 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.144 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.144 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.144 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.144 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.144 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.144 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.144 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.144 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.145 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.146 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.147 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.148 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.149 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.150 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.151 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "periodic_async_work": { 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "periodic_async_work", 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.152 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.153 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.153 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.153 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.153 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Periodically check for async work", 2024-03-19T01:28:02.153 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.153 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.153 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.153 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.153 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-19T01:28:02.153 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-19T01:28:02.153 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.153 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.153 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.154 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:02.154 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.154 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.154 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.154 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-19T01:28:02.154 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.154 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.154 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.154 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.154 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-19T01:28:02.154 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-19T01:28:02.154 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.155 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.155 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.155 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.155 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.155 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.155 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.155 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-19T01:28:02.155 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.155 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.155 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.155 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.155 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.155 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.156 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.157 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.158 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.158 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.158 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.158 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.158 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:02.158 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:02.158 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.158 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.158 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.158 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:02.158 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.158 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.158 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.159 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.160 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.161 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.162 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.163 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.164 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-19T01:28:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-19T01:28:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-19T01:28:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.165 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.166 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.166 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.166 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.166 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.166 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.166 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-19T01:28:02.166 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-19T01:28:02.166 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.166 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.166 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.166 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-19T01:28:02.166 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.166 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.167 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.167 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.167 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.167 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.167 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.167 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.167 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.167 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.167 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T01:28:02.167 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.167 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.167 INFO:teuthology.orchestra.run.smithi116.stdout: "gid": 24113, 2024-03-19T01:28:02.167 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "y", 2024-03-19T01:28:02.168 INFO:teuthology.orchestra.run.smithi116.stdout: "mgr_features": 4540701547738038271, 2024-03-19T01:28:02.168 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-19T01:28:02.168 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.168 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-19T01:28:02.168 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.168 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.168 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.168 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:02.168 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:02.168 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.168 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.168 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.168 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:02.169 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.169 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.169 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.169 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-19T01:28:02.169 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.169 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.169 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.169 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.169 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.169 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.169 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.169 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.169 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.170 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.171 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.171 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.171 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.171 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.171 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.171 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.171 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.171 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.171 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.171 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.171 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.171 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.171 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.172 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.173 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.174 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.174 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.174 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.174 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.174 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.174 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.174 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.174 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.174 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.174 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.174 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-19T01:28:02.174 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-19T01:28:02.175 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-19T01:28:02.176 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.177 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.177 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.177 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.177 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.177 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.177 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.177 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-19T01:28:02.177 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.177 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.177 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.177 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.177 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-19T01:28:02.177 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-19T01:28:02.178 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-19T01:28:02.179 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-19T01:28:02.180 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.181 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.181 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.181 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.181 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.181 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.181 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.181 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-19T01:28:02.181 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.181 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.181 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.181 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.181 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-19T01:28:02.181 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-19T01:28:02.182 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.182 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.182 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.182 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.182 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.182 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.182 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.182 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-19T01:28:02.182 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.182 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.182 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.182 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.183 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.184 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.185 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.186 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-19T01:28:02.187 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-19T01:28:02.188 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-19T01:28:02.189 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-19T01:28:02.190 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.191 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.192 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.193 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.194 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.195 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-19T01:28:02.196 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-19T01:28:02.197 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-19T01:28:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: "read", 2024-03-19T01:28:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap", 2024-03-19T01:28:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap-read" 2024-03-19T01:28:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-19T01:28:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-19T01:28:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-19T01:28:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.198 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.199 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.200 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:02.201 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.202 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.203 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.204 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-19T01:28:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-19T01:28:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-19T01:28:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-19T01:28:02.205 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.206 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.206 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.206 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.206 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-19T01:28:02.206 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-19T01:28:02.206 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.206 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.206 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.206 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.206 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.206 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.206 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.207 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.208 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm_log_destination": { 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm_log_destination", 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.209 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.210 INFO:teuthology.orchestra.run.smithi116.stdout: "file", 2024-03-19T01:28:02.210 INFO:teuthology.orchestra.run.smithi116.stdout: "file,syslog", 2024-03-19T01:28:02.210 INFO:teuthology.orchestra.run.smithi116.stdout: "syslog" 2024-03-19T01:28:02.210 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.210 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-19T01:28:02.210 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.210 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.210 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.210 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.210 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-19T01:28:02.210 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-19T01:28:02.210 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.210 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.211 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.212 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.212 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.212 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.212 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.212 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-19T01:28:02.212 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.212 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.212 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.212 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.212 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-19T01:28:02.212 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-19T01:28:02.212 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.212 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.213 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.214 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.215 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.216 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.217 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-19T01:28:02.218 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.218 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.218 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.218 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-19T01:28:02.218 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.218 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.218 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.218 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.218 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-19T01:28:02.218 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-19T01:28:02.218 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.218 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.219 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.219 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-19T01:28:02.219 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.219 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.219 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.219 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-19T01:28:02.219 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.219 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.219 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.219 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.219 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-19T01:28:02.220 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-19T01:28:02.221 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.221 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.221 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.221 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-19T01:28:02.221 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.221 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.221 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.221 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-19T01:28:02.221 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.221 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.221 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.221 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.221 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-19T01:28:02.222 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-19T01:28:02.222 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.222 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.222 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.222 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-19T01:28:02.222 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.222 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.222 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.222 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-19T01:28:02.222 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.222 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.222 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.222 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.223 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.224 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-19T01:28:02.225 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-19T01:28:02.226 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-19T01:28:02.227 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.227 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.227 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.227 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-19T01:28:02.227 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.227 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.227 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.227 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-19T01:28:02.227 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.227 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.227 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.227 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.227 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-19T01:28:02.228 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.229 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.230 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.231 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.232 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.233 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.234 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.234 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.234 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.234 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-19T01:28:02.234 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.234 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.234 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.234 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.234 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-19T01:28:02.234 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-19T01:28:02.234 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.234 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.234 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.235 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:02.235 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.235 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.235 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.235 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-19T01:28:02.235 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.235 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.235 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.235 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.235 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-19T01:28:02.235 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-19T01:28:02.235 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.235 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.236 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.237 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.238 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.239 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.240 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.241 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.242 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.242 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.242 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.242 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.242 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.242 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.242 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.242 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.242 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.242 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.242 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.242 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.243 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.243 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.243 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.243 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.243 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.243 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.243 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.243 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.243 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.243 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.243 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.243 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.244 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.246 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.247 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.248 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.248 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.248 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-19T01:28:02.248 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.248 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.248 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.248 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.248 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-19T01:28:02.248 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-19T01:28:02.248 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.248 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.248 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.248 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.249 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.249 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.249 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.249 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-19T01:28:02.249 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.249 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.249 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.249 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.249 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-19T01:28:02.249 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-19T01:28:02.249 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.249 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-19T01:28:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-19T01:28:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-19T01:28:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-19T01:28:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.250 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.251 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-19T01:28:02.252 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-19T01:28:02.252 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.252 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.252 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.252 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-19T01:28:02.252 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.252 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.252 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.252 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-19T01:28:02.252 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.252 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.252 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.252 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.253 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-19T01:28:02.254 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-19T01:28:02.255 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-19T01:28:02.256 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-19T01:28:02.257 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-19T01:28:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-19T01:28:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-19T01:28:02.258 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-19T01:28:02.259 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-19T01:28:02.260 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-19T01:28:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-19T01:28:02.261 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-19T01:28:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-19T01:28:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.262 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.263 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.264 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.265 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.266 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.267 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.268 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.269 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.269 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.269 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.269 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.269 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.269 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.269 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.269 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.269 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.269 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.269 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.269 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.269 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.270 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.271 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-19T01:28:02.272 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.273 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.273 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.273 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.273 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.273 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.273 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.273 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.273 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.273 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.273 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.273 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.274 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-19T01:28:02.274 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-19T01:28:02.274 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.274 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.274 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.274 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.274 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.274 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.274 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.274 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.274 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.274 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.274 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.275 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.276 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-19T01:28:02.277 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-19T01:28:02.277 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.277 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.277 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.277 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.277 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.277 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.277 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.277 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.277 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.277 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.277 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.277 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.278 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.279 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-19T01:28:02.280 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-19T01:28:02.280 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.280 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.280 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.280 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.280 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.280 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.280 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.280 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.280 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.280 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.280 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.280 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.281 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-19T01:28:02.282 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-19T01:28:02.282 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.282 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.282 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.282 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.282 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.282 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.282 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.282 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.282 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.282 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.282 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.283 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.284 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-19T01:28:02.285 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-19T01:28:02.286 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-19T01:28:02.287 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.287 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.287 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.287 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T01:28:02.287 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.287 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.287 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.287 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.287 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.287 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.287 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.287 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.287 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.288 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.289 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-19T01:28:02.290 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-19T01:28:02.290 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.290 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.290 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.290 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T01:28:02.290 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.290 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.290 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.290 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.290 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.290 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.290 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.290 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.291 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.292 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-19T01:28:02.293 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-19T01:28:02.294 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.294 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.294 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.294 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.294 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.294 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.294 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.294 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.294 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.294 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.294 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.294 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.295 INFO:teuthology.orchestra.run.smithi116.stdout: "MULTICLUSTER_CONFIG": { 2024-03-19T01:28:02.295 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-19T01:28:02.295 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.295 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.295 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.295 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "{}", 2024-03-19T01:28:02.295 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.295 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.295 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.295 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.295 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.295 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.295 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.296 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.297 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.298 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.299 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.300 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.301 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.302 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.303 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-19T01:28:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-19T01:28:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.304 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.305 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.306 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.307 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-19T01:28:02.308 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-19T01:28:02.308 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.308 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.308 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.308 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-19T01:28:02.308 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.308 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.308 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.308 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.308 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.308 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.308 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.309 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.309 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-19T01:28:02.309 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-19T01:28:02.309 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.309 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.309 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.309 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-19T01:28:02.309 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.309 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.309 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.309 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.309 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.309 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.310 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-19T01:28:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-19T01:28:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T01:28:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.311 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.312 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.312 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.312 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.312 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-19T01:28:02.312 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-19T01:28:02.312 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.312 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.312 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.312 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.312 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.312 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.312 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.312 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.313 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.314 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.314 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.314 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.314 INFO:teuthology.orchestra.run.smithi116.stdout: "UNSAFE_TLS_v1_2": { 2024-03-19T01:28:02.314 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-19T01:28:02.314 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.314 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.314 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.314 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.314 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.314 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.314 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.314 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.315 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.315 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.315 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.315 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.315 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-19T01:28:02.315 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-19T01:28:02.315 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.315 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.315 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.315 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:02.315 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.315 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.315 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.316 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.316 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.316 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.316 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.316 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.316 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-19T01:28:02.316 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-19T01:28:02.316 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.316 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.316 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.316 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.316 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.316 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.317 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.317 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.317 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.317 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.317 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.317 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.317 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-19T01:28:02.317 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-19T01:28:02.317 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.317 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.317 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.317 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T01:28:02.318 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.318 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.318 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.318 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.318 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.318 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.318 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.318 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.318 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-19T01:28:02.318 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-19T01:28:02.318 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.318 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.318 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.319 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.320 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.321 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-19T01:28:02.322 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.322 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.322 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.322 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.322 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.322 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.322 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.322 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.322 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-19T01:28:02.322 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-19T01:28:02.322 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.322 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.323 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.324 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.325 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.325 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.325 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.325 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.325 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.325 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.325 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.325 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.325 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.325 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.325 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.325 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.325 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.326 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.327 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.327 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.327 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.327 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.327 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.327 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.327 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.327 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.327 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.327 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.327 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.327 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.327 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-19T01:28:02.328 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-19T01:28:02.329 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.329 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.329 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.329 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.329 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.329 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.329 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.329 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-19T01:28:02.329 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.329 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.329 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.329 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.329 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.330 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T01:28:02.331 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-19T01:28:02.332 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-19T01:28:02.333 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-19T01:28:02.334 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.335 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-19T01:28:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-19T01:28:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-19T01:28:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-19T01:28:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-19T01:28:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.336 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.337 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-19T01:28:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-19T01:28:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-19T01:28:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.338 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.339 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.340 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.341 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.342 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.343 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-19T01:28:02.344 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-19T01:28:02.345 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-19T01:28:02.346 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-19T01:28:02.347 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-19T01:28:02.348 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-19T01:28:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:02.349 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-19T01:28:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.350 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.351 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.352 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.353 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.354 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.355 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.356 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.357 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-19T01:28:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-19T01:28:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-19T01:28:02.358 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:02.359 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-19T01:28:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-19T01:28:02.360 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-19T01:28:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-19T01:28:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-19T01:28:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-19T01:28:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-19T01:28:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.361 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.362 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-19T01:28:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-19T01:28:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.363 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-19T01:28:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-19T01:28:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-19T01:28:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.364 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.365 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.366 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.367 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.368 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.369 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.370 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.371 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.372 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-19T01:28:02.373 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.374 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.375 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.376 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.377 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.378 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.379 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.380 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.381 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-19T01:28:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.382 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.383 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.384 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.385 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.386 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.387 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.388 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.389 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.390 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.391 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.392 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.393 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.394 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.395 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-19T01:28:02.396 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-19T01:28:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.397 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.398 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.399 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.400 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.401 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.402 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.403 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.404 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-19T01:28:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-19T01:28:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-19T01:28:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.405 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-19T01:28:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-19T01:28:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-19T01:28:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.406 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-19T01:28:02.407 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.408 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.409 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.410 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.411 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.412 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.413 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.414 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.415 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.416 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.417 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.418 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.419 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.420 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.421 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.422 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.423 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.424 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.425 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.426 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.427 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.428 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.429 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.430 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.431 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.432 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.433 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.434 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.434 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.434 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.434 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.434 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.434 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.434 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.434 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.434 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.434 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.434 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.434 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.434 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.435 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.435 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.435 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.435 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.435 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.435 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.435 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.435 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.435 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.435 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.435 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.435 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.435 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.436 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.437 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.437 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.437 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.437 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.437 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.437 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-19T01:28:02.437 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.437 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.437 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.437 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.437 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.437 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.438 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.438 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.438 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.438 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.438 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.438 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.438 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.438 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.438 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.438 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.438 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.438 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.439 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.439 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.439 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.439 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.439 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.439 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.439 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.439 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.439 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.439 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.439 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.439 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.439 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.440 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.441 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.442 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.443 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.444 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.445 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.447 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.448 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.449 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.450 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.451 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.452 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.453 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.454 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.455 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.456 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.457 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T01:28:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-19T01:28:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-19T01:28:02.458 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.459 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-19T01:28:02.460 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.461 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.462 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.463 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.464 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.465 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-19T01:28:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-19T01:28:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.466 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-19T01:28:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-19T01:28:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-19T01:28:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.467 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-19T01:28:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-19T01:28:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-19T01:28:02.468 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.469 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-19T01:28:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T01:28:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T01:28:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-19T01:28:02.470 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.471 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.472 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-19T01:28:02.473 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-19T01:28:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-19T01:28:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-19T01:28:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.474 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.475 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.476 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.477 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.478 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.479 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.480 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.481 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.482 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-19T01:28:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-19T01:28:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-19T01:28:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.483 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.484 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.485 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.486 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.487 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.488 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.489 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.490 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T01:28:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T01:28:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.491 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.492 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.493 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.494 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.495 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.496 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.497 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.498 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.499 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.500 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.500 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.500 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.500 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.500 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.500 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.500 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.500 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.500 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.500 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.500 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.500 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.500 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.501 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.502 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.503 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.504 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.505 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.506 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.507 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.508 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.509 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.510 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-19T01:28:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-19T01:28:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.511 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.512 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.513 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.514 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-19T01:28:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-19T01:28:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.515 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.516 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.517 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.518 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.519 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.521 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.522 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-19T01:28:02.523 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.524 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.525 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.526 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.527 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.528 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.529 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-19T01:28:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.530 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.531 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.532 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.533 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.534 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.535 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.536 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.537 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.538 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.539 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.540 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.541 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.542 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-19T01:28:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.543 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.544 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.545 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.546 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.547 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.548 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-19T01:28:02.549 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.550 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.551 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-19T01:28:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-19T01:28:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.552 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-19T01:28:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-19T01:28:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-19T01:28:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.553 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.554 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.555 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.556 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-19T01:28:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-19T01:28:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.557 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-19T01:28:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-19T01:28:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-19T01:28:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.558 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.559 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.560 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-19T01:28:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-19T01:28:02.561 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-19T01:28:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.562 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-19T01:28:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-19T01:28:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.563 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.564 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.565 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.566 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.567 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.568 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.569 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.570 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.571 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.572 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.573 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.574 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.575 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.576 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.577 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.578 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.579 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.580 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.581 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.582 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "periodic_async_work": { 2024-03-19T01:28:02.583 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "periodic_async_work", 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Periodically check for async work", 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-19T01:28:02.584 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-19T01:28:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-19T01:28:02.585 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-19T01:28:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-19T01:28:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.586 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-19T01:28:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-19T01:28:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-19T01:28:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-19T01:28:02.587 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.588 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.589 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.590 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.591 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.592 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.593 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.594 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.595 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.596 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-19T01:28:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-19T01:28:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-19T01:28:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.597 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-19T01:28:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-19T01:28:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-19T01:28:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.598 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T01:28:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "modules": [ 2024-03-19T01:28:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-19T01:28:02.599 INFO:teuthology.orchestra.run.smithi116.stdout: "dashboard", 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "iostat", 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "nfs", 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "restful" 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "available_modules": [ 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "alerts", 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.600 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How frequently to reexamine health status", 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.601 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.602 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.604 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.605 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.606 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_destination": { 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_destination", 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email address to send alerts to", 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.607 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_from_name": { 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_from_name", 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "Ceph", 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Email From: name", 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.608 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_host": { 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_host", 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP server", 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.609 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_password": { 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_password", 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Password to authenticate with", 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_port": { 2024-03-19T01:28:02.610 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_port", 2024-03-19T01:28:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "465", 2024-03-19T01:28:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP port", 2024-03-19T01:28:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.611 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_sender": { 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_sender", 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SMTP envelope sender", 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.612 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_ssl": { 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_ssl", 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.613 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "smtp_user": { 2024-03-19T01:28:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "smtp_user", 2024-03-19T01:28:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "User to authenticate as", 2024-03-19T01:28:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.614 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "balancer", 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "active": { 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "active", 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.615 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically balance PGs across cluster", 2024-03-19T01:28:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_time": { 2024-03-19T01:28:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_time", 2024-03-19T01:28:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.616 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0000", 2024-03-19T01:28:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "beginning time of day to automatically balance", 2024-03-19T01:28:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T01:28:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "begin_weekday": { 2024-03-19T01:28:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "begin_weekday", 2024-03-19T01:28:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:02.617 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-19T01:28:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-19T01:28:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-19T01:28:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T01:28:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_max_iterations": { 2024-03-19T01:28:02.618 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_max_iterations", 2024-03-19T01:28:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "25", 2024-03-19T01:28:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T01:28:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "250", 2024-03-19T01:28:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-19T01:28:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.619 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_metrics": { 2024-03-19T01:28:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_metrics", 2024-03-19T01:28:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "pgs,objects,bytes", 2024-03-19T01:28:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-19T01:28:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-19T01:28:02.620 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "crush_compat_step": { 2024-03-19T01:28:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crush_compat_step", 2024-03-19T01:28:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.5", 2024-03-19T01:28:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0.001", 2024-03-19T01:28:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "0.999", 2024-03-19T01:28:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "aggressiveness of optimization", 2024-03-19T01:28:02.621 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "end_time": { 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_time", 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2359", 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ending time of day to automatically balance", 2024-03-19T01:28:02.622 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "end_weekday": { 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "end_weekday", 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "0", 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "6", 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.623 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.624 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.625 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.626 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.627 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.628 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "min_score": { 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_score", 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-19T01:28:02.629 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "upmap", 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "crush-compat", 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "none", 2024-03-19T01:28:02.630 INFO:teuthology.orchestra.run.smithi116.stdout: "read", 2024-03-19T01:28:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap", 2024-03-19T01:28:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap-read" 2024-03-19T01:28:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Balancer mode", 2024-03-19T01:28:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_ids": { 2024-03-19T01:28:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_ids", 2024-03-19T01:28:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.631 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-19T01:28:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:02.632 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_deviation": { 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_deviation", 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.633 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T01:28:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T01:28:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-19T01:28:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-19T01:28:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "upmap_max_optimizations": { 2024-03-19T01:28:02.634 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "upmap_max_optimizations", 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.635 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm", 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_down_multiplier": { 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_down_multiplier", 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.636 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-19T01:28:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_refresh_rate": { 2024-03-19T01:28:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_refresh_rate", 2024-03-19T01:28:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "20", 2024-03-19T01:28:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.637 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-19T01:28:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "agent_starting_port": { 2024-03-19T01:28:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "agent_starting_port", 2024-03-19T01:28:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4721", 2024-03-19T01:28:02.638 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_ptrace": { 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_ptrace", 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.639 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-19T01:28:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-19T01:28:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_interval": { 2024-03-19T01:28:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_interval", 2024-03-19T01:28:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.640 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:02.641 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "autotune_memory_target_ratio": { 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "autotune_memory_target_ratio", 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0.7", 2024-03-19T01:28:02.642 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm_log_destination": { 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cephadm_log_destination", 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.643 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "file", 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "file,syslog", 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "syslog" 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "cgroups_split": { 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cgroups_split", 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.644 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "config_checks_enabled": { 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_checks_enabled", 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.645 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-19T01:28:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "config_dashboard": { 2024-03-19T01:28:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_dashboard", 2024-03-19T01:28:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.646 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_alertmanager": { 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_alertmanager", 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.647 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_base": { 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_base", 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.648 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-19T01:28:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Container image name, without the tag", 2024-03-19T01:28:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_elasticsearch": { 2024-03-19T01:28:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_elasticsearch", 2024-03-19T01:28:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.649 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-19T01:28:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "elasticsearch container image", 2024-03-19T01:28:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_grafana": { 2024-03-19T01:28:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_grafana", 2024-03-19T01:28:02.650 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-19T01:28:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T01:28:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_haproxy": { 2024-03-19T01:28:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_haproxy", 2024-03-19T01:28:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-19T01:28:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "HAproxy container image", 2024-03-19T01:28:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.652 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_agent": { 2024-03-19T01:28:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_agent", 2024-03-19T01:28:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-19T01:28:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger agent container image", 2024-03-19T01:28:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.653 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_collector": { 2024-03-19T01:28:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_collector", 2024-03-19T01:28:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-19T01:28:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger collector container image", 2024-03-19T01:28:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.654 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_jaeger_query": { 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_jaeger_query", 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Jaeger query container image", 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.655 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_keepalived": { 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_keepalived", 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Keepalived container image", 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.656 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_loki": { 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_loki", 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Loki container image", 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_node_exporter": { 2024-03-19T01:28:02.657 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_node_exporter", 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_nvmeof": { 2024-03-19T01:28:02.658 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_nvmeof", 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Nvme-of container image", 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_prometheus": { 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_prometheus", 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.659 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Prometheus container image", 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_promtail": { 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_promtail", 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.660 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Promtail container image", 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "container_image_snmp_gateway": { 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_image_snmp_gateway", 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-19T01:28:02.661 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "SNMP Gateway container image", 2024-03-19T01:28:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "container_init": { 2024-03-19T01:28:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "container_init", 2024-03-19T01:28:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.662 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Run podman/docker with `--init`", 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "daemon_cache_timeout": { 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "daemon_cache_timeout", 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:02.663 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-19T01:28:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "default_cephadm_command_timeout": { 2024-03-19T01:28:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_cephadm_command_timeout", 2024-03-19T01:28:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.664 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "900", 2024-03-19T01:28:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-19T01:28:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "default_registry": { 2024-03-19T01:28:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "default_registry", 2024-03-19T01:28:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.665 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "docker.io", 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "device_cache_timeout": { 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_cache_timeout", 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.666 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1800", 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache device inventory", 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "device_enhanced_scan": { 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_enhanced_scan", 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.667 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-19T01:28:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "facts_cache_timeout": { 2024-03-19T01:28:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "facts_cache_timeout", 2024-03-19T01:28:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.668 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "seconds to cache host facts data", 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "host_check_interval": { 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "host_check_interval", 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to perform a host check", 2024-03-19T01:28:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "hw_monitoring": { 2024-03-19T01:28:02.670 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hw_monitoring", 2024-03-19T01:28:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-19T01:28:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.671 INFO:teuthology.orchestra.run.smithi116.stdout: "inventory_list_all": { 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "inventory_list_all", 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.672 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.673 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "log_refresh_metadata": { 2024-03-19T01:28:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_refresh_metadata", 2024-03-19T01:28:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.674 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.675 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-19T01:28:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.676 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.677 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.678 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.679 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "*", 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.680 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "max_count_per_host": { 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_count_per_host", 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of daemons per service per host", 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.681 INFO:teuthology.orchestra.run.smithi116.stdout: "max_osd_draining_count": { 2024-03-19T01:28:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_osd_draining_count", 2024-03-19T01:28:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-19T01:28:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.682 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "migration_current": { 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "migration_current", 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "internal - do not modify", 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.683 INFO:teuthology.orchestra.run.smithi116.stdout: "mode": { 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mode", 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "root", 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm-package", 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "root" 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "mode for remote execution of cephadm", 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.684 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "oob_default_addr": { 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "oob_default_addr", 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "169.254.1.1", 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.685 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "prometheus_alerts_path": { 2024-03-19T01:28:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus_alerts_path", 2024-03-19T01:28:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-19T01:28:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-19T01:28:02.686 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_insecure": { 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_insecure", 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.687 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_password": { 2024-03-19T01:28:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_password", 2024-03-19T01:28:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-19T01:28:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.688 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_url": { 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_url", 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.689 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "registry_username": { 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "registry_username", 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.690 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "secure_monitoring_stack": { 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "secure_monitoring_stack", 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.691 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "service_discovery_port": { 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "service_discovery_port", 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8765", 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "cephadm service discovery port", 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.692 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "ssh_config_file": { 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssh_config_file", 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.693 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "use_agent": { 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_agent", 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.694 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "use_repo_digest": { 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "use_repo_digest", 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.695 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_failed_host_check": { 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_failed_host_check", 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if the host check fails", 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_daemons": { 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_daemons", 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_on_stray_hosts": { 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_on_stray_hosts", 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.697 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-19T01:28:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crash", 2024-03-19T01:28:02.698 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.699 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.700 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.701 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.702 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.703 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "retain_interval": { 2024-03-19T01:28:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retain_interval", 2024-03-19T01:28:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "31536000", 2024-03-19T01:28:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-19T01:28:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.704 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_recent_interval": { 2024-03-19T01:28:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_recent_interval", 2024-03-19T01:28:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1209600", 2024-03-19T01:28:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-19T01:28:02.705 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dashboard", 2024-03-19T01:28:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-19T01:28:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-19T01:28:02.706 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_HOST": { 2024-03-19T01:28:02.707 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-19T01:28:02.708 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_ENABLED": { 2024-03-19T01:28:02.709 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_ENABLED", 2024-03-19T01:28:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.710 INFO:teuthology.orchestra.run.smithi116.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "ENABLE_BROWSABLE_API": { 2024-03-19T01:28:02.711 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-19T01:28:02.712 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.713 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.714 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.715 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.716 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.717 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-19T01:28:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-19T01:28:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.718 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-19T01:28:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-19T01:28:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.719 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_PASSWORD": { 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.720 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T01:28:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-19T01:28:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-19T01:28:02.721 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_URL": { 2024-03-19T01:28:02.722 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_URL", 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_API_USERNAME": { 2024-03-19T01:28:02.723 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-19T01:28:02.724 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-19T01:28:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.725 INFO:teuthology.orchestra.run.smithi116.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-19T01:28:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-19T01:28:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.726 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.727 INFO:teuthology.orchestra.run.smithi116.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.728 INFO:teuthology.orchestra.run.smithi116.stdout: "MULTICLUSTER_CONFIG": { 2024-03-19T01:28:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-19T01:28:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "{}", 2024-03-19T01:28:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.729 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_HOST": { 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.730 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.731 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.732 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.733 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.734 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-19T01:28:02.735 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-19T01:28:02.736 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.737 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_ENABLED": { 2024-03-19T01:28:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-19T01:28:02.738 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-19T01:28:02.739 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-19T01:28:02.740 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-19T01:28:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.741 INFO:teuthology.orchestra.run.smithi116.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-19T01:28:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-19T01:28:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8", 2024-03-19T01:28:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.742 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "45", 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.743 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ACCESS_KEY": { 2024-03-19T01:28:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-19T01:28:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.744 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "admin", 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.745 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SECRET_KEY": { 2024-03-19T01:28:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-19T01:28:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.746 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "RGW_API_SSL_VERIFY": { 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.747 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "UNSAFE_TLS_v1_2": { 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.748 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.749 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.750 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.751 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "cross_origin_url": { 2024-03-19T01:28:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cross_origin_url", 2024-03-19T01:28:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.752 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "crt_file": { 2024-03-19T01:28:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "crt_file", 2024-03-19T01:28:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.753 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "debug": { 2024-03-19T01:28:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "debug", 2024-03-19T01:28:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.754 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Enable/disable debug options", 2024-03-19T01:28:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "jwt_token_ttl": { 2024-03-19T01:28:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "jwt_token_ttl", 2024-03-19T01:28:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.755 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "28800", 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.756 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.757 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.758 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.759 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.760 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.761 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "motd": { 2024-03-19T01:28:02.762 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "motd", 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "The message of the day", 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect_resolve_ip_addr": { 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "redirect_resolve_ip_addr", 2024-03-19T01:28:02.763 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T01:28:02.764 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.765 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8080", 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.766 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl_server_port": { 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl_server_port", 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.767 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8443", 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.768 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "redirect", 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "redirect" 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.769 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.770 INFO:teuthology.orchestra.run.smithi116.stdout: "url_prefix": { 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url_prefix", 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.771 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_monitoring": { 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_monitoring", 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.772 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "monitor device health metrics", 2024-03-19T01:28:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.773 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.774 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.775 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.777 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.778 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "mark_out_threshold": { 2024-03-19T01:28:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mark_out_threshold", 2024-03-19T01:28:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "2419200", 2024-03-19T01:28:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-19T01:28:02.779 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "pool_name": { 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pool_name", 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "device_health_metrics", 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-19T01:28:02.780 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "retention_period": { 2024-03-19T01:28:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "retention_period", 2024-03-19T01:28:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15552000", 2024-03-19T01:28:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.781 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long to retain device health metrics", 2024-03-19T01:28:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.782 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_frequency": { 2024-03-19T01:28:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_frequency", 2024-03-19T01:28:02.783 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-19T01:28:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-19T01:28:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "self_heal": { 2024-03-19T01:28:02.784 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "self_heal", 2024-03-19T01:28:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-19T01:28:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.785 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how frequently to wake up and check device health", 2024-03-19T01:28:02.786 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "warn_threshold": { 2024-03-19T01:28:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "warn_threshold", 2024-03-19T01:28:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7257600", 2024-03-19T01:28:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.787 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "diskprediction_local", 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.788 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.789 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.790 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.791 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.792 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "predict_interval": { 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predict_interval", 2024-03-19T01:28:02.793 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "86400", 2024-03-19T01:28:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "predictor_model": { 2024-03-19T01:28:02.794 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "predictor_model", 2024-03-19T01:28:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "prophetstor", 2024-03-19T01:28:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.795 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "600", 2024-03-19T01:28:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.796 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "influx", 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": false, 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "influxdb python module not found", 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "batch_size": { 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "batch_size", 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.797 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5000", 2024-03-19T01:28:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-19T01:28:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "database": { 2024-03-19T01:28:02.798 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "database", 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "ceph", 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": { 2024-03-19T01:28:02.799 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "hostname", 2024-03-19T01:28:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server hostname", 2024-03-19T01:28:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.800 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "30", 2024-03-19T01:28:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "5", 2024-03-19T01:28:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-19T01:28:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.801 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.802 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.803 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.804 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.805 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.806 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "password": { 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "password", 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "password of InfluxDB server user", 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.807 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "port": { 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "port", 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "8086", 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "InfluxDB server port", 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.808 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "ssl": { 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ssl", 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "false", 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.809 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "threads": { 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threads", 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "32", 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.810 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "username": { 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "username", 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "username of InfluxDB server user", 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.811 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "verify_ssl": { 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "verify_ssl", 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "true", 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.812 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "insights", 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.813 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.814 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.815 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.816 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.817 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "iostat", 2024-03-19T01:28:02.818 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.819 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.820 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.821 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.822 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.823 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "k8sevents", 2024-03-19T01:28:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_event_retention_days": { 2024-03-19T01:28:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "ceph_event_retention_days", 2024-03-19T01:28:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.824 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "7", 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "config_check_secs": { 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "config_check_secs", 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.825 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "10", 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.826 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.827 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.828 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.829 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.830 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.831 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "localpool", 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "failure_domain": { 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "failure_domain", 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "host", 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "failure domain for any created local pool", 2024-03-19T01:28:02.832 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.833 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.834 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.835 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.836 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.837 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "min_size": { 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "min_size", 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default min_size for any created local pool", 2024-03-19T01:28:02.838 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "num_rep": { 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "num_rep", 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3", 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default replica count for any created local pool", 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.839 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_num": { 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_num", 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "128", 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "default pg_num for any created local pool", 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.840 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "prefix": { 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prefix", 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "name prefix for any created local pool", 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.841 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "subtree": { 2024-03-19T01:28:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "subtree", 2024-03-19T01:28:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "rack", 2024-03-19T01:28:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-19T01:28:02.842 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mds_autoscaler", 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.843 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.844 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.845 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.846 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.847 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.848 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirroring", 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.849 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.850 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.851 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.852 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.853 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.854 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "nfs", 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.855 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.856 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.857 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.858 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.859 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "fail_fs": { 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "fail_fs", 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.860 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.861 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.862 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.863 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.864 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.865 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator": { 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "orchestrator", 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.866 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "cephadm", 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "rook", 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "test_orchestrator" 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Orchestrator backend", 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.867 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_perf_query", 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.869 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.870 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.871 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.872 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "osd_support", 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.873 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.874 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.875 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.876 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.877 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.878 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "pg_autoscaler", 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.879 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.880 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.881 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.882 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.883 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.884 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "threshold": { 2024-03-19T01:28:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "threshold", 2024-03-19T01:28:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.885 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "3.0", 2024-03-19T01:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1.0", 2024-03-19T01:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "scaling threshold", 2024-03-19T01:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-19T01:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "progress", 2024-03-19T01:28:02.886 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_pg_recovery_event": { 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_pg_recovery_event", 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.887 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.888 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.889 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.890 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.891 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.892 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.893 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "max_completed_events": { 2024-03-19T01:28:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_completed_events", 2024-03-19T01:28:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "50", 2024-03-19T01:28:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "number of past completed events to remember", 2024-03-19T01:28:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.894 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "sleep_interval": { 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "sleep_interval", 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "5", 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "how long the module is going to sleep", 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.895 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "prometheus", 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "cache": { 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "cache", 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.896 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "exclude_perf_counters": { 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "exclude_perf_counters", 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.897 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-19T01:28:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-19T01:28:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.898 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.899 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.900 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.901 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.902 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools": { 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools", 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.903 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "300", 2024-03-19T01:28:02.904 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "scrape_interval": { 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "scrape_interval", 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15.0", 2024-03-19T01:28:02.905 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "::", 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.906 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "9283", 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.907 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "stale_cache_strategy": { 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stale_cache_strategy", 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "log", 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.908 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_behaviour": { 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_behaviour", 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "default", 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.909 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "default", 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "error" 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "standby_error_status_code": { 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "standby_error_status_code", 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.910 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "500", 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "400", 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "599", 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.911 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.912 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.913 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.914 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.915 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.916 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_snap_create": { 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_snap_create", 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10", 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.917 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "mirror_snapshot_schedule": { 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "mirror_snapshot_schedule", 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.918 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "trash_purge_schedule": { 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "trash_purge_schedule", 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.919 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "restful", 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "enable_auth": { 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enable_auth", 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.920 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "key_file": { 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "key_file", 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.921 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.922 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.923 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.924 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.925 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.926 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "server_addr": { 2024-03-19T01:28:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_addr", 2024-03-19T01:28:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.927 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "server_port": { 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "server_port", 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.928 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rgw", 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.929 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.930 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.931 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.932 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.933 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.934 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rook", 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.935 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.936 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.937 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.938 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.939 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "storage_class": { 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "storage_class", 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "local", 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.940 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "selftest", 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.941 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.942 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.943 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.944 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.945 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "roption1": { 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption1", 2024-03-19T01:28:02.946 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "roption2": { 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "roption2", 2024-03-19T01:28:02.947 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "xyz", 2024-03-19T01:28:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption1": { 2024-03-19T01:28:02.948 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption1", 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption2": { 2024-03-19T01:28:02.949 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption2", 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption3": { 2024-03-19T01:28:02.950 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption3", 2024-03-19T01:28:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "float", 2024-03-19T01:28:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.951 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption4": { 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption4", 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.952 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption5": { 2024-03-19T01:28:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption5", 2024-03-19T01:28:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.953 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption6": { 2024-03-19T01:28:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption6", 2024-03-19T01:28:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.954 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "rwoption7": { 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rwoption7", 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "1", 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "42", 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.955 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "testkey": { 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testkey", 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.956 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "testlkey": { 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testlkey", 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.957 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "testnewline": { 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "testnewline", 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.958 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snap_schedule", 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "allow_m_granularity": { 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "allow_m_granularity", 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.959 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "allow minute scheduled snapshots", 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "dump_on_update": { 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "dump_on_update", 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.960 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "dump database to debug log on update", 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.961 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.962 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.963 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.964 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.965 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.966 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "stats", 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.967 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.970 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.971 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "status", 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.972 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.973 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.974 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.975 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telegraf", 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "address": { 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "address", 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.978 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "15", 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.979 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.980 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:02.981 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:02.982 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:02.983 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.984 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "telemetry", 2024-03-19T01:28:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_basic": { 2024-03-19T01:28:02.985 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_basic", 2024-03-19T01:28:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-19T01:28:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.986 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_crash": { 2024-03-19T01:28:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_crash", 2024-03-19T01:28:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-19T01:28:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_device": { 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_device", 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.988 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_ident": { 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_ident", 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.989 INFO:teuthology.orchestra.run.smithi116.stdout: "channel_perf": { 2024-03-19T01:28:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "channel_perf", 2024-03-19T01:28:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-19T01:28:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "contact": { 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "contact", 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "description": { 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "description", 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.992 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "device_url": { 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "device_url", 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "enabled": { 2024-03-19T01:28:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "enabled", 2024-03-19T01:28:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.994 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "24", 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "8", 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.995 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "last_opt_revision": { 2024-03-19T01:28:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "last_opt_revision", 2024-03-19T01:28:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "1", 2024-03-19T01:28:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.996 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard": { 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard", 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.997 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "leaderboard_description": { 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "leaderboard_description", 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:02.998 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:02.999 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.000 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.001 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.002 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.003 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "organization": { 2024-03-19T01:28:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "organization", 2024-03-19T01:28:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.004 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "proxy": { 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "proxy", 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.005 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "url": { 2024-03-19T01:28:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "url", 2024-03-19T01:28:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-19T01:28:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.006 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "test_orchestrator", 2024-03-19T01:28:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:03.007 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:03.008 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.009 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.010 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:03.011 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.012 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:03.013 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:03.014 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:03.015 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:03.016 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.017 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "max_concurrent_clones": { 2024-03-19T01:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "max_concurrent_clones", 2024-03-19T01:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "4", 2024-03-19T01:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.018 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "periodic_async_work": { 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "periodic_async_work", 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.019 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Periodically check for async work", 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_delay": { 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_delay", 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "0", 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.020 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-19T01:28:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "snapshot_clone_no_wait": { 2024-03-19T01:28:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "snapshot_clone_no_wait", 2024-03-19T01:28:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "True", 2024-03-19T01:28:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.021 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix", 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "can_run": true, 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "error_string": "", 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "module_options": { 2024-03-19T01:28:03.022 INFO:teuthology.orchestra.run.smithi116.stdout: "discovery_interval": { 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "discovery_interval", 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "uint", 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "100", 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "identifier": { 2024-03-19T01:28:03.023 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "identifier", 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "interval": { 2024-03-19T01:28:03.024 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "interval", 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "secs", 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "60", 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "log_level": { 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_level", 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:03.025 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:03.026 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster": { 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster", 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.027 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_cluster_level": { 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_cluster_level", 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "info", 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [ 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "", 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "critical", 2024-03-19T01:28:03.028 INFO:teuthology.orchestra.run.smithi116.stdout: "debug", 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "error", 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "info", 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "warning" 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "log_to_file": { 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "log_to_file", 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "bool", 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 1, 2024-03-19T01:28:03.029 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "False", 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_host": { 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_host", 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:03.030 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "", 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_port": { 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_port", 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "int", 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "10051", 2024-03-19T01:28:03.031 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "zabbix_sender": { 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "zabbix_sender", 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "str", 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "level": "advanced", 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "flags": 0, 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-19T01:28:03.032 INFO:teuthology.orchestra.run.smithi116.stdout: "min": "", 2024-03-19T01:28:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "max": "", 2024-03-19T01:28:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "enum_allowed": [], 2024-03-19T01:28:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "desc": "", 2024-03-19T01:28:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "long_desc": "", 2024-03-19T01:28:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "tags": [], 2024-03-19T01:28:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "see_also": [] 2024-03-19T01:28:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "services": { 2024-03-19T01:28:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: "dashboard": "https://172.21.15.116:8443/" 2024-03-19T01:28:03.033 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "always_on_modules": { 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "octopus": [ 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:03.034 INFO:teuthology.orchestra.run.smithi116.stdout: "pacific": [ 2024-03-19T01:28:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-19T01:28:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-19T01:28:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-19T01:28:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-19T01:28:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-19T01:28:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-19T01:28:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-19T01:28:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-19T01:28:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-19T01:28:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-19T01:28:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "quincy": [ 2024-03-19T01:28:03.035 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-19T01:28:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-19T01:28:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-19T01:28:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-19T01:28:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-19T01:28:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-19T01:28:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-19T01:28:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-19T01:28:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-19T01:28:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-19T01:28:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "reef": [ 2024-03-19T01:28:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-19T01:28:03.036 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: ], 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "squid": [ 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "balancer", 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "crash", 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "devicehealth", 2024-03-19T01:28:03.037 INFO:teuthology.orchestra.run.smithi116.stdout: "orchestrator", 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "pg_autoscaler", 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "progress", 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "rbd_support", 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "status", 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "telemetry", 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "volumes" 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "last_failure_osd_epoch": 3, 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "active_clients": [ 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "devicehealth", 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-19T01:28:03.038 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1958516329 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "libcephsqlite", 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 3839697157 2024-03-19T01:28:03.039 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "rbd_support", 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1321758041 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:03.040 INFO:teuthology.orchestra.run.smithi116.stdout: "name": "volumes", 2024-03-19T01:28:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "addrvec": [ 2024-03-19T01:28:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-03-19T01:28:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v2", 2024-03-19T01:28:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "172.21.15.116:0", 2024-03-19T01:28:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 4070733263 2024-03-19T01:28:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T01:28:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:03.041 INFO:teuthology.orchestra.run.smithi116.stdout: ] 2024-03-19T01:28:03.041 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:28:03.043 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:01 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:03.043 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:01 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/81282286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:28:03.043 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:01 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1648003344' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:28:03.043 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:01 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3325227269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:28:03.043 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:02 smithi162 ceph-mon[67178]: pgmap v814: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 27 MiB/s wr, 1.68k op/s 2024-03-19T01:28:03.044 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:02 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:03.044 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:02 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2639695340' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-19T01:28:03.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:01 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:03.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:01 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/81282286' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:28:03.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:01 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1648003344' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:28:03.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:01 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3325227269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:28:03.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:02 smithi156 ceph-mon[66799]: pgmap v814: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 27 MiB/s wr, 1.68k op/s 2024-03-19T01:28:03.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:02 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:03.044 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:02 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2639695340' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-19T01:28:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.681+0000 7f89ce259640 1 -- 172.21.15.116:0/2639695340 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f89b40a2800 msgr2=0x7f89b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.681+0000 7f89ce259640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f89b40a2800 0x7f89b40a4cc0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f89c8199080 tx=0x7f89b0005f90 comp rx=0 tx=0).stop 2024-03-19T01:28:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.681+0000 7f89ce259640 1 -- 172.21.15.116:0/2639695340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8101840 msgr2=0x7f89c8197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.681+0000 7f89ce259640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8101840 0x7f89c8197b70 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f89b800ebc0 tx=0x7f89b800c6a0 comp rx=0 tx=0).stop 2024-03-19T01:28:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.681+0000 7f89ce259640 1 -- 172.21.15.116:0/2639695340 shutdown_connections 2024-03-19T01:28:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.681+0000 7f89ce259640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f89b40a2800 0x7f89b40a4cc0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.681+0000 7f89ce259640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89c81033f0 0x7f89c819f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.681+0000 7f89ce259640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f89c8102a30 0x7f89c81980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.681+0000 7f89ce259640 1 --2- 172.21.15.116:0/2639695340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89c8101840 0x7f89c8197b70 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.681+0000 7f89ce259640 1 -- 172.21.15.116:0/2639695340 >> 172.21.15.116:0/2639695340 conn(0x7f89c80fcdd0 msgr2=0x7f89c8105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.681+0000 7f89ce259640 1 -- 172.21.15.116:0/2639695340 shutdown_connections 2024-03-19T01:28:03.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:01.681+0000 7f89ce259640 1 -- 172.21.15.116:0/2639695340 wait complete. 2024-03-19T01:28:03.055 DEBUG:tasks.check_counter:Getting stats from a 2024-03-19T01:28:03.055 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-19T01:28:03.055 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:02 smithi116 ceph-mon[62701]: pgmap v814: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 27 MiB/s wr, 1.68k op/s 2024-03-19T01:28:03.056 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:02 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:03.056 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:02 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2639695340' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-19T01:28:03.310 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:28:04.701 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:04 smithi116 ceph-mon[62701]: pgmap v815: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 860 op/s 2024-03-19T01:28:04.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:04 smithi156 ceph-mon[66799]: pgmap v815: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 860 op/s 2024-03-19T01:28:04.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:04 smithi162 ceph-mon[67178]: pgmap v815: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 10 MiB/s wr, 860 op/s 2024-03-19T01:28:05.087 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:28:05.087 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T01:28:05.087 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 4001, 2024-03-19T01:28:05.087 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 4722, 2024-03-19T01:28:05.087 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 1469854, 2024-03-19T01:28:05.087 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 4919159, 2024-03-19T01:28:05.087 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 11, 2024-03-19T01:28:05.087 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 3, 2024-03-19T01:28:05.087 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.314566725, 2024-03-19T01:28:05.087 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.119973047, 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.169399703, 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.040179546, 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 4720, 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.345439726, 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000073186 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 5878, 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.001344426, 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000228 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T01:28:05.088 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T01:28:05.089 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.089 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T01:28:05.089 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 5527, 2024-03-19T01:28:05.089 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 6621, 2024-03-19T01:28:05.089 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 1846599, 2024-03-19T01:28:05.089 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 3871612, 2024-03-19T01:28:05.089 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 18, 2024-03-19T01:28:05.089 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 4, 2024-03-19T01:28:05.089 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.804837101, 2024-03-19T01:28:05.089 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.572262027, 2024-03-19T01:28:05.089 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.308041157, 2024-03-19T01:28:05.089 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000630832, 2024-03-19T01:28:05.089 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:05.090 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 6619, 2024-03-19T01:28:05.090 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.510563537, 2024-03-19T01:28:05.090 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000077136 2024-03-19T01:28:05.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.090 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:05.090 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 11055, 2024-03-19T01:28:05.090 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.001950282, 2024-03-19T01:28:05.090 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000176 2024-03-19T01:28:05.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.090 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 1095056, 2024-03-19T01:28:05.090 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 2942528 2024-03-19T01:28:05.090 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.090 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T01:28:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1086, 2024-03-19T01:28:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 897, 2024-03-19T01:28:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 1051045, 2024-03-19T01:28:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 557743, 2024-03-19T01:28:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 9, 2024-03-19T01:28:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 4, 2024-03-19T01:28:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.099215663, 2024-03-19T01:28:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.032626889, 2024-03-19T01:28:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.058844418, 2024-03-19T01:28:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.015937349, 2024-03-19T01:28:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 897, 2024-03-19T01:28:05.091 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.070193965, 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000078254 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 867, 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000196736, 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000226 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 731792, 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 182304 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 1, 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.092 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-MDSRank": { 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2524, 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 1.235049964, 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000489322 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-PurgeQueue": { 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2, 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000004417, 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000002208 2024-03-19T01:28:05.093 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-19T01:28:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "request": 1777, 2024-03-19T01:28:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "reply": 1725, 2024-03-19T01:28:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "reply_latency": { 2024-03-19T01:28:05.094 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1725, 2024-03-19T01:28:05.098 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 5.578089473, 2024-03-19T01:28:05.098 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.003233675 2024-03-19T01:28:05.098 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.098 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_reply": 0, 2024-03-19T01:28:05.098 INFO:teuthology.orchestra.run.smithi116.stdout: "forward": 52, 2024-03-19T01:28:05.098 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_complete": 0, 2024-03-19T01:28:05.098 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_keys": 0, 2024-03-19T01:28:05.098 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_commit": 11, 2024-03-19T01:28:05.098 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_split": 0, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_merge": 0, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes": 455, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_top": 7, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_bottom": 398, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pin_tail": 50, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pinned": 441, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_expired": 209, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_with_caps": 74, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": 74, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "subtrees": 9, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse": 6153, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_hit": 4735, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_forward": 51, 2024-03-19T01:28:05.099 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_discover": 31, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_dir_fetch": 0, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_remote_ino": 0, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_lock": 377, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "load_cent": 33851, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "q": 0, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "exported": 14, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "exported_inodes": 203, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "imported": 15, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "imported_inodes": 213, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_dir_fetch": 0, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_backtrace_fetch": 0, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_peer_discover": 0, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rfiles": 4, 2024-03-19T01:28:05.100 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rbytes": 234, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rsnaps": 0, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_set_tag": 0, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_inotable_repaired": 0, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_inodes": 0, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_file_inodes": 0, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_inode_file_caps": 6, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_revoke": 366, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_grant": 869, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T01:28:05.101 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T01:28:05.102 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T01:28:05.102 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps": 1044, 2024-03-19T01:28:05.102 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps_dirty": 462, 2024-03-19T01:28:05.102 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_cap_release": 76, 2024-03-19T01:28:05.102 INFO:teuthology.orchestra.run.smithi116.stdout: "process_request_cap_release": 991 2024-03-19T01:28:05.102 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.102 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_cache": { 2024-03-19T01:28:05.102 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update": 199, 2024-03-19T01:28:05.102 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update_receipt": 232, 2024-03-19T01:28:05.103 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_try_discover": 129, 2024-03-19T01:28:05.103 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_send_discover": 182, 2024-03-19T01:28:05.103 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_handle_discover": 240, 2024-03-19T01:28:05.103 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays": 0, 2024-03-19T01:28:05.103 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_delayed": 0, 2024-03-19T01:28:05.103 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_enqueuing": 0, 2024-03-19T01:28:05.103 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_created": 234, 2024-03-19T01:28:05.104 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_enqueued": 0, 2024-03-19T01:28:05.104 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_reintegrated": 234, 2024-03-19T01:28:05.104 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_migrated": 0, 2024-03-19T01:28:05.104 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_processing": 0, 2024-03-19T01:28:05.104 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_enqueued": 0, 2024-03-19T01:28:05.104 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_prioritized": 0, 2024-03-19T01:28:05.104 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_started": 0, 2024-03-19T01:28:05.104 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_completed": 0, 2024-03-19T01:28:05.105 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_path": 0, 2024-03-19T01:28:05.105 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_inode": 0, 2024-03-19T01:28:05.105 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T01:28:05.105 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_exportdir": 14, 2024-03-19T01:28:05.105 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_flush": 0, 2024-03-19T01:28:05.105 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragmentdir": 0, 2024-03-19T01:28:05.106 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragstats": 0, 2024-03-19T01:28:05.106 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_inodestats": 0 2024-03-19T01:28:05.106 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.106 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_log": { 2024-03-19T01:28:05.106 INFO:teuthology.orchestra.run.smithi116.stdout: "evlrg": 0, 2024-03-19T01:28:05.106 INFO:teuthology.orchestra.run.smithi116.stdout: "evadd": 1814, 2024-03-19T01:28:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: "evex": 0, 2024-03-19T01:28:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: "evtrm": 0, 2024-03-19T01:28:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: "ev": 1814, 2024-03-19T01:28:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: "evexg": 0, 2024-03-19T01:28:05.107 INFO:teuthology.orchestra.run.smithi116.stdout: "evexd": 0, 2024-03-19T01:28:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "segadd": 3, 2024-03-19T01:28:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "segex": 0, 2024-03-19T01:28:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "segtrm": 0, 2024-03-19T01:28:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "seg": 3, 2024-03-19T01:28:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "segmjr": 2, 2024-03-19T01:28:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "segexg": 0, 2024-03-19T01:28:05.108 INFO:teuthology.orchestra.run.smithi116.stdout: "segexd": 0, 2024-03-19T01:28:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "expos": 4194304, 2024-03-19T01:28:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "wrpos": 7379692, 2024-03-19T01:28:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "rdpos": 0, 2024-03-19T01:28:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "jlat": { 2024-03-19T01:28:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1206, 2024-03-19T01:28:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 4.860175140, 2024-03-19T01:28:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.004029995 2024-03-19T01:28:05.109 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: "replayed": 0 2024-03-19T01:28:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_mem": { 2024-03-19T01:28:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 113, 2024-03-19T01:28:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: "ino+": 311, 2024-03-19T01:28:05.110 INFO:teuthology.orchestra.run.smithi116.stdout: "ino-": 198, 2024-03-19T01:28:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: "dir": 51, 2024-03-19T01:28:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: "dir+": 133, 2024-03-19T01:28:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: "dir-": 82, 2024-03-19T01:28:05.111 INFO:teuthology.orchestra.run.smithi116.stdout: "dn": 455, 2024-03-19T01:28:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: "dn+": 664, 2024-03-19T01:28:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: "dn-": 209, 2024-03-19T01:28:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: "cap": 74, 2024-03-19T01:28:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: "cap+": 289, 2024-03-19T01:28:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: "cap-": 215, 2024-03-19T01:28:05.112 INFO:teuthology.orchestra.run.smithi116.stdout: "rss": 60936, 2024-03-19T01:28:05.113 INFO:teuthology.orchestra.run.smithi116.stdout: "heap": 223612 2024-03-19T01:28:05.113 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.113 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_server": { 2024-03-19T01:28:05.113 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_client_request": 4077, 2024-03-19T01:28:05.113 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_server_request": 1, 2024-03-19T01:28:05.113 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_request": 1777, 2024-03-19T01:28:05.113 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_session": 483, 2024-03-19T01:28:05.113 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_peer_request": 845, 2024-03-19T01:28:05.113 INFO:teuthology.orchestra.run.smithi116.stdout: "req_create_latency": { 2024-03-19T01:28:05.114 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 59, 2024-03-19T01:28:05.114 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.096363425, 2024-03-19T01:28:05.114 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001633278 2024-03-19T01:28:05.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.114 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getattr_latency": { 2024-03-19T01:28:05.114 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 54, 2024-03-19T01:28:05.114 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.030262118, 2024-03-19T01:28:05.114 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000560409 2024-03-19T01:28:05.114 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.114 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getfilelock_latency": { 2024-03-19T01:28:05.114 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.114 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.114 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: "req_link_latency": { 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 234, 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.801035911, 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.003423230 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookup_latency": { 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 828, 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 1.286748020, 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.001554043 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookuphash_latency": { 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.115 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupino_latency": { 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupname_latency": { 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupparent_latency": { 2024-03-19T01:28:05.116 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.117 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.117 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.117 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupsnap_latency": { 2024-03-19T01:28:05.117 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.117 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.117 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.117 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.117 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lssnap_latency": { 2024-03-19T01:28:05.117 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.118 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.118 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.119 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mkdir_latency": { 2024-03-19T01:28:05.119 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.119 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.119 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.119 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mknod_latency": { 2024-03-19T01:28:05.119 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.119 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.119 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.119 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.119 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mksnap_latency": { 2024-03-19T01:28:05.119 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.120 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.120 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.120 INFO:teuthology.orchestra.run.smithi116.stdout: "req_open_latency": { 2024-03-19T01:28:05.120 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 76, 2024-03-19T01:28:05.120 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.036891179, 2024-03-19T01:28:05.120 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000485410 2024-03-19T01:28:05.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.120 INFO:teuthology.orchestra.run.smithi116.stdout: "req_readdir_latency": { 2024-03-19T01:28:05.120 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 6, 2024-03-19T01:28:05.120 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.013241520, 2024-03-19T01:28:05.120 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.002206920 2024-03-19T01:28:05.120 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rename_latency": { 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 234, 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 2.587361432, 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.011057100 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: "req_renamesnap_latency": { 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: "req_snapdiff_latency": { 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.121 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.122 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmdir_latency": { 2024-03-19T01:28:05.122 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.122 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.122 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.122 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmsnap_latency": { 2024-03-19T01:28:05.122 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.122 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.122 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.122 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.123 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmxattr_latency": { 2024-03-19T01:28:05.123 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.123 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.123 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.123 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setattr_latency": { 2024-03-19T01:28:05.123 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.123 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.123 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.123 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.123 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setdirlayout_latency": { 2024-03-19T01:28:05.124 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.124 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.124 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.124 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.124 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setfilelock_latency": { 2024-03-19T01:28:05.124 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.124 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.124 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.125 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setlayout_latency": { 2024-03-19T01:28:05.125 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.125 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.125 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.125 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.126 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setxattr_latency": { 2024-03-19T01:28:05.126 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.126 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.126 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.126 INFO:teuthology.orchestra.run.smithi116.stdout: "req_symlink_latency": { 2024-03-19T01:28:05.126 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.126 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.126 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.126 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.126 INFO:teuthology.orchestra.run.smithi116.stdout: "req_unlink_latency": { 2024-03-19T01:28:05.126 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 234, 2024-03-19T01:28:05.126 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.726185868, 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.003103358 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_revoke_eviction": 0, 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_acquisition_throttle": 0, 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getvxattr_latency": { 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_sessions": { 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: "session_count": 1, 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: "session_add": 1, 2024-03-19T01:28:05.127 INFO:teuthology.orchestra.run.smithi116.stdout: "session_remove": 0, 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_open": 1, 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_stale": 0, 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: "total_load": 330, 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: "average_load": 330, 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: "avg_session_uptime": 922, 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: "mdthresh_evicted": 0 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T01:28:05.128 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-19T01:28:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T01:28:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T01:28:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T01:28:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T01:28:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T01:28:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-19T01:28:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T01:28:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T01:28:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-19T01:28:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-19T01:28:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-19T01:28:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-19T01:28:05.129 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T01:28:05.130 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T01:28:05.130 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T01:28:05.130 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T01:28:05.130 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T01:28:05.130 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-19T01:28:05.130 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-19T01:28:05.130 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-19T01:28:05.130 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T01:28:05.130 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T01:28:05.130 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-19T01:28:05.130 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-19T01:28:05.130 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-19T01:28:05.130 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 304576, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 197, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-19T01:28:05.131 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 7720, 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 162, 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 725596, 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 21051, 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.132 INFO:teuthology.orchestra.run.smithi116.stdout: "objecter": { 2024-03-19T01:28:05.133 INFO:teuthology.orchestra.run.smithi116.stdout: "op_active": 0, 2024-03-19T01:28:05.133 INFO:teuthology.orchestra.run.smithi116.stdout: "op_laggy": 0, 2024-03-19T01:28:05.133 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send": 1480, 2024-03-19T01:28:05.133 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send_bytes": 3241637, 2024-03-19T01:28:05.133 INFO:teuthology.orchestra.run.smithi116.stdout: "op_resend": 28, 2024-03-19T01:28:05.133 INFO:teuthology.orchestra.run.smithi116.stdout: "op_reply": 1480, 2024-03-19T01:28:05.133 INFO:teuthology.orchestra.run.smithi116.stdout: "op_latency": { 2024-03-19T01:28:05.133 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1480, 2024-03-19T01:28:05.133 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 8.149871433, 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.005506669 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: "op_inflight": 0, 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: "oplen_avg": { 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1480, 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 1704 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: "op": 1480, 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: "op_r": 0, 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: "op_w": 1480, 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: "op_rmw": 0, 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: "op_pg": 0, 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_stat": 0, 2024-03-19T01:28:05.134 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_create": 10, 2024-03-19T01:28:05.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_read": 0, 2024-03-19T01:28:05.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_write": 1206, 2024-03-19T01:28:05.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writefull": 58, 2024-03-19T01:28:05.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writesame": 0, 2024-03-19T01:28:05.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_append": 0, 2024-03-19T01:28:05.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_zero": 0, 2024-03-19T01:28:05.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_truncate": 0, 2024-03-19T01:28:05.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_delete": 6, 2024-03-19T01:28:05.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_mapext": 0, 2024-03-19T01:28:05.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_sparse_read": 0, 2024-03-19T01:28:05.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_clonerange": 0, 2024-03-19T01:28:05.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_getxattr": 0, 2024-03-19T01:28:05.135 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_setxattr": 20, 2024-03-19T01:28:05.136 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_cmpxattr": 0, 2024-03-19T01:28:05.136 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_rmxattr": 0, 2024-03-19T01:28:05.136 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_resetxattrs": 0, 2024-03-19T01:28:05.136 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_call": 0, 2024-03-19T01:28:05.136 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_watch": 0, 2024-03-19T01:28:05.136 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_notify": 0, 2024-03-19T01:28:05.136 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T01:28:05.136 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls": 0, 2024-03-19T01:28:05.136 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls_filter": 0, 2024-03-19T01:28:05.137 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_other": 0, 2024-03-19T01:28:05.137 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_active": 0, 2024-03-19T01:28:05.137 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_send": 0, 2024-03-19T01:28:05.137 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_resend": 0, 2024-03-19T01:28:05.137 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_ping": 0, 2024-03-19T01:28:05.137 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_active": 0, 2024-03-19T01:28:05.137 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_send": 0, 2024-03-19T01:28:05.137 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_resend": 0, 2024-03-19T01:28:05.137 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_active": 0, 2024-03-19T01:28:05.137 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_send": 0, 2024-03-19T01:28:05.137 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_resend": 0, 2024-03-19T01:28:05.137 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_active": 0, 2024-03-19T01:28:05.137 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_send": 0, 2024-03-19T01:28:05.138 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_resend": 0, 2024-03-19T01:28:05.138 INFO:teuthology.orchestra.run.smithi116.stdout: "command_active": 0, 2024-03-19T01:28:05.138 INFO:teuthology.orchestra.run.smithi116.stdout: "command_send": 0, 2024-03-19T01:28:05.138 INFO:teuthology.orchestra.run.smithi116.stdout: "command_resend": 0, 2024-03-19T01:28:05.138 INFO:teuthology.orchestra.run.smithi116.stdout: "map_epoch": 87, 2024-03-19T01:28:05.138 INFO:teuthology.orchestra.run.smithi116.stdout: "map_full": 0, 2024-03-19T01:28:05.139 INFO:teuthology.orchestra.run.smithi116.stdout: "map_inc": 2, 2024-03-19T01:28:05.139 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_sessions": 9, 2024-03-19T01:28:05.139 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_open": 15, 2024-03-19T01:28:05.139 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_close": 6, 2024-03-19T01:28:05.139 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_laggy": 0, 2024-03-19T01:28:05.139 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_wr": 326, 2024-03-19T01:28:05.140 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_rd": 0, 2024-03-19T01:28:05.140 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_del": 78 2024-03-19T01:28:05.140 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.140 INFO:teuthology.orchestra.run.smithi116.stdout: "oft": { 2024-03-19T01:28:05.140 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_objs": 1, 2024-03-19T01:28:05.140 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_kv_pairs": 0, 2024-03-19T01:28:05.141 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_updates": 0, 2024-03-19T01:28:05.141 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_removes": 0 2024-03-19T01:28:05.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.141 INFO:teuthology.orchestra.run.smithi116.stdout: "osd-slow-ops": { 2024-03-19T01:28:05.141 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_ops_count": 0 2024-03-19T01:28:05.141 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.142 INFO:teuthology.orchestra.run.smithi116.stdout: "purge_queue": { 2024-03-19T01:28:05.142 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops": 0, 2024-03-19T01:28:05.142 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T01:28:05.142 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing": 0, 2024-03-19T01:28:05.142 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_high_water": 0, 2024-03-19T01:28:05.142 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed_ops": 0, 2024-03-19T01:28:05.143 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed": 0, 2024-03-19T01:28:05.143 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_item_in_journal": 0 2024-03-19T01:28:05.143 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.143 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T01:28:05.143 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T01:28:05.143 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-19T01:28:05.143 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T01:28:05.143 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 10610, 2024-03-19T01:28:05.143 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 3240666, 2024-03-19T01:28:05.143 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:05.143 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 10610, 2024-03-19T01:28:05.143 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T01:28:05.143 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 10610, 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 3240666, 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_bytes": { 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T01:28:05.144 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T01:28:05.145 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:05.145 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:05.145 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 1480, 2024-03-19T01:28:05.145 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 3241637, 2024-03-19T01:28:05.145 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 1480, 2024-03-19T01:28:05.145 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 3241637, 2024-03-19T01:28:05.145 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T01:28:05.145 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.145 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.145 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.145 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:05.145 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.146 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_ops": { 2024-03-19T01:28:05.146 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T01:28:05.146 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 1024, 2024-03-19T01:28:05.146 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T01:28:05.146 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T01:28:05.146 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T01:28:05.146 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:05.146 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:05.146 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 1480, 2024-03-19T01:28:05.146 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 1480, 2024-03-19T01:28:05.147 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 1480, 2024-03-19T01:28:05.147 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 1480, 2024-03-19T01:28:05.147 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T01:28:05.147 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.147 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.147 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.147 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:05.147 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.147 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle": { 2024-03-19T01:28:05.147 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T01:28:05.147 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-19T01:28:05.147 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T01:28:05.147 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T01:28:05.148 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T01:28:05.148 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:05.148 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:05.148 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T01:28:05.148 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T01:28:05.148 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-19T01:28:05.148 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-19T01:28:05.148 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T01:28:05.148 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.148 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.148 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.148 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:05.148 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle-0x558e9f0f2500": { 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 1814, 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 3185388, 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 1814, 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 1206, 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 3185388, 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T01:28:05.149 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:05.150 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:05.150 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:05.150 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:05.150 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:05.150 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:28:05.935 INFO:tasks.check_counter:Daemon mds.a mds.exported=14 2024-03-19T01:28:05.935 INFO:tasks.check_counter:Daemon mds.a mds.imported=15 2024-03-19T01:28:05.935 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=199 2024-03-19T01:28:05.935 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=232 2024-03-19T01:28:05.935 DEBUG:tasks.check_counter:Getting stats from d 2024-03-19T01:28:05.936 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-19T01:28:06.197 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:28:06.578 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:06 smithi116 ceph-mon[62701]: pgmap v816: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 5.9 MiB/s wr, 641 op/s 2024-03-19T01:28:06.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:06 smithi156 ceph-mon[66799]: pgmap v816: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 5.9 MiB/s wr, 641 op/s 2024-03-19T01:28:06.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:06 smithi162 ceph-mon[67178]: pgmap v816: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 5.9 MiB/s wr, 641 op/s 2024-03-19T01:28:07.969 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:28:07.969 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T01:28:07.969 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 0, 2024-03-19T01:28:07.969 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2, 2024-03-19T01:28:07.969 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 0, 2024-03-19T01:28:07.969 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 0, 2024-03-19T01:28:07.970 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-19T01:28:07.970 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 0, 2024-03-19T01:28:07.970 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.001130032, 2024-03-19T01:28:07.970 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.000000000, 2024-03-19T01:28:07.970 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.000269932, 2024-03-19T01:28:07.970 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T01:28:07.970 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:07.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:07.970 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:07.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:07.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:07.970 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:07.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 297, 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 292, 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 501200, 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 178928, 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 2, 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 1, 2024-03-19T01:28:07.971 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.039217152, 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.015680862, 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.016432227, 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.006371968, 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 292, 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.022814517, 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000078131 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:07.972 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 501200, 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 178928 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 2, 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 221, 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 320, 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 194736, 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 2, 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.015166404, 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.011453247, 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.001512950, 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T01:28:07.973 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 219, 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.019241665, 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000087861 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 194736 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-19T01:28:07.974 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 0, 2024-03-19T01:28:07.975 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-19T01:28:07.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:07.975 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-19T01:28:07.975 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-19T01:28:07.975 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-19T01:28:07.975 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T01:28:07.975 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-19T01:28:07.975 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T01:28:07.975 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-19T01:28:07.975 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T01:28:07.975 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T01:28:07.975 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T01:28:07.975 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T01:28:07.976 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T01:28:07.976 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-19T01:28:07.976 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T01:28:07.976 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T01:28:07.976 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-19T01:28:07.976 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-19T01:28:07.976 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-19T01:28:07.976 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-19T01:28:07.976 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T01:28:07.976 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T01:28:07.976 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T01:28:07.976 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T01:28:07.976 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T01:28:07.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-19T01:28:07.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-19T01:28:07.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-19T01:28:07.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T01:28:07.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T01:28:07.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-19T01:28:07.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-19T01:28:07.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-19T01:28:07.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-19T01:28:07.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T01:28:07.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-19T01:28:07.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T01:28:07.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 147960, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 28, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 0, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 0, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T01:28:07.978 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-19T01:28:07.979 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-19T01:28:07.979 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-19T01:28:07.979 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 928, 2024-03-19T01:28:07.979 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 2, 2024-03-19T01:28:07.979 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-19T01:28:07.979 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-19T01:28:07.979 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-19T01:28:07.979 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-19T01:28:07.979 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:07.979 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T01:28:07.979 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T01:28:07.979 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 295, 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 456793, 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 295, 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 295, 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 456793, 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:07.981 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:28:08.610 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-19T01:28:08.610 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-19T01:28:08.611 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-19T01:28:08.611 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-19T01:28:08.611 DEBUG:tasks.check_counter:Getting stats from g 2024-03-19T01:28:08.611 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-19T01:28:08.636 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:08 smithi116 ceph-mon[62701]: pgmap v817: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 3.3 MiB/s wr, 367 op/s 2024-03-19T01:28:08.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:08 smithi156 ceph-mon[66799]: pgmap v817: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 3.3 MiB/s wr, 367 op/s 2024-03-19T01:28:08.862 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:28:08.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:08 smithi162 ceph-mon[67178]: pgmap v817: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 3.3 MiB/s wr, 367 op/s 2024-03-19T01:28:10.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:10 smithi156 ceph-mon[66799]: pgmap v818: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-03-19T01:28:10.822 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:10 smithi116 ceph-mon[62701]: pgmap v818: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-03-19T01:28:10.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:10 smithi162 ceph-mon[67178]: pgmap v818: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 2.1 MiB/s wr, 235 op/s 2024-03-19T01:28:10.949 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:28:10.949 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T01:28:10.949 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 1045, 2024-03-19T01:28:10.949 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 1045, 2024-03-19T01:28:10.949 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 352745, 2024-03-19T01:28:10.949 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 331487, 2024-03-19T01:28:10.949 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 7, 2024-03-19T01:28:10.949 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 2, 2024-03-19T01:28:10.949 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.121444773, 2024-03-19T01:28:10.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.041564878, 2024-03-19T01:28:10.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.027495375, 2024-03-19T01:28:10.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.038717619, 2024-03-19T01:28:10.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:10.950 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 1044, 2024-03-19T01:28:10.950 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.087466016, 2024-03-19T01:28:10.950 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000083779 2024-03-19T01:28:10.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.950 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:10.950 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.950 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.950 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.950 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-19T01:28:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T01:28:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T01:28:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 2469, 2024-03-19T01:28:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2438, 2024-03-19T01:28:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 2164793, 2024-03-19T01:28:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 859575, 2024-03-19T01:28:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-19T01:28:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 3, 2024-03-19T01:28:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.208170256, 2024-03-19T01:28:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.055144492, 2024-03-19T01:28:10.951 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.055529933, 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.088532505, 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 2437, 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.121121365, 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000049701 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 702080, 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 177408 2024-03-19T01:28:10.952 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 5, 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 221, 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 1668, 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 553093, 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 7, 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 4, 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.015952582, 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.011833434, 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.002479099, 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000050932, 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 219, 2024-03-19T01:28:10.953 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.011629154, 2024-03-19T01:28:10.954 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000053101 2024-03-19T01:28:10.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.954 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:10.954 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.954 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.954 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.954 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-19T01:28:10.954 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 552128 2024-03-19T01:28:10.954 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.954 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-19T01:28:10.954 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 1, 2024-03-19T01:28:10.954 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-19T01:28:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-MDSRank": { 2024-03-19T01:28:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-19T01:28:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-19T01:28:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 5371, 2024-03-19T01:28:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.140771977, 2024-03-19T01:28:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000026209 2024-03-19T01:28:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "finisher-PurgeQueue": { 2024-03-19T01:28:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "queue_len": 0, 2024-03-19T01:28:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "complete_latency": { 2024-03-19T01:28:10.955 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "mds": { 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "request": 0, 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "reply": 0, 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "reply_latency": { 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_reply": 0, 2024-03-19T01:28:10.956 INFO:teuthology.orchestra.run.smithi116.stdout: "forward": 0, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_complete": 0, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_fetch_keys": 0, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_commit": 0, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_split": 0, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_merge": 0, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes": 275, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_top": 5, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_bottom": 270, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pin_tail": 0, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_pinned": 269, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_expired": 0, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "inodes_with_caps": 0, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "caps": 0, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "subtrees": 11, 2024-03-19T01:28:10.957 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse": 0, 2024-03-19T01:28:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_hit": 0, 2024-03-19T01:28:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_forward": 0, 2024-03-19T01:28:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_discover": 0, 2024-03-19T01:28:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_dir_fetch": 0, 2024-03-19T01:28:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_remote_ino": 0, 2024-03-19T01:28:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "traverse_lock": 0, 2024-03-19T01:28:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "load_cent": 0, 2024-03-19T01:28:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "q": 0, 2024-03-19T01:28:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "exported": 0, 2024-03-19T01:28:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "exported_inodes": 0, 2024-03-19T01:28:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "imported": 0, 2024-03-19T01:28:10.958 INFO:teuthology.orchestra.run.smithi116.stdout: "imported_inodes": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_dir_fetch": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_backtrace_fetch": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "openino_peer_discover": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rfiles": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rbytes": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "root_rsnaps": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_set_tag": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_inotable_repaired": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_inodes": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "scrub_file_inodes": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_inode_file_caps": 0, 2024-03-19T01:28:10.959 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_grant": 0, 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps": 0, 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_caps_dirty": 0, 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_cap_release": 0, 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "process_request_cap_release": 0 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_cache": { 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update": 0, 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_update_receipt": 0, 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_try_discover": 0, 2024-03-19T01:28:10.960 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_send_discover": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "dir_handle_discover": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_delayed": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "num_strays_enqueuing": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_created": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_enqueued": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_reintegrated": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "strays_migrated": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_processing": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_enqueued": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "num_recovering_prioritized": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_started": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "recovery_completed": 0, 2024-03-19T01:28:10.961 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_path": 0, 2024-03-19T01:28:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_quiesce_inode": 0, 2024-03-19T01:28:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T01:28:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_exportdir": 0, 2024-03-19T01:28:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_flush": 0, 2024-03-19T01:28:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragmentdir": 0, 2024-03-19T01:28:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_fragstats": 0, 2024-03-19T01:28:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "ireq_inodestats": 0 2024-03-19T01:28:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_log": { 2024-03-19T01:28:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "evlrg": 0, 2024-03-19T01:28:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "evadd": 0, 2024-03-19T01:28:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "evex": 0, 2024-03-19T01:28:10.962 INFO:teuthology.orchestra.run.smithi116.stdout: "evtrm": 0, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "ev": 630, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "evexg": 0, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "evexd": 0, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "segadd": 0, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "segex": 0, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "segtrm": 0, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "seg": 2, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "segmjr": 2, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "segexg": 0, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "segexd": 0, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "expos": 4194304, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "wrpos": 5112643, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "rdpos": 5111518, 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "jlat": { 2024-03-19T01:28:10.963 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "replayed": 630 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_mem": { 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "ino": 269, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "ino+": 271, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "ino-": 2, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "dir": 17, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "dir+": 17, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "dir-": 0, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "dn": 275, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "dn+": 275, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "dn-": 0, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "cap": 0, 2024-03-19T01:28:10.964 INFO:teuthology.orchestra.run.smithi116.stdout: "cap+": 0, 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "cap-": 0, 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "rss": 52440, 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "heap": 215420 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_server": { 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_client_request": 0, 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "dispatch_server_request": 0, 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_request": 0, 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_client_session": 0, 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "handle_peer_request": 0, 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "req_create_latency": { 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.965 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getattr_latency": { 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getfilelock_latency": { 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "req_link_latency": { 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.966 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookup_latency": { 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookuphash_latency": { 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupino_latency": { 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.967 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupname_latency": { 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupparent_latency": { 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lookupsnap_latency": { 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.968 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.969 INFO:teuthology.orchestra.run.smithi116.stdout: "req_lssnap_latency": { 2024-03-19T01:28:10.969 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.969 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.969 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.969 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.969 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mkdir_latency": { 2024-03-19T01:28:10.969 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.969 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mknod_latency": { 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: "req_mksnap_latency": { 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: "req_open_latency": { 2024-03-19T01:28:10.970 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: "req_readdir_latency": { 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rename_latency": { 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.971 INFO:teuthology.orchestra.run.smithi116.stdout: "req_renamesnap_latency": { 2024-03-19T01:28:10.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.972 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.972 INFO:teuthology.orchestra.run.smithi116.stdout: "req_snapdiff_latency": { 2024-03-19T01:28:10.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.972 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.972 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.972 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmdir_latency": { 2024-03-19T01:28:10.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.972 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.972 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmsnap_latency": { 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: "req_rmxattr_latency": { 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setattr_latency": { 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.973 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setdirlayout_latency": { 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setfilelock_latency": { 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setlayout_latency": { 2024-03-19T01:28:10.974 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: "req_setxattr_latency": { 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: "req_symlink_latency": { 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.975 INFO:teuthology.orchestra.run.smithi116.stdout: "req_unlink_latency": { 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_revoke_eviction": 0, 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "cap_acquisition_throttle": 0, 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "req_getvxattr_latency": { 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_sessions": { 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "session_count": 1, 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "session_add": 1, 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "session_remove": 0, 2024-03-19T01:28:10.976 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_open": 1, 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "sessions_stale": 0, 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "total_load": 0, 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "average_load": 0, 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "avg_session_uptime": 3712, 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "mdthresh_evicted": 0 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T01:28:10.977 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T01:28:10.978 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T01:28:10.978 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T01:28:10.978 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T01:28:10.978 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-19T01:28:10.978 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T01:28:10.978 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T01:28:10.978 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-19T01:28:10.978 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-19T01:28:10.978 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-19T01:28:10.978 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-19T01:28:10.978 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T01:28:10.978 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-19T01:28:10.979 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 207104, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 314, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 7720, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 162, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-19T01:28:10.980 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 824602, 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 7011, 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: "objecter": { 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: "op_active": 0, 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: "op_laggy": 0, 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send": 3189, 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: "op_send_bytes": 21, 2024-03-19T01:28:10.981 INFO:teuthology.orchestra.run.smithi116.stdout: "op_resend": 3, 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: "op_reply": 3189, 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: "op_latency": { 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3189, 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 13.234791267, 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.004150138 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: "op_inflight": 0, 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: "oplen_avg": { 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 3189, 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 3190 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: "op": 3189, 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: "op_r": 3189, 2024-03-19T01:28:10.982 INFO:teuthology.orchestra.run.smithi116.stdout: "op_w": 0, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "op_rmw": 0, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "op_pg": 0, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_stat": 1965, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_create": 0, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_read": 1222, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_write": 0, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writefull": 0, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_writesame": 0, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_append": 0, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_zero": 0, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_truncate": 0, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_delete": 0, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_mapext": 0, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_sparse_read": 0, 2024-03-19T01:28:10.983 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_clonerange": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_getxattr": 1, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_setxattr": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_cmpxattr": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_rmxattr": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_resetxattrs": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_call": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_watch": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_notify": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_pgls_filter": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "osdop_other": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_active": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_send": 0, 2024-03-19T01:28:10.984 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_resend": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "linger_ping": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_active": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_send": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "poolop_resend": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_active": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_send": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "poolstat_resend": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_active": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_send": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "statfs_resend": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "command_active": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "command_send": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "command_resend": 0, 2024-03-19T01:28:10.985 INFO:teuthology.orchestra.run.smithi116.stdout: "map_epoch": 87, 2024-03-19T01:28:10.986 INFO:teuthology.orchestra.run.smithi116.stdout: "map_full": 0, 2024-03-19T01:28:10.986 INFO:teuthology.orchestra.run.smithi116.stdout: "map_inc": 2, 2024-03-19T01:28:10.986 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_sessions": 6, 2024-03-19T01:28:10.986 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_open": 8, 2024-03-19T01:28:10.986 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_session_close": 2, 2024-03-19T01:28:10.986 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_laggy": 0, 2024-03-19T01:28:10.986 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_wr": 0, 2024-03-19T01:28:10.986 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_rd": 2, 2024-03-19T01:28:10.986 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_del": 0 2024-03-19T01:28:10.986 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.986 INFO:teuthology.orchestra.run.smithi116.stdout: "oft": { 2024-03-19T01:28:10.986 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_objs": 0, 2024-03-19T01:28:10.986 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_kv_pairs": 0, 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_updates": 0, 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: "omap_total_removes": 0 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: "osd-slow-ops": { 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: "slow_ops_count": 0 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: "purge_queue": { 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops": 0, 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing": 0, 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executing_high_water": 0, 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed_ops": 0, 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_executed": 0, 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: "pq_item_in_journal": 0 2024-03-19T01:28:10.987 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 3515, 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 2183242, 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 3515, 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 3515, 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 2183242, 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T01:28:10.988 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_bytes": { 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 3189, 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 918344, 2024-03-19T01:28:10.989 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 3189, 2024-03-19T01:28:10.990 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 918344, 2024-03-19T01:28:10.990 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T01:28:10.990 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.990 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.990 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.990 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:10.990 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.990 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-objecter_ops": { 2024-03-19T01:28:10.990 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T01:28:10.990 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 1024, 2024-03-19T01:28:10.990 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T01:28:10.990 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 3189, 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 3189, 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 3189, 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 3189, 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle": { 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T01:28:10.991 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.992 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-write_buf_throttle-0x55925675c6e0": { 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 3758096384, 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 0, 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 0, 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 0, 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 0, 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T01:28:10.993 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:10.994 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:10.994 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:10.994 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:10.994 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:10.994 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:28:11.738 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-19T01:28:11.738 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-19T01:28:11.738 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update=0 2024-03-19T01:28:11.739 INFO:tasks.check_counter:Daemon mds.g mds_cache.dir_update_receipt=0 2024-03-19T01:28:11.739 DEBUG:tasks.check_counter:Getting stats from j 2024-03-19T01:28:11.739 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-19T01:28:12.033 INFO:teuthology.orchestra.run.smithi116.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.a/config 2024-03-19T01:28:12.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:12 smithi156 ceph-mon[66799]: pgmap v819: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 969 KiB/s wr, 122 op/s 2024-03-19T01:28:12.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:12 smithi162 ceph-mon[67178]: pgmap v819: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 969 KiB/s wr, 122 op/s 2024-03-19T01:28:12.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:12 smithi116 ceph-mon[62701]: pgmap v819: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 969 KiB/s wr, 122 op/s 2024-03-19T01:28:13.737 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-03-19T01:28:13.737 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T01:28:13.737 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 296, 2024-03-19T01:28:13.737 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 293, 2024-03-19T01:28:13.737 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 498992, 2024-03-19T01:28:13.737 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 180384, 2024-03-19T01:28:13.737 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-19T01:28:13.737 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 0, 2024-03-19T01:28:13.738 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.039879094, 2024-03-19T01:28:13.738 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.015435786, 2024-03-19T01:28:13.738 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.016765643, 2024-03-19T01:28:13.738 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.006383187, 2024-03-19T01:28:13.738 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:13.738 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 292, 2024-03-19T01:28:13.738 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.019457328, 2024-03-19T01:28:13.738 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000066634 2024-03-19T01:28:13.738 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:13.738 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:13.738 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:13.738 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 498992, 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 180384 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 6, 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 225, 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 3344, 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 195536, 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 4, 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 2, 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.016483731, 2024-03-19T01:28:13.739 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.012563680, 2024-03-19T01:28:13.740 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.001779647, 2024-03-19T01:28:13.740 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T01:28:13.740 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:13.740 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 224, 2024-03-19T01:28:13.740 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.021912564, 2024-03-19T01:28:13.740 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000097823 2024-03-19T01:28:13.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:13.740 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:13.740 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:13.740 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:13.740 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:13.740 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:13.740 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-19T01:28:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 195536 2024-03-19T01:28:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T01:28:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_messages": 0, 2024-03-19T01:28:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages": 2, 2024-03-19T01:28:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_bytes": 0, 2024-03-19T01:28:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_bytes": 0, 2024-03-19T01:28:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_created_connections": 2, 2024-03-19T01:28:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_active_connections": 1, 2024-03-19T01:28:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_total_time": 0.000496891, 2024-03-19T01:28:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_send_time": 0.000000000, 2024-03-19T01:28:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_recv_time": 0.000424075, 2024-03-19T01:28:13.741 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-19T01:28:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T01:28:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T01:28:13.742 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "cct": { 2024-03-19T01:28:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "total_workers": 0, 2024-03-19T01:28:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "unhealthy_workers": 0 2024-03-19T01:28:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "mempool": { 2024-03-19T01:28:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_bytes": 0, 2024-03-19T01:28:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "bloom_filter_items": 0, 2024-03-19T01:28:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T01:28:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_alloc_items": 0, 2024-03-19T01:28:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T01:28:13.743 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_data_items": 0, 2024-03-19T01:28:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T01:28:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T01:28:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T01:28:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T01:28:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T01:28:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_other_items": 0, 2024-03-19T01:28:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T01:28:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T01:28:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_bytes": 0, 2024-03-19T01:28:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_extent_items": 0, 2024-03-19T01:28:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_bytes": 0, 2024-03-19T01:28:13.744 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_blob_items": 0, 2024-03-19T01:28:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T01:28:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T01:28:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T01:28:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T01:28:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T01:28:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_fsck_items": 0, 2024-03-19T01:28:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_bytes": 0, 2024-03-19T01:28:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_txc_items": 0, 2024-03-19T01:28:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T01:28:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T01:28:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_bytes": 0, 2024-03-19T01:28:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "bluestore_writing_items": 0, 2024-03-19T01:28:13.745 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_bytes": 0, 2024-03-19T01:28:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_items": 0, 2024-03-19T01:28:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T01:28:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_reader_items": 0, 2024-03-19T01:28:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T01:28:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "bluefs_file_writer_items": 0, 2024-03-19T01:28:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_bytes": 148784, 2024-03-19T01:28:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_anon_items": 31, 2024-03-19T01:28:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_bytes": 0, 2024-03-19T01:28:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "buffer_meta_items": 0, 2024-03-19T01:28:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_bytes": 0, 2024-03-19T01:28:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_items": 0, 2024-03-19T01:28:13.746 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_bytes": 0, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_mapbl_items": 0, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_bytes": 0, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_pglog_items": 0, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_bytes": 0, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_items": 0, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_mapping_items": 0, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_bytes": 0, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "pgmap_items": 0, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_bytes": 928, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_co_items": 2, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_bytes": 0, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_1_items": 0, 2024-03-19T01:28:13.747 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_bytes": 0, 2024-03-19T01:28:13.748 INFO:teuthology.orchestra.run.smithi116.stdout: "unittest_2_items": 0 2024-03-19T01:28:13.748 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-03-19T01:28:13.748 INFO:teuthology.orchestra.run.smithi116.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T01:28:13.748 INFO:teuthology.orchestra.run.smithi116.stdout: "val": 0, 2024-03-19T01:28:13.748 INFO:teuthology.orchestra.run.smithi116.stdout: "max": 104857600, 2024-03-19T01:28:13.748 INFO:teuthology.orchestra.run.smithi116.stdout: "get_started": 0, 2024-03-19T01:28:13.748 INFO:teuthology.orchestra.run.smithi116.stdout: "get": 298, 2024-03-19T01:28:13.748 INFO:teuthology.orchestra.run.smithi116.stdout: "get_sum": 457183, 2024-03-19T01:28:13.748 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:13.748 INFO:teuthology.orchestra.run.smithi116.stdout: "get_or_fail_success": 298, 2024-03-19T01:28:13.748 INFO:teuthology.orchestra.run.smithi116.stdout: "take": 0, 2024-03-19T01:28:13.748 INFO:teuthology.orchestra.run.smithi116.stdout: "take_sum": 0, 2024-03-19T01:28:13.748 INFO:teuthology.orchestra.run.smithi116.stdout: "put": 298, 2024-03-19T01:28:13.749 INFO:teuthology.orchestra.run.smithi116.stdout: "put_sum": 457183, 2024-03-19T01:28:13.749 INFO:teuthology.orchestra.run.smithi116.stdout: "wait": { 2024-03-19T01:28:13.749 INFO:teuthology.orchestra.run.smithi116.stdout: "avgcount": 0, 2024-03-19T01:28:13.749 INFO:teuthology.orchestra.run.smithi116.stdout: "sum": 0.000000000, 2024-03-19T01:28:13.749 INFO:teuthology.orchestra.run.smithi116.stdout: "avgtime": 0.000000000 2024-03-19T01:28:13.749 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:13.749 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-03-19T01:28:13.749 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-03-19T01:28:14.363 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-19T01:28:14.363 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-19T01:28:14.363 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-19T01:28:14.363 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-19T01:28:14.363 DEBUG:tasks.check_counter:Getting stats from b 2024-03-19T01:28:14.363 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-19T01:28:14.619 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T01:28:14.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:14 smithi156 ceph-mon[66799]: pgmap v820: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 474 KiB/s wr, 65 op/s 2024-03-19T01:28:14.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:14 smithi162 ceph-mon[67178]: pgmap v820: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 474 KiB/s wr, 65 op/s 2024-03-19T01:28:14.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:14 smithi116 ceph-mon[62701]: pgmap v820: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 474 KiB/s wr, 65 op/s 2024-03-19T01:28:16.513 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:16 smithi156 ceph-mon[66799]: pgmap v821: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-03-19T01:28:16.514 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-19T01:28:16.514 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T01:28:16.514 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 6833, 2024-03-19T01:28:16.514 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 5714, 2024-03-19T01:28:16.514 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 2351126, 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 9405778, 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 10, 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.418942184, 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.145952041, 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.216689538, 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.077344980, 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 5719, 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1.587481398, 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000277580 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 9136, 2024-03-19T01:28:16.515 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.001536698, 2024-03-19T01:28:16.516 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000168 2024-03-19T01:28:16.516 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.516 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T01:28:16.516 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T01:28:16.516 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.516 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T01:28:16.516 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 5253, 2024-03-19T01:28:16.516 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 7186, 2024-03-19T01:28:16.516 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 2515106, 2024-03-19T01:28:16.516 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 4382956, 2024-03-19T01:28:16.516 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 15, 2024-03-19T01:28:16.516 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 7, 2024-03-19T01:28:16.516 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.421672287, 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.149326169, 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.307881435, 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.075777203, 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 7182, 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1.096904382, 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000152729 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 9770, 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.001205685, 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000123 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 2249104, 2024-03-19T01:28:16.517 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 3451888 2024-03-19T01:28:16.518 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.518 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T01:28:16.518 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 3735, 2024-03-19T01:28:16.518 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 4097, 2024-03-19T01:28:16.518 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 1454112, 2024-03-19T01:28:16.518 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 4678739, 2024-03-19T01:28:16.518 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 150, 2024-03-19T01:28:16.518 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 8, 2024-03-19T01:28:16.518 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.339657311, 2024-03-19T01:28:16.518 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.122263324, 2024-03-19T01:28:16.518 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.166687965, 2024-03-19T01:28:16.518 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.052262562, 2024-03-19T01:28:16.518 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 4092, 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.462354131, 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000112989 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 5950, 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.001041568, 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000175 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 471664 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-03-19T01:28:16.519 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2263, 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1.674891531, 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000740119 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 95, 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.016500723, 2024-03-19T01:28:16.520 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000173691 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 1802, 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 1632, 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1632, 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 4.757583225, 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002915185 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 170, 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-03-19T01:28:16.521 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 482, 2024-03-19T01:28:16.522 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 3, 2024-03-19T01:28:16.522 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-03-19T01:28:16.522 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 50, 2024-03-19T01:28:16.522 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 17, 2024-03-19T01:28:16.522 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 20, 2024-03-19T01:28:16.522 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 13, 2024-03-19T01:28:16.522 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 22, 2024-03-19T01:28:16.522 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 1097, 2024-03-19T01:28:16.522 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 6, 2024-03-19T01:28:16.522 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 6, 2024-03-19T01:28:16.522 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 9, 2024-03-19T01:28:16.522 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 2162, 2024-03-19T01:28:16.522 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 1478, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 166, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 36, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 16, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 808, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 48, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 1025, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 19, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 3, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2024-03-19T01:28:16.523 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 1028, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 1730125108, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 2, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 98, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 1216, 2024-03-19T01:28:16.524 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 1054, 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 927, 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 20, 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 196 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 154, 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 95, 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 48, 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 123, 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 318, 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2024-03-19T01:28:16.525 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 39, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 24, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 15, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T01:28:16.526 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 48, 2024-03-19T01:28:16.527 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-03-19T01:28:16.527 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 3, 2024-03-19T01:28:16.527 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-03-19T01:28:16.527 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-03-19T01:28:16.527 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.527 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-03-19T01:28:16.527 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-03-19T01:28:16.527 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 2735, 2024-03-19T01:28:16.527 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 2732, 2024-03-19T01:28:16.527 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 2732, 2024-03-19T01:28:16.527 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 3, 2024-03-19T01:28:16.527 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-03-19T01:28:16.528 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 2, 2024-03-19T01:28:16.528 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 74, 2024-03-19T01:28:16.528 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 73, 2024-03-19T01:28:16.528 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 73, 2024-03-19T01:28:16.528 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 1, 2024-03-19T01:28:16.528 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 1, 2024-03-19T01:28:16.528 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-03-19T01:28:16.528 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 1, 2024-03-19T01:28:16.528 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 11143753, 2024-03-19T01:28:16.528 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 11159494, 2024-03-19T01:28:16.528 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 0, 2024-03-19T01:28:16.528 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-03-19T01:28:16.528 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 699, 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 3.584677834, 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.005128294 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 0 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 53, 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 1066, 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 1013, 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 49, 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 126, 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 77, 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 50, 2024-03-19T01:28:16.529 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 1439, 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 1389, 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 6, 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 1050, 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 1044, 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 72688, 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 231804 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 1942, 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 209, 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 1802, 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 100, 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 350, 2024-03-19T01:28:16.530 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1003, 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 3.023779339, 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003014735 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 22, 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.020002179, 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000909189 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.531 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 15, 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.031917610, 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002127840 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 39, 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.866244902, 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.022211407 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.532 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1, 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000145268, 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000145268 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.533 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.534 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-03-19T01:28:16.534 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.534 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.534 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.534 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.534 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-03-19T01:28:16.534 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.535 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.535 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.535 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.535 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-03-19T01:28:16.535 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 37, 2024-03-19T01:28:16.535 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.172279944, 2024-03-19T01:28:16.535 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.004656214 2024-03-19T01:28:16.535 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.535 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-03-19T01:28:16.535 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 403, 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.123748439, 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000307068 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-03-19T01:28:16.536 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 39, 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.126853258, 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003252647 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 38, 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.242953734, 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.006393519 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.537 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-03-19T01:28:16.538 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.538 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.538 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.538 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.538 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-03-19T01:28:16.538 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2, 2024-03-19T01:28:16.538 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.018343858, 2024-03-19T01:28:16.538 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.009171929 2024-03-19T01:28:16.538 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.538 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-03-19T01:28:16.538 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.538 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.538 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1, 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.001426024, 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001426024 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.539 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-03-19T01:28:16.540 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 7, 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.017537755, 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002505393 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 25, 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.112350915, 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.004494036 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.541 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 2, 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 1, 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 1, 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-03-19T01:28:16.542 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 7, 2024-03-19T01:28:16.543 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 7, 2024-03-19T01:28:16.543 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 940, 2024-03-19T01:28:16.543 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-03-19T01:28:16.543 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.543 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-03-19T01:28:16.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 240, 2024-03-19T01:28:16.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 240, 2024-03-19T01:28:16.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T01:28:16.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-03-19T01:28:16.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T01:28:16.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-03-19T01:28:16.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T01:28:16.543 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T01:28:16.544 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T01:28:16.545 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 360365, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 149, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 7720, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 162, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-03-19T01:28:16.546 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 234509, 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 3347, 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 3064, 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 8697419, 2024-03-19T01:28:16.547 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 37, 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 3064, 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3064, 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 88.382712405, 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.028845532 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3064, 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 5920 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 3064, 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 8, 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 3056, 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-03-19T01:28:16.548 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 422, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 963, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 8, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 707, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 232, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 3, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 552, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-03-19T01:28:16.549 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 0, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 1926, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-03-19T01:28:16.550 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-03-19T01:28:16.551 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 87, 2024-03-19T01:28:16.553 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-03-19T01:28:16.553 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 3, 2024-03-19T01:28:16.553 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 12, 2024-03-19T01:28:16.553 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 12, 2024-03-19T01:28:16.553 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2024-03-19T01:28:16.553 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-03-19T01:28:16.553 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 1068, 2024-03-19T01:28:16.553 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 0, 2024-03-19T01:28:16.553 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 39 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 1, 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 13, 2024-03-19T01:28:16.554 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 6, 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 38, 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 24, 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 15817, 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 4676442, 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 15817, 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-19T01:28:16.555 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 15817, 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 4676442, 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:16.556 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 3064, 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 8699843, 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 3064, 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 8699843, 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-19T01:28:16.557 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 3064, 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 3064, 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 3064, 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 3064, 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-19T01:28:16.558 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 24, 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 2424, 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 24, 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 8, 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 2424, 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:16.559 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x557d072dc500": { 2024-03-19T01:28:16.560 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:16.560 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-19T01:28:16.560 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:16.560 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 2735, 2024-03-19T01:28:16.560 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 6965190, 2024-03-19T01:28:16.560 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:16.560 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 2735, 2024-03-19T01:28:16.560 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-19T01:28:16.560 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-19T01:28:16.560 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 699, 2024-03-19T01:28:16.560 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 6965190, 2024-03-19T01:28:16.560 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:16.561 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:16.561 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:16.561 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:16.561 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:16.561 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:16.561 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-19T01:28:16.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:16 smithi162 ceph-mon[67178]: pgmap v821: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-03-19T01:28:16.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:16 smithi116 ceph-mon[62701]: pgmap v821: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-03-19T01:28:17.361 INFO:tasks.check_counter:Daemon mds.b mds.exported=48 2024-03-19T01:28:17.361 INFO:tasks.check_counter:Daemon mds.b mds.imported=19 2024-03-19T01:28:17.361 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=154 2024-03-19T01:28:17.361 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=95 2024-03-19T01:28:17.362 DEBUG:tasks.check_counter:Getting stats from e 2024-03-19T01:28:17.362 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-19T01:28:17.636 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T01:28:18.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:18 smithi156 ceph-mon[66799]: pgmap v822: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-03-19T01:28:18.724 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:18 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:28:18.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:18 smithi162 ceph-mon[67178]: pgmap v822: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-03-19T01:28:18.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:18 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:28:18.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:18 smithi116 ceph-mon[62701]: pgmap v822: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.1 KiB/s wr, 11 op/s 2024-03-19T01:28:18.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:18 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-19T01:28:19.500 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:28:19.500 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:28:19.500 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:19 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:28:19.500 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-19T01:28:19.500 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T01:28:19.500 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1199, 2024-03-19T01:28:19.500 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1201, 2024-03-19T01:28:19.500 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 4586517, 2024-03-19T01:28:19.500 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 381292, 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 7, 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 2, 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.106959867, 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.035053880, 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.028639003, 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.034386706, 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1199, 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.055216231, 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000046051 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.501 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.502 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.502 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.502 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T01:28:19.502 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T01:28:19.502 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.502 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T01:28:19.502 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1263, 2024-03-19T01:28:19.502 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1230, 2024-03-19T01:28:19.502 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 936506, 2024-03-19T01:28:19.502 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 477044, 2024-03-19T01:28:19.502 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 5, 2024-03-19T01:28:19.502 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.104992613, 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.029827091, 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.033555842, 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.035795781, 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1230, 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.064790478, 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000052675 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.503 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 707792, 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 182192 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1222, 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1443, 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 3091544, 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 856967, 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 4, 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 3, 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.091259058, 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.030906838, 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.024260060, 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.027830768, 2024-03-19T01:28:19.504 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1441, 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.065143483, 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000045207 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 468688 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-03-19T01:28:19.505 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 5558, 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.155862303, 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000028042 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.506 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 0, 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 0, 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 0, 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-03-19T01:28:19.507 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 0, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 56, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 32, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 24, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 0, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 10, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 0, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 0, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 0, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 3, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 0, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 0, 2024-03-19T01:28:19.508 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-03-19T01:28:19.509 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 1028, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 1730125108, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 0, 2024-03-19T01:28:19.510 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T01:28:19.511 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T01:28:19.511 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T01:28:19.511 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 0, 2024-03-19T01:28:19.511 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2024-03-19T01:28:19.511 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 0, 2024-03-19T01:28:19.511 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 0 2024-03-19T01:28:19.511 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.511 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-03-19T01:28:19.511 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 0, 2024-03-19T01:28:19.511 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 0, 2024-03-19T01:28:19.511 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 0, 2024-03-19T01:28:19.511 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 0, 2024-03-19T01:28:19.511 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-03-19T01:28:19.512 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 0, 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 2736, 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-03-19T01:28:19.513 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 0, 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 2, 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 72, 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 11143753, 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 11164707, 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 11159494, 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.514 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 2736 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 18, 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 1049, 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 1031, 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 13, 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 68, 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 55, 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 56, 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 1082, 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 1026, 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 0, 2024-03-19T01:28:19.515 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 0, 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 0, 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 51152, 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 215420 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 0, 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 0, 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 0, 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 0, 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.516 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.517 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.518 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.519 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.520 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.520 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.520 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-03-19T01:28:19.520 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.520 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.520 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.520 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.521 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-03-19T01:28:19.521 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.521 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.521 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.521 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.521 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-03-19T01:28:19.521 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.521 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.521 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.521 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.521 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-03-19T01:28:19.521 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.522 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-03-19T01:28:19.523 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.523 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.523 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.523 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.523 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-03-19T01:28:19.523 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.523 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.523 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.523 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.523 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-03-19T01:28:19.523 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.523 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.523 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.524 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.525 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-03-19T01:28:19.526 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.527 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 2, 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 1, 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 1, 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 0, 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 0, 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 4795, 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-03-19T01:28:19.528 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T01:28:19.529 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-03-19T01:28:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-03-19T01:28:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-03-19T01:28:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-03-19T01:28:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T01:28:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T01:28:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T01:28:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T01:28:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T01:28:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-03-19T01:28:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-03-19T01:28:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-03-19T01:28:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T01:28:19.530 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 190284, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 55, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-03-19T01:28:19.531 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 7720, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 162, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 96139, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 467, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-03-19T01:28:19.532 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-03-19T01:28:19.533 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-03-19T01:28:19.533 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-03-19T01:28:19.533 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.533 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-03-19T01:28:19.533 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-03-19T01:28:19.533 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-03-19T01:28:19.533 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 3347, 2024-03-19T01:28:19.533 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 26, 2024-03-19T01:28:19.533 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 4, 2024-03-19T01:28:19.533 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 3347, 2024-03-19T01:28:19.533 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-03-19T01:28:19.533 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3347, 2024-03-19T01:28:19.533 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 9.933860496, 2024-03-19T01:28:19.534 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002967989 2024-03-19T01:28:19.534 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.534 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-03-19T01:28:19.534 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-03-19T01:28:19.534 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3347, 2024-03-19T01:28:19.534 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 3348 2024-03-19T01:28:19.534 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.534 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 3347, 2024-03-19T01:28:19.534 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 3347, 2024-03-19T01:28:19.534 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 0, 2024-03-19T01:28:19.534 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-03-19T01:28:19.534 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-03-19T01:28:19.534 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 2083, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 0, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 1261, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 0, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 0, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 0, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 2, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 0, 2024-03-19T01:28:19.535 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-03-19T01:28:19.536 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 87, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 2, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 5, 2024-03-19T01:28:19.537 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 7, 2024-03-19T01:28:19.538 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 2, 2024-03-19T01:28:19.538 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-03-19T01:28:19.538 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 0, 2024-03-19T01:28:19.538 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 2, 2024-03-19T01:28:19.538 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 0 2024-03-19T01:28:19.538 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.538 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-03-19T01:28:19.538 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 0, 2024-03-19T01:28:19.538 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-03-19T01:28:19.538 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-03-19T01:28:19.538 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-03-19T01:28:19.538 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.538 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 0, 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 0, 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 0, 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:19.539 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 3680, 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 8263399, 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 3680, 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 3680, 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 8263399, 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:19.540 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 3347, 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 6970413, 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 3347, 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 6970413, 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.541 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.542 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.542 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:19.542 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.542 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-03-19T01:28:19.542 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:19.542 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-03-19T01:28:19.542 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:19.542 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-19T01:28:19.542 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-19T01:28:19.542 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:19.542 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:19.542 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 3347, 2024-03-19T01:28:19.542 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 3347, 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 3347, 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 3347, 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:19.543 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x55cc6e2c26e0": { 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-19T01:28:19.544 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-19T01:28:19.545 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:19.545 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:19.545 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-19T01:28:19.545 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-19T01:28:19.545 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-03-19T01:28:19.545 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-03-19T01:28:19.545 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:19.545 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:19.545 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:19.545 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:19.545 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:19.545 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:19.545 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-19T01:28:19.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:28:19.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:28:19.915 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:19 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:28:19.916 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-19T01:28:19.916 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-19T01:28:19.916 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:19 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' 2024-03-19T01:28:20.309 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-19T01:28:20.310 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-19T01:28:20.310 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-03-19T01:28:20.310 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-03-19T01:28:20.310 DEBUG:tasks.check_counter:Getting stats from h 2024-03-19T01:28:20.310 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-19T01:28:20.565 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T01:28:21.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:21 smithi162 ceph-mon[67178]: pgmap v823: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-03-19T01:28:21.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:21 smithi116 ceph-mon[62701]: pgmap v823: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-03-19T01:28:21.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:21 smithi156 ceph-mon[66799]: pgmap v823: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 13 op/s 2024-03-19T01:28:22.325 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-19T01:28:22.325 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T01:28:22.325 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 3815, 2024-03-19T01:28:22.325 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 4583, 2024-03-19T01:28:22.325 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 1557427, 2024-03-19T01:28:22.325 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 5193132, 2024-03-19T01:28:22.326 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 11, 2024-03-19T01:28:22.326 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 5, 2024-03-19T01:28:22.326 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.326361155, 2024-03-19T01:28:22.326 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.152838062, 2024-03-19T01:28:22.326 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.180552104, 2024-03-19T01:28:22.326 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.020018374, 2024-03-19T01:28:22.326 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:22.326 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 4584, 2024-03-19T01:28:22.326 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.291647822, 2024-03-19T01:28:22.326 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000063622 2024-03-19T01:28:22.326 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.326 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:22.326 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 6982, 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.001224643, 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000175 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 588112, 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 1587856 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 2174, 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 2075, 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 1405703, 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 865332, 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 9, 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 7, 2024-03-19T01:28:22.327 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.152498990, 2024-03-19T01:28:22.328 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.063824146, 2024-03-19T01:28:22.328 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.090295178, 2024-03-19T01:28:22.328 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.012984913, 2024-03-19T01:28:22.328 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:22.328 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2074, 2024-03-19T01:28:22.328 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.151887753, 2024-03-19T01:28:22.328 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000073234 2024-03-19T01:28:22.328 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.328 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:22.328 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2899, 2024-03-19T01:28:22.328 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000429783, 2024-03-19T01:28:22.328 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000148 2024-03-19T01:28:22.328 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 727600, 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 187024 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1089, 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1134, 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 608302, 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 1260676, 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 16, 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 6, 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.076036768, 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.031135050, 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.045374448, 2024-03-19T01:28:22.329 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.006229819, 2024-03-19T01:28:22.330 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:22.330 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1130, 2024-03-19T01:28:22.330 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.067394449, 2024-03-19T01:28:22.330 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000059641 2024-03-19T01:28:22.330 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.330 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:22.330 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1414, 2024-03-19T01:28:22.330 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000239823, 2024-03-19T01:28:22.330 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000169 2024-03-19T01:28:22.330 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.330 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 1248, 2024-03-19T01:28:22.330 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 560032 2024-03-19T01:28:22.331 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.331 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-03-19T01:28:22.331 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-03-19T01:28:22.331 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-03-19T01:28:22.331 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.331 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-03-19T01:28:22.331 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-19T01:28:22.331 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-19T01:28:22.331 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 875, 2024-03-19T01:28:22.331 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.600175049, 2024-03-19T01:28:22.331 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000685914 2024-03-19T01:28:22.331 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:22.331 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 711, 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.030736685, 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000043230 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 894, 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 875, 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 875, 2024-03-19T01:28:22.332 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 2.542530605, 2024-03-19T01:28:22.333 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002905749 2024-03-19T01:28:22.333 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.333 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-03-19T01:28:22.333 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 19, 2024-03-19T01:28:22.333 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-03-19T01:28:22.333 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-03-19T01:28:22.333 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 11, 2024-03-19T01:28:22.333 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2024-03-19T01:28:22.333 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-03-19T01:28:22.333 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 190, 2024-03-19T01:28:22.333 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 10, 2024-03-19T01:28:22.333 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 159, 2024-03-19T01:28:22.333 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 21, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 175, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 105, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 150, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 150, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 11, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 2373, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 1975, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 19, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 33, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 76, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 5253, 2024-03-19T01:28:22.334 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 17, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 96, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 20, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 244, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 4, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 234, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T01:28:22.335 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 43, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 649, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 597, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 298, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 134, 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 885 2024-03-19T01:28:22.336 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.337 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-03-19T01:28:22.337 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 109, 2024-03-19T01:28:22.337 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 316, 2024-03-19T01:28:22.337 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 169, 2024-03-19T01:28:22.337 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 277, 2024-03-19T01:28:22.337 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 112, 2024-03-19T01:28:22.337 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 3, 2024-03-19T01:28:22.337 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-03-19T01:28:22.337 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 217, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 214, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 17, 2024-03-19T01:28:22.338 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 1714, 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 1714, 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 3, 2024-03-19T01:28:22.339 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 3, 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 2, 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 4194304, 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 7008114, 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 0, 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 356, 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 6.765888862, 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.019005305 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 0 2024-03-19T01:28:22.340 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 185, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 496, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 311, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 55, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 177, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 122, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 185, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 509, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 324, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 150, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 600, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 450, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 57560, 2024-03-19T01:28:22.341 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 223612 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 1489, 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 894, 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 19, 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 449, 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 214, 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.352189645, 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001645746 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-03-19T01:28:22.342 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 2, 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.006457902, 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003228951 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.343 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 295, 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1.371950736, 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.004650680 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.344 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.345 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-03-19T01:28:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.346 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.347 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 144, 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.092409918, 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000641735 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3, 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.003027618, 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.001009206 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-03-19T01:28:22.348 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 213, 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.705803003, 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003313629 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.349 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.350 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-03-19T01:28:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-03-19T01:28:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-03-19T01:28:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.351 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-03-19T01:28:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-03-19T01:28:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.352 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 4, 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.010691783, 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.002672945 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-03-19T01:28:22.353 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-03-19T01:28:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2024-03-19T01:28:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 2, 2024-03-19T01:28:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 1, 2024-03-19T01:28:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 1, 2024-03-19T01:28:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-03-19T01:28:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 95, 2024-03-19T01:28:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 95, 2024-03-19T01:28:22.354 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 940, 2024-03-19T01:28:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-03-19T01:28:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-03-19T01:28:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-03-19T01:28:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-03-19T01:28:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T01:28:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-03-19T01:28:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T01:28:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-03-19T01:28:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T01:28:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T01:28:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T01:28:22.355 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T01:28:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T01:28:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-03-19T01:28:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T01:28:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T01:28:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-03-19T01:28:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-03-19T01:28:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-03-19T01:28:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-03-19T01:28:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T01:28:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T01:28:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T01:28:22.356 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T01:28:22.357 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-03-19T01:28:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 359690, 2024-03-19T01:28:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 314, 2024-03-19T01:28:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 704, 2024-03-19T01:28:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 8, 2024-03-19T01:28:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-03-19T01:28:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-03-19T01:28:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-03-19T01:28:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-03-19T01:28:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-03-19T01:28:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-03-19T01:28:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 7720, 2024-03-19T01:28:22.358 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 162, 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 719647, 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 7956, 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-03-19T01:28:22.359 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 1005, 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 2899855, 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 28, 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 1005, 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1005, 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 10.977845773, 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.010923229 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1005, 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 1132 2024-03-19T01:28:22.360 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 1005, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 84, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 921, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 0, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 10, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 84, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 440, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 134, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-03-19T01:28:22.361 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 226, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 0, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 20, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T01:28:22.362 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-03-19T01:28:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-03-19T01:28:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-03-19T01:28:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-03-19T01:28:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-03-19T01:28:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-03-19T01:28:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-03-19T01:28:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-03-19T01:28:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-03-19T01:28:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-03-19T01:28:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-03-19T01:28:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-03-19T01:28:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-03-19T01:28:22.363 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-03-19T01:28:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-03-19T01:28:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-03-19T01:28:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-03-19T01:28:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-03-19T01:28:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-03-19T01:28:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 87, 2024-03-19T01:28:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-03-19T01:28:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 2, 2024-03-19T01:28:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 12, 2024-03-19T01:28:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 12, 2024-03-19T01:28:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 0, 2024-03-19T01:28:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-03-19T01:28:22.364 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 157, 2024-03-19T01:28:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 0, 2024-03-19T01:28:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 61 2024-03-19T01:28:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-03-19T01:28:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 1, 2024-03-19T01:28:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-03-19T01:28:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-03-19T01:28:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-03-19T01:28:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-03-19T01:28:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-03-19T01:28:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.365 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 6, 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 6, 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 214, 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 214, 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 7074, 2024-03-19T01:28:22.366 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 2836527, 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 7074, 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 7074, 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 2836527, 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-03-19T01:28:22.367 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 1005, 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 2921469, 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1005, 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 2921469, 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.368 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-03-19T01:28:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-03-19T01:28:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-19T01:28:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-19T01:28:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 1005, 2024-03-19T01:28:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 1005, 2024-03-19T01:28:22.369 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 1005, 2024-03-19T01:28:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 1005, 2024-03-19T01:28:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-03-19T01:28:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-19T01:28:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 214, 2024-03-19T01:28:22.370 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 21614, 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 214, 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 84, 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 21614, 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x5566816ec500": { 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 6762, 2024-03-19T01:28:22.371 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 1714, 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 2813810, 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 1714, 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 356, 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 2807048, 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:22.372 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:22.373 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:22.373 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-19T01:28:23.156 INFO:tasks.check_counter:Daemon mds.h mds.exported=17 2024-03-19T01:28:23.156 INFO:tasks.check_counter:Daemon mds.h mds.imported=20 2024-03-19T01:28:23.157 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=109 2024-03-19T01:28:23.157 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=316 2024-03-19T01:28:23.157 DEBUG:tasks.check_counter:Getting stats from k 2024-03-19T01:28:23.157 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-19T01:28:23.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:23 smithi162 ceph-mon[67178]: pgmap v824: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.4 KiB/s wr, 15 op/s 2024-03-19T01:28:23.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:23 smithi116 ceph-mon[62701]: pgmap v824: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.4 KiB/s wr, 15 op/s 2024-03-19T01:28:23.416 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.b/config 2024-03-19T01:28:23.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:23 smithi156 ceph-mon[66799]: pgmap v824: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 7.4 KiB/s wr, 15 op/s 2024-03-19T01:28:25.202 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:25 smithi156 ceph-mon[66799]: pgmap v825: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 9.8 KiB/s wr, 12 op/s 2024-03-19T01:28:25.202 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:25 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:25.246 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-03-19T01:28:25.247 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T01:28:25.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1242, 2024-03-19T01:28:25.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1244, 2024-03-19T01:28:25.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 3130163, 2024-03-19T01:28:25.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 395020, 2024-03-19T01:28:25.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 7, 2024-03-19T01:28:25.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 2, 2024-03-19T01:28:25.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.095639724, 2024-03-19T01:28:25.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.031736346, 2024-03-19T01:28:25.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.025300324, 2024-03-19T01:28:25.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.029794444, 2024-03-19T01:28:25.247 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:25.248 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1242, 2024-03-19T01:28:25.248 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.062452157, 2024-03-19T01:28:25.248 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000050283 2024-03-19T01:28:25.248 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.248 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:25.248 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.248 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.248 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.248 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.248 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T01:28:25.248 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T01:28:25.248 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.248 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T01:28:25.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 981, 2024-03-19T01:28:25.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1205, 2024-03-19T01:28:25.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 241262, 2024-03-19T01:28:25.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 782477, 2024-03-19T01:28:25.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 5, 2024-03-19T01:28:25.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 4, 2024-03-19T01:28:25.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.083559020, 2024-03-19T01:28:25.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.025541942, 2024-03-19T01:28:25.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.017992876, 2024-03-19T01:28:25.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.031701532, 2024-03-19T01:28:25.249 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:25.249 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1203, 2024-03-19T01:28:25.249 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.050175804, 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000041708 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 470832 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_messages": 1395, 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages": 1361, 2024-03-19T01:28:25.250 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_bytes": 1070016, 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_bytes": 519760, 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_created_connections": 3, 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_active_connections": 2, 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_total_time": 0.126099861, 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_send_time": 0.040113937, 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_recv_time": 0.041915122, 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_running_fast_dispatch_time": 0.037705921, 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 1361, 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.061073012, 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000044873 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:25.251 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_recv_encrypted_bytes": 715200, 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: "msgr_send_encrypted_bytes": 183952 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: "cct": { 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: "total_workers": 1, 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: "unhealthy_workers": 0 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-MDSRank": { 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-19T01:28:25.252 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 5524, 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.146558434, 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000026531 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: "finisher-PurgeQueue": { 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: "queue_len": 0, 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: "complete_latency": { 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: "mds": { 2024-03-19T01:28:25.253 INFO:teuthology.orchestra.run.smithi156.stdout: "request": 0, 2024-03-19T01:28:25.254 INFO:teuthology.orchestra.run.smithi156.stdout: "reply": 0, 2024-03-19T01:28:25.254 INFO:teuthology.orchestra.run.smithi156.stdout: "reply_latency": { 2024-03-19T01:28:25.254 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.254 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.254 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.254 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.254 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_reply": 0, 2024-03-19T01:28:25.254 INFO:teuthology.orchestra.run.smithi156.stdout: "forward": 0, 2024-03-19T01:28:25.254 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_complete": 0, 2024-03-19T01:28:25.254 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_fetch_keys": 0, 2024-03-19T01:28:25.254 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_commit": 0, 2024-03-19T01:28:25.254 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_split": 0, 2024-03-19T01:28:25.254 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_merge": 0, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes": 377, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_top": 4, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_bottom": 373, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pin_tail": 0, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_pinned": 371, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_expired": 0, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "inodes_with_caps": 0, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "caps": 0, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "subtrees": 5, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse": 0, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_hit": 0, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_forward": 0, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_discover": 0, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_dir_fetch": 0, 2024-03-19T01:28:25.255 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_remote_ino": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "traverse_lock": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "load_cent": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "q": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "exported": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "exported_inodes": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "imported": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "imported_inodes": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_dir_fetch": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_backtrace_fetch": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "openino_peer_discover": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rfiles": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rbytes": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "root_rsnaps": 0, 2024-03-19T01:28:25.256 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_set_tag": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_inotable_repaired": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_inodes": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "scrub_file_inodes": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_inode_file_caps": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_grant": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps": 0, 2024-03-19T01:28:25.257 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_caps_dirty": 0, 2024-03-19T01:28:25.258 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_cap_release": 0, 2024-03-19T01:28:25.258 INFO:teuthology.orchestra.run.smithi156.stdout: "process_request_cap_release": 0 2024-03-19T01:28:25.258 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.258 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_cache": { 2024-03-19T01:28:25.258 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update": 0, 2024-03-19T01:28:25.258 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_update_receipt": 0, 2024-03-19T01:28:25.258 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_try_discover": 0, 2024-03-19T01:28:25.258 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_send_discover": 0, 2024-03-19T01:28:25.258 INFO:teuthology.orchestra.run.smithi156.stdout: "dir_handle_discover": 0, 2024-03-19T01:28:25.258 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays": 0, 2024-03-19T01:28:25.258 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_delayed": 0, 2024-03-19T01:28:25.258 INFO:teuthology.orchestra.run.smithi156.stdout: "num_strays_enqueuing": 0, 2024-03-19T01:28:25.258 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_created": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_enqueued": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_reintegrated": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "strays_migrated": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_processing": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_enqueued": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "num_recovering_prioritized": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_started": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "recovery_completed": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_path": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_quiesce_inode": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_exportdir": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_flush": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragmentdir": 0, 2024-03-19T01:28:25.259 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_fragstats": 0, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "ireq_inodestats": 0 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_log": { 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "evlrg": 0, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "evadd": 0, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "evex": 0, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "evtrm": 0, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "ev": 1711, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "evexg": 0, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "evexd": 0, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "segadd": 0, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "segex": 0, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "segtrm": 0, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "seg": 3, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "segmjr": 2, 2024-03-19T01:28:25.260 INFO:teuthology.orchestra.run.smithi156.stdout: "segexg": 0, 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: "segexd": 0, 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: "expos": 4194304, 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: "wrpos": 7001352, 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: "rdpos": 7000214, 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: "jlat": { 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: "replayed": 1711 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_mem": { 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: "ino": 159, 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: "ino+": 476, 2024-03-19T01:28:25.261 INFO:teuthology.orchestra.run.smithi156.stdout: "ino-": 317, 2024-03-19T01:28:25.262 INFO:teuthology.orchestra.run.smithi156.stdout: "dir": 14, 2024-03-19T01:28:25.262 INFO:teuthology.orchestra.run.smithi156.stdout: "dir+": 31, 2024-03-19T01:28:25.262 INFO:teuthology.orchestra.run.smithi156.stdout: "dir-": 17, 2024-03-19T01:28:25.262 INFO:teuthology.orchestra.run.smithi156.stdout: "dn": 377, 2024-03-19T01:28:25.262 INFO:teuthology.orchestra.run.smithi156.stdout: "dn+": 488, 2024-03-19T01:28:25.262 INFO:teuthology.orchestra.run.smithi156.stdout: "dn-": 111, 2024-03-19T01:28:25.262 INFO:teuthology.orchestra.run.smithi156.stdout: "cap": 0, 2024-03-19T01:28:25.262 INFO:teuthology.orchestra.run.smithi156.stdout: "cap+": 0, 2024-03-19T01:28:25.262 INFO:teuthology.orchestra.run.smithi156.stdout: "cap-": 0, 2024-03-19T01:28:25.262 INFO:teuthology.orchestra.run.smithi156.stdout: "rss": 50300, 2024-03-19T01:28:25.262 INFO:teuthology.orchestra.run.smithi156.stdout: "heap": 215420 2024-03-19T01:28:25.262 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.262 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_server": { 2024-03-19T01:28:25.263 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_client_request": 0, 2024-03-19T01:28:25.263 INFO:teuthology.orchestra.run.smithi156.stdout: "dispatch_server_request": 0, 2024-03-19T01:28:25.263 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_request": 0, 2024-03-19T01:28:25.263 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_client_session": 0, 2024-03-19T01:28:25.263 INFO:teuthology.orchestra.run.smithi156.stdout: "handle_peer_request": 0, 2024-03-19T01:28:25.263 INFO:teuthology.orchestra.run.smithi156.stdout: "req_create_latency": { 2024-03-19T01:28:25.263 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.263 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.263 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.263 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.263 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getattr_latency": { 2024-03-19T01:28:25.263 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.263 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getfilelock_latency": { 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: "req_link_latency": { 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookup_latency": { 2024-03-19T01:28:25.264 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.265 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.265 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.265 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.265 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookuphash_latency": { 2024-03-19T01:28:25.265 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.265 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.265 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.265 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.265 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupino_latency": { 2024-03-19T01:28:25.265 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.265 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.265 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.265 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupname_latency": { 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupparent_latency": { 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lookupsnap_latency": { 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.266 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.267 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.267 INFO:teuthology.orchestra.run.smithi156.stdout: "req_lssnap_latency": { 2024-03-19T01:28:25.267 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.267 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.267 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.267 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.267 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mkdir_latency": { 2024-03-19T01:28:25.268 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.268 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.268 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.268 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.268 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mknod_latency": { 2024-03-19T01:28:25.268 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.268 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.268 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.268 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.268 INFO:teuthology.orchestra.run.smithi156.stdout: "req_mksnap_latency": { 2024-03-19T01:28:25.268 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.268 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.268 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.269 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.269 INFO:teuthology.orchestra.run.smithi156.stdout: "req_open_latency": { 2024-03-19T01:28:25.269 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.269 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.269 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.269 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.269 INFO:teuthology.orchestra.run.smithi156.stdout: "req_readdir_latency": { 2024-03-19T01:28:25.269 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.269 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.269 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.269 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.269 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rename_latency": { 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: "req_renamesnap_latency": { 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: "req_snapdiff_latency": { 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.270 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.271 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmdir_latency": { 2024-03-19T01:28:25.271 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.271 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.271 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.271 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.271 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmsnap_latency": { 2024-03-19T01:28:25.271 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.271 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.271 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.271 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.271 INFO:teuthology.orchestra.run.smithi156.stdout: "req_rmxattr_latency": { 2024-03-19T01:28:25.271 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.271 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setattr_latency": { 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setdirlayout_latency": { 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setfilelock_latency": { 2024-03-19T01:28:25.272 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setlayout_latency": { 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: "req_setxattr_latency": { 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: "req_symlink_latency": { 2024-03-19T01:28:25.273 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: "req_unlink_latency": { 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_revoke_eviction": 0, 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: "cap_acquisition_throttle": 0, 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: "req_getvxattr_latency": { 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.274 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_sessions": { 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: "session_count": 1, 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: "session_add": 2, 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: "session_remove": 1, 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_open": 1, 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: "sessions_stale": 0, 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: "total_load": 0, 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: "average_load": 0, 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: "avg_session_uptime": 4801, 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: "mdthresh_evicted": 0 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.275 INFO:teuthology.orchestra.run.smithi156.stdout: "mempool": { 2024-03-19T01:28:25.276 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_bytes": 0, 2024-03-19T01:28:25.276 INFO:teuthology.orchestra.run.smithi156.stdout: "bloom_filter_items": 0, 2024-03-19T01:28:25.276 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T01:28:25.276 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_alloc_items": 0, 2024-03-19T01:28:25.276 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T01:28:25.276 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_data_items": 0, 2024-03-19T01:28:25.276 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T01:28:25.276 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T01:28:25.276 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_other_items": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_bytes": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_extent_items": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_bytes": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_blob_items": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T01:28:25.277 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T01:28:25.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_fsck_items": 0, 2024-03-19T01:28:25.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_bytes": 0, 2024-03-19T01:28:25.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_txc_items": 0, 2024-03-19T01:28:25.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T01:28:25.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T01:28:25.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_bytes": 0, 2024-03-19T01:28:25.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluestore_writing_items": 0, 2024-03-19T01:28:25.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_bytes": 0, 2024-03-19T01:28:25.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_items": 0, 2024-03-19T01:28:25.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T01:28:25.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_reader_items": 0, 2024-03-19T01:28:25.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T01:28:25.278 INFO:teuthology.orchestra.run.smithi156.stdout: "bluefs_file_writer_items": 0, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_bytes": 195734, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_anon_items": 203, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_bytes": 0, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "buffer_meta_items": 0, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_bytes": 0, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_items": 0, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_bytes": 0, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_mapbl_items": 0, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_bytes": 0, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_pglog_items": 0, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_bytes": 7720, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_items": 162, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T01:28:25.279 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_mapping_items": 0, 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_bytes": 0, 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: "pgmap_items": 0, 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_bytes": 644033, 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_co_items": 5252, 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_bytes": 0, 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_1_items": 0, 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_bytes": 0, 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: "unittest_2_items": 0 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: "objecter": { 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: "op_active": 0, 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: "op_laggy": 0, 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send": 3278, 2024-03-19T01:28:25.280 INFO:teuthology.orchestra.run.smithi156.stdout: "op_send_bytes": 21, 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: "op_resend": 3, 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: "op_reply": 3278, 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: "op_latency": { 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3278, 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 10.966845999, 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.003345590 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: "op_inflight": 0, 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: "oplen_avg": { 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 3278, 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 3279 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: "op": 3278, 2024-03-19T01:28:25.281 INFO:teuthology.orchestra.run.smithi156.stdout: "op_r": 3278, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "op_w": 0, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "op_rmw": 0, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "op_pg": 0, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_stat": 2034, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_create": 0, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_read": 1242, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_write": 0, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writefull": 0, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_writesame": 0, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_append": 0, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_zero": 0, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_truncate": 0, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_delete": 0, 2024-03-19T01:28:25.282 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_mapext": 0, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_sparse_read": 0, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_clonerange": 0, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_getxattr": 1, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_setxattr": 0, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_cmpxattr": 0, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_rmxattr": 0, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_resetxattrs": 0, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_call": 0, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_watch": 0, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_notify": 0, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls": 0, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_pgls_filter": 0, 2024-03-19T01:28:25.283 INFO:teuthology.orchestra.run.smithi156.stdout: "osdop_other": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_active": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_send": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_resend": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "linger_ping": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_active": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_send": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "poolop_resend": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_active": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_send": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "poolstat_resend": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_active": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_send": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "statfs_resend": 0, 2024-03-19T01:28:25.284 INFO:teuthology.orchestra.run.smithi156.stdout: "command_active": 0, 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "command_send": 0, 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "command_resend": 0, 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "map_epoch": 87, 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "map_full": 0, 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "map_inc": 2, 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_sessions": 5, 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_open": 6, 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_session_close": 1, 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_laggy": 0, 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_wr": 0, 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_rd": 2, 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_del": 0 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "oft": { 2024-03-19T01:28:25.285 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_objs": 0, 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_kv_pairs": 0, 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_updates": 0, 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: "omap_total_removes": 0 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: "osd-slow-ops": { 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: "slow_ops_count": 0 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: "purge_queue": { 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops": 0, 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing": 0, 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executing_high_water": 0, 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed_ops": 0, 2024-03-19T01:28:25.286 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_executed": 0, 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "pq_item_in_journal": 0 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 3614, 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 4096062, 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 3614, 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 3614, 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 4096062, 2024-03-19T01:28:25.287 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:25.288 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.288 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.288 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.288 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:25.288 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.288 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_bytes": { 2024-03-19T01:28:25.288 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:25.288 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 104857600, 2024-03-19T01:28:25.288 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:25.288 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-19T01:28:25.288 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-19T01:28:25.288 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:25.288 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 3278, 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 2807053, 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 3278, 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 2807053, 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-objecter_ops": { 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 1024, 2024-03-19T01:28:25.289 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 3278, 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 3278, 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 3278, 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 3278, 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:25.290 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle": { 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:25.291 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.292 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.292 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.292 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:25.292 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-03-19T01:28:25.292 INFO:teuthology.orchestra.run.smithi156.stdout: "throttle-write_buf_throttle-0x55b7a8c946e0": { 2024-03-19T01:28:25.292 INFO:teuthology.orchestra.run.smithi156.stdout: "val": 0, 2024-03-19T01:28:25.292 INFO:teuthology.orchestra.run.smithi156.stdout: "max": 3758096384, 2024-03-19T01:28:25.292 INFO:teuthology.orchestra.run.smithi156.stdout: "get_started": 0, 2024-03-19T01:28:25.292 INFO:teuthology.orchestra.run.smithi156.stdout: "get": 0, 2024-03-19T01:28:25.292 INFO:teuthology.orchestra.run.smithi156.stdout: "get_sum": 0, 2024-03-19T01:28:25.292 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:25.292 INFO:teuthology.orchestra.run.smithi156.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:25.292 INFO:teuthology.orchestra.run.smithi156.stdout: "take": 0, 2024-03-19T01:28:25.293 INFO:teuthology.orchestra.run.smithi156.stdout: "take_sum": 0, 2024-03-19T01:28:25.293 INFO:teuthology.orchestra.run.smithi156.stdout: "put": 0, 2024-03-19T01:28:25.293 INFO:teuthology.orchestra.run.smithi156.stdout: "put_sum": 0, 2024-03-19T01:28:25.293 INFO:teuthology.orchestra.run.smithi156.stdout: "wait": { 2024-03-19T01:28:25.293 INFO:teuthology.orchestra.run.smithi156.stdout: "avgcount": 0, 2024-03-19T01:28:25.293 INFO:teuthology.orchestra.run.smithi156.stdout: "sum": 0.000000000, 2024-03-19T01:28:25.293 INFO:teuthology.orchestra.run.smithi156.stdout: "avgtime": 0.000000000 2024-03-19T01:28:25.293 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:25.293 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-03-19T01:28:25.293 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-03-19T01:28:25.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:25 smithi162 ceph-mon[67178]: pgmap v825: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 9.8 KiB/s wr, 12 op/s 2024-03-19T01:28:25.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:25 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:25.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:25 smithi116 ceph-mon[62701]: pgmap v825: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 9.8 KiB/s wr, 12 op/s 2024-03-19T01:28:25.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:25 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:26.101 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-19T01:28:26.101 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-19T01:28:26.101 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2024-03-19T01:28:26.101 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2024-03-19T01:28:26.101 DEBUG:tasks.check_counter:Getting stats from c 2024-03-19T01:28:26.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-19T01:28:26.353 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:28:27.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:27 smithi162 ceph-mon[67178]: pgmap v826: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 37 KiB/s wr, 18 op/s 2024-03-19T01:28:27.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:27 smithi116 ceph-mon[62701]: pgmap v826: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 37 KiB/s wr, 18 op/s 2024-03-19T01:28:27.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:27 smithi156 ceph-mon[66799]: pgmap v826: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 37 KiB/s wr, 18 op/s 2024-03-19T01:28:28.296 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-19T01:28:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T01:28:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 3756, 2024-03-19T01:28:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 3446, 2024-03-19T01:28:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 2416275, 2024-03-19T01:28:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 2282198, 2024-03-19T01:28:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 9, 2024-03-19T01:28:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 4, 2024-03-19T01:28:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.254158471, 2024-03-19T01:28:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.108060388, 2024-03-19T01:28:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.153796016, 2024-03-19T01:28:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.018087996, 2024-03-19T01:28:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3447, 2024-03-19T01:28:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.198481665, 2024-03-19T01:28:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000057580 2024-03-19T01:28:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 5696, 2024-03-19T01:28:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000979437, 2024-03-19T01:28:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000171 2024-03-19T01:28:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 730048, 2024-03-19T01:28:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 186224 2024-03-19T01:28:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T01:28:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 680, 2024-03-19T01:28:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 1275, 2024-03-19T01:28:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 368731, 2024-03-19T01:28:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 1199195, 2024-03-19T01:28:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 16, 2024-03-19T01:28:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 5, 2024-03-19T01:28:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.076220924, 2024-03-19T01:28:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.034445946, 2024-03-19T01:28:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.046930609, 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.000341763, 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1274, 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.073506902, 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000057697 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1593, 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000313348, 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000196 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 100464, 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 871568 2024-03-19T01:28:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 101, 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 105, 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 26391, 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 60904, 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 11, 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 6, 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.011286829, 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.002259750, 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.005240158, 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.003042167, 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 101, 2024-03-19T01:28:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.029904445, 2024-03-19T01:28:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000296083 2024-03-19T01:28:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-19T01:28:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 0 2024-03-19T01:28:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-03-19T01:28:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-03-19T01:28:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-03-19T01:28:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-03-19T01:28:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T01:28:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T01:28:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 609, 2024-03-19T01:28:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.097199127, 2024-03-19T01:28:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000159604 2024-03-19T01:28:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-03-19T01:28:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T01:28:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T01:28:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 17, 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000373933, 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000021996 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 112, 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 97, 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 97, 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.051355461, 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000529437 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-03-19T01:28:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 15, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 0, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 11, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 0, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 290, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 7, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 283, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 0, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 278, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 4, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 261, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 261, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 17, 2024-03-19T01:28:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 388, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 349, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 15, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 22, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 0, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 1166, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 0, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 0, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 8, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 263, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-03-19T01:28:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 0, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 4, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 234, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 0, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 0, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 4, 2024-03-19T01:28:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 2, 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 50, 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 135, 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 75, 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 7, 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 9 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 19, 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 372, 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 197, 2024-03-19T01:28:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 323, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 24, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 2, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 2, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 0, 2024-03-19T01:28:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 0, 2024-03-19T01:28:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T01:28:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 0, 2024-03-19T01:28:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-03-19T01:28:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 0, 2024-03-19T01:28:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-03-19T01:28:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-03-19T01:28:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-03-19T01:28:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-03-19T01:28:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 633, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 633, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 2, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 2, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 2, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 4194304, 2024-03-19T01:28:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 5114916, 2024-03-19T01:28:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 0, 2024-03-19T01:28:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-03-19T01:28:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 287, 2024-03-19T01:28:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 3.544112822, 2024-03-19T01:28:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.012348825 2024-03-19T01:28:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 0 2024-03-19T01:28:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-03-19T01:28:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 292, 2024-03-19T01:28:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 296, 2024-03-19T01:28:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 4, 2024-03-19T01:28:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 49, 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 195, 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 146, 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 290, 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 301, 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 11, 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 261, 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 477, 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 216, 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 57432, 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 223612 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 117, 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 2, 2024-03-19T01:28:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 112, 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 13, 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 7, 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-03-19T01:28:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 78, 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.031324232, 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000401592 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1, 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000105098, 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000105098 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-03-19T01:28:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-03-19T01:28:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-03-19T01:28:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-03-19T01:28:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-03-19T01:28:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-03-19T01:28:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-03-19T01:28:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 8, 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.005845517, 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000730689 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 8, 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.008879840, 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.001109980 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-03-19T01:28:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-03-19T01:28:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-03-19T01:28:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-03-19T01:28:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2, 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.005200774, 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002600387 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-03-19T01:28:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 1, 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 1, 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 0, 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 1, 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 25, 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 25, 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 954, 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T01:28:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-03-19T01:28:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T01:28:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-03-19T01:28:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T01:28:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T01:28:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T01:28:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T01:28:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T01:28:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-03-19T01:28:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T01:28:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T01:28:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-03-19T01:28:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-03-19T01:28:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-03-19T01:28:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-03-19T01:28:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-03-19T01:28:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T01:28:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-03-19T01:28:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T01:28:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-03-19T01:28:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 301036, 2024-03-19T01:28:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 389, 2024-03-19T01:28:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-03-19T01:28:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-03-19T01:28:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-03-19T01:28:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-03-19T01:28:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-03-19T01:28:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 7720, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 162, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 1044654, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 10195, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-03-19T01:28:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 411, 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 955409, 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 28, 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 411, 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 411, 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 4.786905535, 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.011646972 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 411, 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 458 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 411, 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 1, 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 410, 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 0, 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 10, 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 1, 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 288, 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 61, 2024-03-19T01:28:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 0, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 14, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 0, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 20, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-03-19T01:28:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-03-19T01:28:28.333 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 87, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 2, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 11, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 14, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 3, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 53, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 0, 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 11 2024-03-19T01:28:28.334 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 1, 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 0 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 2, 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 2, 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 2, 2024-03-19T01:28:28.335 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 2, 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 4533, 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 2369530, 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 4533, 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 4533, 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 2369530, 2024-03-19T01:28:28.336 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-03-19T01:28:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T01:28:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T01:28:28.337 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T01:28:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 411, 2024-03-19T01:28:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 955611, 2024-03-19T01:28:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 411, 2024-03-19T01:28:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 955611, 2024-03-19T01:28:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:28.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 411, 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 411, 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 411, 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 411, 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:28.339 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 2, 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 202, 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 2, 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T01:28:28.340 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 1, 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 202, 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x564e32360500": { 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 574, 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 633, 2024-03-19T01:28:28.341 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 920612, 2024-03-19T01:28:28.342 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:28.342 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 633, 2024-03-19T01:28:28.342 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T01:28:28.342 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T01:28:28.342 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 287, 2024-03-19T01:28:28.342 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 920038, 2024-03-19T01:28:28.342 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:28.342 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:28.342 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:28.342 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:28.342 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:28.342 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:28.342 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-19T01:28:29.142 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-19T01:28:29.142 INFO:tasks.check_counter:Daemon mds.c mds.imported=8 2024-03-19T01:28:29.142 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=19 2024-03-19T01:28:29.142 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=372 2024-03-19T01:28:29.142 DEBUG:tasks.check_counter:Getting stats from f 2024-03-19T01:28:29.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-19T01:28:29.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:29 smithi162 ceph-mon[67178]: pgmap v827: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 35 KiB/s wr, 15 op/s 2024-03-19T01:28:29.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:29 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:29.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:29 smithi116 ceph-mon[62701]: pgmap v827: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 35 KiB/s wr, 15 op/s 2024-03-19T01:28:29.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:29 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:29.427 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:28:29.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:29 smithi156 ceph-mon[66799]: pgmap v827: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 35 KiB/s wr, 15 op/s 2024-03-19T01:28:29.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:29 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:30.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:30 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:30.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:30 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:30.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:30 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:31.272 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:31 smithi162 ceph-mon[67178]: pgmap v828: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 36 KiB/s wr, 16 op/s 2024-03-19T01:28:31.273 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-19T01:28:31.273 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T01:28:31.273 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 1403, 2024-03-19T01:28:31.273 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 1369, 2024-03-19T01:28:31.273 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 1083952, 2024-03-19T01:28:31.273 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 522170, 2024-03-19T01:28:31.274 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 5, 2024-03-19T01:28:31.274 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 1, 2024-03-19T01:28:31.274 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.126962386, 2024-03-19T01:28:31.274 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.040043540, 2024-03-19T01:28:31.274 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.042229320, 2024-03-19T01:28:31.275 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.037190468, 2024-03-19T01:28:31.275 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:31.275 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1368, 2024-03-19T01:28:31.275 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.061384625, 2024-03-19T01:28:31.275 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000044871 2024-03-19T01:28:31.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.276 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:31.276 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.276 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.276 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.276 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 725440, 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 182864 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 4, 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 227, 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 876, 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 562138, 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 6, 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 3, 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.011895267, 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.007847078, 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.002342646, 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.000048748, 2024-03-19T01:28:31.277 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:31.278 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 225, 2024-03-19T01:28:31.278 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.016939042, 2024-03-19T01:28:31.278 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000075284 2024-03-19T01:28:31.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.278 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:31.278 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.278 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.278 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.278 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-19T01:28:31.278 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 561440 2024-03-19T01:28:31.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.278 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 2228, 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 2228, 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 3756201, 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 707687, 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 5, 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 3, 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.135311673, 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.037215253, 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.035785954, 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.054595458, 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2227, 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.081076212, 2024-03-19T01:28:31.279 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000036406 2024-03-19T01:28:31.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.280 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:31.280 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.280 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.280 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.280 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-19T01:28:31.280 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T01:28:31.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.280 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-03-19T01:28:31.280 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-03-19T01:28:31.281 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-03-19T01:28:31.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.281 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-03-19T01:28:31.281 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T01:28:31.281 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T01:28:31.281 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 5562, 2024-03-19T01:28:31.281 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.121417654, 2024-03-19T01:28:31.281 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000021829 2024-03-19T01:28:31.282 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:31.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.282 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-03-19T01:28:31.282 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T01:28:31.282 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T01:28:31.282 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.282 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.282 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.283 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:31.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.283 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-03-19T01:28:31.283 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 0, 2024-03-19T01:28:31.283 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 0, 2024-03-19T01:28:31.283 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-03-19T01:28:31.283 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.283 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.284 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.284 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-03-19T01:28:31.284 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 0, 2024-03-19T01:28:31.284 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 0, 2024-03-19T01:28:31.284 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-03-19T01:28:31.284 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 0, 2024-03-19T01:28:31.284 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 0, 2024-03-19T01:28:31.285 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-03-19T01:28:31.285 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 421, 2024-03-19T01:28:31.285 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 0, 2024-03-19T01:28:31.285 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 421, 2024-03-19T01:28:31.285 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 0, 2024-03-19T01:28:31.285 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 421, 2024-03-19T01:28:31.286 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 0, 2024-03-19T01:28:31.286 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 0, 2024-03-19T01:28:31.286 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 0, 2024-03-19T01:28:31.286 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 3, 2024-03-19T01:28:31.286 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 0, 2024-03-19T01:28:31.286 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 0, 2024-03-19T01:28:31.286 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 0, 2024-03-19T01:28:31.286 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 0, 2024-03-19T01:28:31.286 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-03-19T01:28:31.286 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-03-19T01:28:31.286 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T01:28:31.287 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 0, 2024-03-19T01:28:31.288 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 0, 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 0, 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 0 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 0, 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 0, 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 0, 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 0, 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 0, 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 0, 2024-03-19T01:28:31.289 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 0, 2024-03-19T01:28:31.290 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-03-19T01:28:31.290 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-03-19T01:28:31.290 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-03-19T01:28:31.290 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-03-19T01:28:31.290 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-03-19T01:28:31.290 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-03-19T01:28:31.290 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-03-19T01:28:31.290 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 0, 2024-03-19T01:28:31.290 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 0, 2024-03-19T01:28:31.290 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T01:28:31.290 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 0, 2024-03-19T01:28:31.290 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-03-19T01:28:31.290 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 0, 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 0, 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 1817, 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 0, 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-03-19T01:28:31.291 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 3, 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 2, 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 4194304, 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 7382660, 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 7381404, 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 1817 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.292 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 77, 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 291, 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 214, 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 12, 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 33, 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 21, 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 421, 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 641, 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 220, 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 0, 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 0, 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 0, 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 49196, 2024-03-19T01:28:31.293 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 215420 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 0, 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 0, 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 0, 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 0, 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 0, 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-03-19T01:28:31.295 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-03-19T01:28:31.296 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.297 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.298 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.298 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-03-19T01:28:31.298 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.299 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.299 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.299 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-03-19T01:28:31.299 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.299 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.299 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.299 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-03-19T01:28:31.299 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.299 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.299 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.300 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-03-19T01:28:31.300 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.300 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.300 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.300 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-03-19T01:28:31.300 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.300 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.300 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.300 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-03-19T01:28:31.300 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.300 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.301 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.302 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 1, 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 1, 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 0, 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 1, 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-03-19T01:28:31.305 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 0, 2024-03-19T01:28:31.306 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 0, 2024-03-19T01:28:31.306 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 4793, 2024-03-19T01:28:31.306 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-03-19T01:28:31.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.306 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-03-19T01:28:31.306 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-03-19T01:28:31.306 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-03-19T01:28:31.306 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T01:28:31.306 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-03-19T01:28:31.306 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T01:28:31.306 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-03-19T01:28:31.306 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T01:28:31.306 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T01:28:31.307 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 184603, 2024-03-19T01:28:31.308 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 116, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 7720, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 162, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 501890, 2024-03-19T01:28:31.309 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 16028, 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 3293, 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 21, 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 3, 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 3293, 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3293, 2024-03-19T01:28:31.310 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 6.403873705, 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.001944692 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3293, 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 3294 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 3293, 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 3293, 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 0, 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 2011, 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 0, 2024-03-19T01:28:31.311 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 1280, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 0, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 0, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 0, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 0, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 1, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 0, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-03-19T01:28:31.312 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-03-19T01:28:31.313 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 87, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 2, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 4, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 6, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 2, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-03-19T01:28:31.314 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 0, 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 2, 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 0 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 0, 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 0 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T01:28:31.315 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 0, 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 0, 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 0, 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 3631, 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 4494035, 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 3631, 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T01:28:31.316 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 3631, 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 4494035, 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T01:28:31.317 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 3293, 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 3188361, 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 3293, 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 3188361, 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:31.318 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 3293, 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 3293, 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 3293, 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 3293, 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:31.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.320 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x557b476286e0": { 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-03-19T01:28:31.321 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:31.322 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:31.322 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:31.322 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:31.322 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:31.322 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:31.322 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-19T01:28:31.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:31 smithi116 ceph-mon[62701]: pgmap v828: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 36 KiB/s wr, 16 op/s 2024-03-19T01:28:31.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:31 smithi156 ceph-mon[66799]: pgmap v828: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 36 KiB/s wr, 16 op/s 2024-03-19T01:28:32.033 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-19T01:28:32.033 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-19T01:28:32.033 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-19T01:28:32.033 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-19T01:28:32.033 DEBUG:tasks.check_counter:Getting stats from i 2024-03-19T01:28:32.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-19T01:28:32.280 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:28:33.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:33 smithi162 ceph-mon[67178]: pgmap v829: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 39 KiB/s wr, 18 op/s 2024-03-19T01:28:33.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:33 smithi116 ceph-mon[62701]: pgmap v829: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 39 KiB/s wr, 18 op/s 2024-03-19T01:28:33.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:33 smithi156 ceph-mon[66799]: pgmap v829: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 39 KiB/s wr, 18 op/s 2024-03-19T01:28:33.995 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-19T01:28:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T01:28:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 3175, 2024-03-19T01:28:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 3682, 2024-03-19T01:28:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 1096519, 2024-03-19T01:28:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 1933629, 2024-03-19T01:28:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 16, 2024-03-19T01:28:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 3, 2024-03-19T01:28:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.214940305, 2024-03-19T01:28:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.085622776, 2024-03-19T01:28:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.155561740, 2024-03-19T01:28:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.007165022, 2024-03-19T01:28:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3681, 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.232298673, 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000063107 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 6025, 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000934505, 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000155 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 466640, 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 1125056 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 1887, 2024-03-19T01:28:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 2343, 2024-03-19T01:28:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 1317442, 2024-03-19T01:28:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 3187356, 2024-03-19T01:28:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 9, 2024-03-19T01:28:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 6, 2024-03-19T01:28:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.170915867, 2024-03-19T01:28:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.075460758, 2024-03-19T01:28:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.078600963, 2024-03-19T01:28:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.023340518, 2024-03-19T01:28:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2342, 2024-03-19T01:28:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.167934632, 2024-03-19T01:28:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000071705 2024-03-19T01:28:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 2168, 2024-03-19T01:28:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000409495, 2024-03-19T01:28:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000188 2024-03-19T01:28:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 750816, 2024-03-19T01:28:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 187376 2024-03-19T01:28:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T01:28:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 4082, 2024-03-19T01:28:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 4651, 2024-03-19T01:28:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 1983509, 2024-03-19T01:28:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 2301855, 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 9, 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 6, 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.243196398, 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.099067115, 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.169837840, 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.005203711, 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 4649, 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.280531622, 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000060342 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 8124, 2024-03-19T01:28:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.001365474, 2024-03-19T01:28:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000168 2024-03-19T01:28:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-19T01:28:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 473536 2024-03-19T01:28:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-03-19T01:28:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-03-19T01:28:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-03-19T01:28:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-03-19T01:28:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T01:28:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T01:28:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1284, 2024-03-19T01:28:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.385862041, 2024-03-19T01:28:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000300515 2024-03-19T01:28:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-03-19T01:28:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T01:28:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T01:28:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 144, 2024-03-19T01:28:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.008116640, 2024-03-19T01:28:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000056365 2024-03-19T01:28:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-03-19T01:28:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 520, 2024-03-19T01:28:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 344, 2024-03-19T01:28:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-03-19T01:28:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 344, 2024-03-19T01:28:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.345922189, 2024-03-19T01:28:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.001005587 2024-03-19T01:28:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-03-19T01:28:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 176, 2024-03-19T01:28:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 2, 2024-03-19T01:28:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-03-19T01:28:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 11, 2024-03-19T01:28:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 0, 2024-03-19T01:28:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-03-19T01:28:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 496, 2024-03-19T01:28:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 1, 2024-03-19T01:28:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 7, 2024-03-19T01:28:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 488, 2024-03-19T01:28:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 494, 2024-03-19T01:28:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 169, 2024-03-19T01:28:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 470, 2024-03-19T01:28:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 470, 2024-03-19T01:28:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 23, 2024-03-19T01:28:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 863, 2024-03-19T01:28:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 589, 2024-03-19T01:28:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 169, 2024-03-19T01:28:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 20, 2024-03-19T01:28:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-03-19T01:28:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-03-19T01:28:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 2, 2024-03-19T01:28:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 2961, 2024-03-19T01:28:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-03-19T01:28:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 9, 2024-03-19T01:28:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 163, 2024-03-19T01:28:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 18, 2024-03-19T01:28:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 607, 2024-03-19T01:28:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-03-19T01:28:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 0, 2024-03-19T01:28:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-03-19T01:28:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 4, 2024-03-19T01:28:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 234, 2024-03-19T01:28:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 0, 2024-03-19T01:28:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T01:28:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-03-19T01:28:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T01:28:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-03-19T01:28:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-03-19T01:28:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T01:28:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T01:28:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 0, 2024-03-19T01:28:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 11, 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 56, 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 360, 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 458, 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 230, 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 18, 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 147 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 46, 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 129, 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 74, 2024-03-19T01:28:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 119, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 330, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 40, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 40, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-03-19T01:28:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 0, 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 0, 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 9, 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 0, 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 1087, 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-03-19T01:28:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 1087, 2024-03-19T01:28:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-03-19T01:28:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-03-19T01:28:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 3, 2024-03-19T01:28:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-03-19T01:28:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-03-19T01:28:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 3, 2024-03-19T01:28:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 2, 2024-03-19T01:28:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-03-19T01:28:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-03-19T01:28:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 4194304, 2024-03-19T01:28:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 6579304, 2024-03-19T01:28:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 0, 2024-03-19T01:28:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 601, 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 4.008165197, 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.006669160 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 0 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 490, 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 652, 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 162, 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 61, 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 99, 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 38, 2024-03-19T01:28:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 496, 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 712, 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 216, 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 470, 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 845, 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 375, 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 58016, 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 223612 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 587, 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 1090, 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 520, 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 238, 2024-03-19T01:28:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 2181, 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 33, 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.075617232, 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002291431 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 7, 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.020940819, 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002991545 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-03-19T01:28:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-03-19T01:28:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 211, 2024-03-19T01:28:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.091353041, 2024-03-19T01:28:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000432952 2024-03-19T01:28:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-03-19T01:28:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-03-19T01:28:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-03-19T01:28:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-03-19T01:28:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-03-19T01:28:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 22, 2024-03-19T01:28:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.020230439, 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000919565 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 28, 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.032335842, 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.001154851 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 8, 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.022524328, 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002815541 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-03-19T01:28:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-03-19T01:28:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-03-19T01:28:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-03-19T01:28:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-03-19T01:28:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-03-19T01:28:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-03-19T01:28:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 35, 2024-03-19T01:28:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.082920488, 2024-03-19T01:28:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002369156 2024-03-19T01:28:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-03-19T01:28:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-03-19T01:28:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-03-19T01:28:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-03-19T01:28:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 1, 2024-03-19T01:28:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 1, 2024-03-19T01:28:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 0, 2024-03-19T01:28:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 1, 2024-03-19T01:28:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-03-19T01:28:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 97, 2024-03-19T01:28:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 97, 2024-03-19T01:28:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 953, 2024-03-19T01:28:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-03-19T01:28:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-03-19T01:28:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T01:28:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T01:28:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-03-19T01:28:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-03-19T01:28:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-03-19T01:28:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-03-19T01:28:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T01:28:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T01:28:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T01:28:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T01:28:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T01:28:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-03-19T01:28:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-03-19T01:28:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 325791, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 609, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 1320, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 15, 2024-03-19T01:28:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 7720, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 162, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 1741177, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 16487, 2024-03-19T01:28:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-03-19T01:28:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-03-19T01:28:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-03-19T01:28:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-03-19T01:28:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-03-19T01:28:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-03-19T01:28:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-03-19T01:28:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 894, 2024-03-19T01:28:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 2441271, 2024-03-19T01:28:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 33, 2024-03-19T01:28:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 894, 2024-03-19T01:28:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-03-19T01:28:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 894, 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 12.079761834, 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.013512037 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 894, 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 1027 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 894, 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 22, 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 872, 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-03-19T01:28:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 0, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 10, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 15, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 616, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 77, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 0, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 52, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 0, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 20, 2024-03-19T01:28:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-03-19T01:28:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 87, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-03-19T01:28:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 2, 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 12, 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 12, 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 0, 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 185, 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 9, 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 43 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 1, 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 0 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 6, 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 6, 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 40, 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 40, 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 9140, 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 3489558, 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 9140, 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 9140, 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 3489558, 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 894, 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 2445151, 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 894, 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 2445151, 2024-03-19T01:28:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 894, 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 894, 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 894, 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 894, 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 40, 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 4040, 2024-03-19T01:28:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 40, 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 15, 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 4040, 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x555b679a8500": { 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T01:28:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 1087, 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 2385000, 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 1087, 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 601, 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 2385000, 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:34.043 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-19T01:28:34.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:34 smithi162 ceph-mon[67178]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:34.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:34 smithi116 ceph-mon[62701]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:34.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:34 smithi156 ceph-mon[66799]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-19T01:28:34.839 INFO:tasks.check_counter:Daemon mds.i mds.exported=9 2024-03-19T01:28:34.839 INFO:tasks.check_counter:Daemon mds.i mds.imported=18 2024-03-19T01:28:34.839 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=46 2024-03-19T01:28:34.839 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=129 2024-03-19T01:28:34.839 DEBUG:tasks.check_counter:Getting stats from l 2024-03-19T01:28:34.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810 shell --fsid 35912c14-e58c-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-19T01:28:35.092 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/mon.c/config 2024-03-19T01:28:35.349 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:35 smithi162 ceph-mon[67178]: pgmap v830: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 37 KiB/s wr, 16 op/s 2024-03-19T01:28:35.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:35 smithi116 ceph-mon[62701]: pgmap v830: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 37 KiB/s wr, 16 op/s 2024-03-19T01:28:35.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:35 smithi156 ceph-mon[66799]: pgmap v830: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 37 KiB/s wr, 16 op/s 2024-03-19T01:28:36.849 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-03-19T01:28:36.849 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-0": { 2024-03-19T01:28:36.849 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 1124, 2024-03-19T01:28:36.849 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 1124, 2024-03-19T01:28:36.849 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 376161, 2024-03-19T01:28:36.849 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 356609, 2024-03-19T01:28:36.849 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 6, 2024-03-19T01:28:36.849 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 2, 2024-03-19T01:28:36.850 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.095774882, 2024-03-19T01:28:36.850 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.028797026, 2024-03-19T01:28:36.850 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.025759851, 2024-03-19T01:28:36.850 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.032326606, 2024-03-19T01:28:36.850 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:36.850 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1123, 2024-03-19T01:28:36.850 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.064996525, 2024-03-19T01:28:36.850 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000057877 2024-03-19T01:28:36.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.850 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:36.850 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.851 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.851 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.851 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-19T01:28:36.851 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 448 2024-03-19T01:28:36.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.851 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-1": { 2024-03-19T01:28:36.851 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 1293, 2024-03-19T01:28:36.851 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 1258, 2024-03-19T01:28:36.851 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 968182, 2024-03-19T01:28:36.851 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 488550, 2024-03-19T01:28:36.851 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 5, 2024-03-19T01:28:36.851 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 3, 2024-03-19T01:28:36.852 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.111111988, 2024-03-19T01:28:36.852 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.036540777, 2024-03-19T01:28:36.852 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.034047957, 2024-03-19T01:28:36.852 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.034733301, 2024-03-19T01:28:36.852 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:36.852 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1257, 2024-03-19T01:28:36.852 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.039249333, 2024-03-19T01:28:36.852 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000031224 2024-03-19T01:28:36.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.852 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:36.852 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.852 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.852 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.853 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 733584, 2024-03-19T01:28:36.853 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 186064 2024-03-19T01:28:36.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.853 INFO:teuthology.orchestra.run.smithi162.stdout: "AsyncMessenger::Worker-2": { 2024-03-19T01:28:36.853 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_messages": 1319, 2024-03-19T01:28:36.853 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages": 1545, 2024-03-19T01:28:36.853 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_bytes": 2725346, 2024-03-19T01:28:36.853 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_bytes": 895014, 2024-03-19T01:28:36.853 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_created_connections": 4, 2024-03-19T01:28:36.853 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_active_connections": 3, 2024-03-19T01:28:36.853 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_total_time": 0.096957789, 2024-03-19T01:28:36.853 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_send_time": 0.032165543, 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_recv_time": 0.026470948, 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_running_fast_dispatch_time": 0.029093887, 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_messages_queue_lat": { 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 1543, 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.065192333, 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000042250 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_handle_ack_lat": { 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-19T01:28:36.854 INFO:teuthology.orchestra.run.smithi162.stdout: "msgr_send_encrypted_bytes": 476208 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: "cct": { 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: "total_workers": 1, 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: "unhealthy_workers": 0 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-MDSRank": { 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 5770, 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.136071408, 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000023582 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.855 INFO:teuthology.orchestra.run.smithi162.stdout: "finisher-PurgeQueue": { 2024-03-19T01:28:36.856 INFO:teuthology.orchestra.run.smithi162.stdout: "queue_len": 0, 2024-03-19T01:28:36.856 INFO:teuthology.orchestra.run.smithi162.stdout: "complete_latency": { 2024-03-19T01:28:36.856 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.856 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.856 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.856 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:36.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.856 INFO:teuthology.orchestra.run.smithi162.stdout: "mds": { 2024-03-19T01:28:36.856 INFO:teuthology.orchestra.run.smithi162.stdout: "request": 0, 2024-03-19T01:28:36.856 INFO:teuthology.orchestra.run.smithi162.stdout: "reply": 0, 2024-03-19T01:28:36.856 INFO:teuthology.orchestra.run.smithi162.stdout: "reply_latency": { 2024-03-19T01:28:36.856 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.856 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.857 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.857 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_reply": 0, 2024-03-19T01:28:36.857 INFO:teuthology.orchestra.run.smithi162.stdout: "forward": 0, 2024-03-19T01:28:36.857 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_complete": 0, 2024-03-19T01:28:36.857 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_fetch_keys": 0, 2024-03-19T01:28:36.857 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_commit": 0, 2024-03-19T01:28:36.857 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_split": 0, 2024-03-19T01:28:36.857 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_merge": 0, 2024-03-19T01:28:36.857 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes": 522, 2024-03-19T01:28:36.857 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_top": 142, 2024-03-19T01:28:36.857 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_bottom": 380, 2024-03-19T01:28:36.857 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pin_tail": 0, 2024-03-19T01:28:36.858 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_pinned": 319, 2024-03-19T01:28:36.858 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_expired": 0, 2024-03-19T01:28:36.858 INFO:teuthology.orchestra.run.smithi162.stdout: "inodes_with_caps": 0, 2024-03-19T01:28:36.858 INFO:teuthology.orchestra.run.smithi162.stdout: "caps": 0, 2024-03-19T01:28:36.858 INFO:teuthology.orchestra.run.smithi162.stdout: "subtrees": 22, 2024-03-19T01:28:36.858 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse": 0, 2024-03-19T01:28:36.858 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_hit": 0, 2024-03-19T01:28:36.858 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_forward": 0, 2024-03-19T01:28:36.858 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_discover": 0, 2024-03-19T01:28:36.858 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_dir_fetch": 0, 2024-03-19T01:28:36.858 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_remote_ino": 0, 2024-03-19T01:28:36.858 INFO:teuthology.orchestra.run.smithi162.stdout: "traverse_lock": 0, 2024-03-19T01:28:36.858 INFO:teuthology.orchestra.run.smithi162.stdout: "load_cent": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "q": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "exported": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "exported_inodes": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "imported": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "imported_inodes": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_dir_fetch": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_backtrace_fetch": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "openino_peer_discover": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rfiles": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rbytes": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "root_rsnaps": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_fetch": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_set_tag": 0, 2024-03-19T01:28:36.859 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_backtrace_repaired": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_inotable_repaired": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_inodes": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dir_base_inodes": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_dirfrag_rstats": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "scrub_file_inodes": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_inode_file_caps": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_revoke": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_grant": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_trunc": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_caps_dirty": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_cap_release": 0, 2024-03-19T01:28:36.860 INFO:teuthology.orchestra.run.smithi162.stdout: "process_request_cap_release": 0 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_cache": { 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update": 0, 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_update_receipt": 0, 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_try_discover": 0, 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_send_discover": 0, 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "dir_handle_discover": 0, 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays": 0, 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_delayed": 0, 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "num_strays_enqueuing": 0, 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_created": 0, 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_enqueued": 0, 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_reintegrated": 0, 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "strays_migrated": 0, 2024-03-19T01:28:36.861 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_processing": 0, 2024-03-19T01:28:36.862 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_enqueued": 0, 2024-03-19T01:28:36.862 INFO:teuthology.orchestra.run.smithi162.stdout: "num_recovering_prioritized": 0, 2024-03-19T01:28:36.862 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_started": 0, 2024-03-19T01:28:36.862 INFO:teuthology.orchestra.run.smithi162.stdout: "recovery_completed": 0, 2024-03-19T01:28:36.862 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_path": 0, 2024-03-19T01:28:36.862 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_quiesce_inode": 0, 2024-03-19T01:28:36.862 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_enqueue_scrub": 0, 2024-03-19T01:28:36.862 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_exportdir": 0, 2024-03-19T01:28:36.862 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_flush": 0, 2024-03-19T01:28:36.862 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragmentdir": 0, 2024-03-19T01:28:36.862 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_fragstats": 0, 2024-03-19T01:28:36.862 INFO:teuthology.orchestra.run.smithi162.stdout: "ireq_inodestats": 0 2024-03-19T01:28:36.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_log": { 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "evlrg": 0, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "evadd": 0, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "evex": 0, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "evtrm": 0, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "ev": 1087, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "evexg": 0, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "evexd": 0, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "segadd": 0, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "segex": 0, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "segtrm": 0, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "seg": 3, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "segmjr": 2, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "segexg": 0, 2024-03-19T01:28:36.863 INFO:teuthology.orchestra.run.smithi162.stdout: "segexd": 0, 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: "expos": 4194304, 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: "wrpos": 6579304, 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: "rdpos": 6578173, 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: "jlat": { 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: "replayed": 1087 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_mem": { 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: "ino": 470, 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: "ino+": 629, 2024-03-19T01:28:36.864 INFO:teuthology.orchestra.run.smithi162.stdout: "ino-": 159, 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: "dir": 53, 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: "dir+": 57, 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: "dir-": 4, 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: "dn": 522, 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: "dn+": 686, 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: "dn-": 164, 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: "cap": 0, 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: "cap+": 0, 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: "cap-": 0, 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: "rss": 50816, 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: "heap": 215420 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_server": { 2024-03-19T01:28:36.865 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_client_request": 0, 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: "dispatch_server_request": 0, 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_request": 0, 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_client_session": 0, 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: "handle_peer_request": 0, 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: "req_create_latency": { 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getattr_latency": { 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.867 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getfilelock_latency": { 2024-03-19T01:28:36.867 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.867 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.867 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.867 INFO:teuthology.orchestra.run.smithi162.stdout: "req_link_latency": { 2024-03-19T01:28:36.867 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.867 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.867 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.867 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookup_latency": { 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookuphash_latency": { 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupino_latency": { 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.868 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupname_latency": { 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupparent_latency": { 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lookupsnap_latency": { 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.869 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.870 INFO:teuthology.orchestra.run.smithi162.stdout: "req_lssnap_latency": { 2024-03-19T01:28:36.870 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.870 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.870 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.870 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.870 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mkdir_latency": { 2024-03-19T01:28:36.870 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.871 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.871 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.871 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mknod_latency": { 2024-03-19T01:28:36.871 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.871 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.871 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.871 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.871 INFO:teuthology.orchestra.run.smithi162.stdout: "req_mksnap_latency": { 2024-03-19T01:28:36.871 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.871 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.871 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.872 INFO:teuthology.orchestra.run.smithi162.stdout: "req_open_latency": { 2024-03-19T01:28:36.872 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.872 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.872 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.872 INFO:teuthology.orchestra.run.smithi162.stdout: "req_readdir_latency": { 2024-03-19T01:28:36.872 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.872 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.872 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.872 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.872 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rename_latency": { 2024-03-19T01:28:36.872 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.873 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.873 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.873 INFO:teuthology.orchestra.run.smithi162.stdout: "req_renamesnap_latency": { 2024-03-19T01:28:36.873 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.873 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.873 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.873 INFO:teuthology.orchestra.run.smithi162.stdout: "req_snapdiff_latency": { 2024-03-19T01:28:36.873 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.873 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.873 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.873 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmdir_latency": { 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmsnap_latency": { 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: "req_rmxattr_latency": { 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.874 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setattr_latency": { 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setdirlayout_latency": { 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setfilelock_latency": { 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.875 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setlayout_latency": { 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: "req_setxattr_latency": { 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: "req_symlink_latency": { 2024-03-19T01:28:36.876 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: "req_unlink_latency": { 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_revoke_eviction": 0, 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: "cap_acquisition_throttle": 0, 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: "req_getvxattr_latency": { 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.877 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.878 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:36.878 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.878 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_sessions": { 2024-03-19T01:28:36.878 INFO:teuthology.orchestra.run.smithi162.stdout: "session_count": 1, 2024-03-19T01:28:36.878 INFO:teuthology.orchestra.run.smithi162.stdout: "session_add": 1, 2024-03-19T01:28:36.878 INFO:teuthology.orchestra.run.smithi162.stdout: "session_remove": 0, 2024-03-19T01:28:36.878 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_open": 1, 2024-03-19T01:28:36.878 INFO:teuthology.orchestra.run.smithi162.stdout: "sessions_stale": 0, 2024-03-19T01:28:36.878 INFO:teuthology.orchestra.run.smithi162.stdout: "total_load": 0, 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: "average_load": 0, 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: "avg_session_uptime": 4798, 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: "mdthresh_evicted": 0 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: "mempool": { 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_bytes": 0, 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: "bloom_filter_items": 0, 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_bytes": 0, 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_alloc_items": 0, 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_bytes": 0, 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_data_items": 0, 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_onode_items": 0, 2024-03-19T01:28:36.879 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_meta_items": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_bytes": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_other_items": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_cache_buffer_items": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_bytes": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_extent_items": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_bytes": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_blob_items": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_shared_blob_items": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_inline_bl_items": 0, 2024-03-19T01:28:36.880 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_bytes": 0, 2024-03-19T01:28:36.881 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_fsck_items": 0, 2024-03-19T01:28:36.881 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_bytes": 0, 2024-03-19T01:28:36.881 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_txc_items": 0, 2024-03-19T01:28:36.881 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-19T01:28:36.881 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_deferred_items": 0, 2024-03-19T01:28:36.881 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_bytes": 0, 2024-03-19T01:28:36.881 INFO:teuthology.orchestra.run.smithi162.stdout: "bluestore_writing_items": 0, 2024-03-19T01:28:36.881 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_bytes": 0, 2024-03-19T01:28:36.881 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_items": 0, 2024-03-19T01:28:36.881 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_bytes": 0, 2024-03-19T01:28:36.882 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_reader_items": 0, 2024-03-19T01:28:36.882 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_bytes": 0, 2024-03-19T01:28:36.882 INFO:teuthology.orchestra.run.smithi162.stdout: "bluefs_file_writer_items": 0, 2024-03-19T01:28:36.882 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_bytes": 207241, 2024-03-19T01:28:36.882 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_anon_items": 514, 2024-03-19T01:28:36.882 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_bytes": 0, 2024-03-19T01:28:36.882 INFO:teuthology.orchestra.run.smithi162.stdout: "buffer_meta_items": 0, 2024-03-19T01:28:36.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_bytes": 0, 2024-03-19T01:28:36.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_items": 0, 2024-03-19T01:28:36.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_bytes": 0, 2024-03-19T01:28:36.882 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_mapbl_items": 0, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_bytes": 0, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_pglog_items": 0, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_bytes": 7720, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_items": 162, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_bytes": 0, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "osdmap_mapping_items": 0, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_bytes": 0, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "pgmap_items": 0, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_bytes": 1498680, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "mds_co_items": 12065, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_bytes": 0, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_1_items": 0, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_bytes": 0, 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: "unittest_2_items": 0 2024-03-19T01:28:36.883 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.884 INFO:teuthology.orchestra.run.smithi162.stdout: "objecter": { 2024-03-19T01:28:36.884 INFO:teuthology.orchestra.run.smithi162.stdout: "op_active": 0, 2024-03-19T01:28:36.884 INFO:teuthology.orchestra.run.smithi162.stdout: "op_laggy": 0, 2024-03-19T01:28:36.884 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send": 3388, 2024-03-19T01:28:36.884 INFO:teuthology.orchestra.run.smithi162.stdout: "op_send_bytes": 21, 2024-03-19T01:28:36.884 INFO:teuthology.orchestra.run.smithi162.stdout: "op_resend": 3, 2024-03-19T01:28:36.884 INFO:teuthology.orchestra.run.smithi162.stdout: "op_reply": 3388, 2024-03-19T01:28:36.884 INFO:teuthology.orchestra.run.smithi162.stdout: "op_latency": { 2024-03-19T01:28:36.884 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3388, 2024-03-19T01:28:36.884 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 9.337815864, 2024-03-19T01:28:36.884 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.002756143 2024-03-19T01:28:36.884 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.884 INFO:teuthology.orchestra.run.smithi162.stdout: "op_inflight": 0, 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: "oplen_avg": { 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 3388, 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 3389 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: "op": 3388, 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: "op_r": 3388, 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: "op_w": 0, 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: "op_rmw": 0, 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: "op_pg": 0, 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_stat": 2017, 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_create": 0, 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_read": 1369, 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_write": 0, 2024-03-19T01:28:36.885 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writefull": 0, 2024-03-19T01:28:36.886 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_writesame": 0, 2024-03-19T01:28:36.886 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_append": 0, 2024-03-19T01:28:36.886 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_zero": 0, 2024-03-19T01:28:36.886 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_truncate": 0, 2024-03-19T01:28:36.886 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_delete": 0, 2024-03-19T01:28:36.886 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_mapext": 0, 2024-03-19T01:28:36.886 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_sparse_read": 0, 2024-03-19T01:28:36.886 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_clonerange": 0, 2024-03-19T01:28:36.886 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_getxattr": 1, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_setxattr": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_cmpxattr": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_rmxattr": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_resetxattrs": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_call": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_watch": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_notify": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_src_cmpxattr": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_pgls_filter": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "osdop_other": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_active": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_send": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_resend": 0, 2024-03-19T01:28:36.887 INFO:teuthology.orchestra.run.smithi162.stdout: "linger_ping": 0, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_active": 0, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_send": 0, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "poolop_resend": 0, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_active": 0, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_send": 0, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "poolstat_resend": 0, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_active": 0, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_send": 0, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "statfs_resend": 0, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "command_active": 0, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "command_send": 0, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "command_resend": 0, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "map_epoch": 87, 2024-03-19T01:28:36.888 INFO:teuthology.orchestra.run.smithi162.stdout: "map_full": 0, 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: "map_inc": 2, 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_sessions": 5, 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_open": 6, 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_session_close": 1, 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: "osd_laggy": 0, 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_wr": 0, 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_rd": 2, 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_del": 0 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: "oft": { 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_objs": 0, 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_kv_pairs": 0, 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_updates": 0, 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: "omap_total_removes": 0 2024-03-19T01:28:36.889 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.890 INFO:teuthology.orchestra.run.smithi162.stdout: "osd-slow-ops": { 2024-03-19T01:28:36.890 INFO:teuthology.orchestra.run.smithi162.stdout: "slow_ops_count": 0 2024-03-19T01:28:36.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.890 INFO:teuthology.orchestra.run.smithi162.stdout: "purge_queue": { 2024-03-19T01:28:36.890 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops": 0, 2024-03-19T01:28:36.890 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_ops_high_water": 0, 2024-03-19T01:28:36.890 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing": 0, 2024-03-19T01:28:36.890 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executing_high_water": 0, 2024-03-19T01:28:36.890 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed_ops": 0, 2024-03-19T01:28:36.890 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_executed": 0, 2024-03-19T01:28:36.890 INFO:teuthology.orchestra.run.smithi162.stdout: "pq_item_in_journal": 0 2024-03-19T01:28:36.890 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.890 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-19T01:28:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T01:28:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 3732, 2024-03-19T01:28:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 3713335, 2024-03-19T01:28:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 3732, 2024-03-19T01:28:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T01:28:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T01:28:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 3732, 2024-03-19T01:28:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 3713335, 2024-03-19T01:28:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:36.891 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_bytes": { 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 104857600, 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 3388, 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 2385005, 2024-03-19T01:28:36.892 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 3388, 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 2385005, 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-objecter_ops": { 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 1024, 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:36.893 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 3388, 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 3388, 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 3388, 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 3388, 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle": { 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T01:28:36.894 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-03-19T01:28:36.895 INFO:teuthology.orchestra.run.smithi162.stdout: "throttle-write_buf_throttle-0x5609635de6e0": { 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "val": 0, 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "max": 3758096384, 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "get_started": 0, 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "get": 0, 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "get_sum": 0, 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_fail": 0, 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "get_or_fail_success": 0, 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "take": 0, 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "take_sum": 0, 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "put": 0, 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "put_sum": 0, 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "wait": { 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "avgcount": 0, 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "sum": 0.000000000, 2024-03-19T01:28:36.896 INFO:teuthology.orchestra.run.smithi162.stdout: "avgtime": 0.000000000 2024-03-19T01:28:36.897 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:36.897 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-03-19T01:28:36.897 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-03-19T01:28:37.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:37 smithi162 ceph-mon[67178]: pgmap v831: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 34 KiB/s wr, 19 op/s 2024-03-19T01:28:37.404 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:37 smithi116 ceph-mon[62701]: pgmap v831: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 34 KiB/s wr, 19 op/s 2024-03-19T01:28:37.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:37 smithi156 ceph-mon[66799]: pgmap v831: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 34 KiB/s wr, 19 op/s 2024-03-19T01:28:37.698 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-19T01:28:37.698 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-19T01:28:37.698 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update=0 2024-03-19T01:28:37.698 INFO:tasks.check_counter:Daemon mds.l mds_cache.dir_update_receipt=0 2024-03-19T01:28:37.698 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-19T01:28:37.710 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-19T01:28:37.710 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:28:37.710 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T01:28:37.727 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-03-19T01:28:37.727 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-19T01:28:37.784 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:28:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.927+0000 7f7e3c509640 1 -- 172.21.15.116:0/1384576340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e34100750 msgr2=0x7f7e34100b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.927+0000 7f7e3c509640 1 --2- 172.21.15.116:0/1384576340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e34100750 0x7f7e34100b70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7e30009900 tx=0x7f7e3002bc30 comp rx=0 tx=0).stop 2024-03-19T01:28:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.927+0000 7f7e3c509640 1 -- 172.21.15.116:0/1384576340 shutdown_connections 2024-03-19T01:28:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.927+0000 7f7e3c509640 1 --2- 172.21.15.116:0/1384576340 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e34102300 0x7f7e34108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.927+0000 7f7e3c509640 1 --2- 172.21.15.116:0/1384576340 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e34101940 0x7f7e34101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:37.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.927+0000 7f7e3c509640 1 --2- 172.21.15.116:0/1384576340 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e34100750 0x7f7e34100b70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.927+0000 7f7e3c509640 1 -- 172.21.15.116:0/1384576340 >> 172.21.15.116:0/1384576340 conn(0x7f7e340fbd00 msgr2=0x7f7e340fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.927+0000 7f7e3c509640 1 -- 172.21.15.116:0/1384576340 shutdown_connections 2024-03-19T01:28:37.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.927+0000 7f7e3c509640 1 -- 172.21.15.116:0/1384576340 wait complete. 2024-03-19T01:28:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.928+0000 7f7e3c509640 1 Processor -- start 2024-03-19T01:28:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.928+0000 7f7e3c509640 1 -- start start 2024-03-19T01:28:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.928+0000 7f7e3c509640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e34100750 0x7f7e34197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.928+0000 7f7e3c509640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e34101940 0x7f7e341981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.928+0000 7f7e3c509640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e34102300 0x7f7e3419f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.928+0000 7f7e3c509640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3410c3d0 con 0x7f7e34100750 2024-03-19T01:28:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.928+0000 7f7e3c509640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3410c250 con 0x7f7e34101940 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.928+0000 7f7e3c509640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3410c550 con 0x7f7e34102300 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.928+0000 7f7e3a2a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e34100750 0x7f7e34197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e3a2a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e34100750 0x7f7e34197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48830/0 (socket says 172.21.15.116:48830) 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e3a2a5640 1 -- 172.21.15.116:0/1431191959 learned_addr learned my addr 172.21.15.116:0/1431191959 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e39aa4640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e34101940 0x7f7e341981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e3aaa6640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e34102300 0x7f7e3419f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e3a2a5640 1 -- 172.21.15.116:0/1431191959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e34102300 msgr2=0x7f7e3419f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e3a2a5640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e34102300 0x7f7e3419f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e3a2a5640 1 -- 172.21.15.116:0/1431191959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e34101940 msgr2=0x7f7e341981e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e3a2a5640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e34101940 0x7f7e341981e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e3a2a5640 1 -- 172.21.15.116:0/1431191959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e3419f840 con 0x7f7e34100750 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e39aa4640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e34101940 0x7f7e341981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e3a2a5640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e34100750 0x7f7e34197ca0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7e30000c00 tx=0x7f7e3002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:37.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e2b7fe640 1 -- 172.21.15.116:0/1431191959 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e30041070 con 0x7f7e34100750 2024-03-19T01:28:37.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e3c509640 1 -- 172.21.15.116:0/1431191959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e3419fad0 con 0x7f7e34100750 2024-03-19T01:28:37.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e2b7fe640 1 -- 172.21.15.116:0/1431191959 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e30004450 con 0x7f7e34100750 2024-03-19T01:28:37.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e2b7fe640 1 -- 172.21.15.116:0/1431191959 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e30034440 con 0x7f7e34100750 2024-03-19T01:28:37.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.929+0000 7f7e3c509640 1 -- 172.21.15.116:0/1431191959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e341a0010 con 0x7f7e34100750 2024-03-19T01:28:37.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.930+0000 7f7e3c509640 1 -- 172.21.15.116:0/1431191959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e341a01e0 con 0x7f7e34100750 2024-03-19T01:28:37.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.931+0000 7f7e2b7fe640 1 -- 172.21.15.116:0/1431191959 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f7e30044070 con 0x7f7e34100750 2024-03-19T01:28:37.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.932+0000 7f7e2b7fe640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7e200a2800 0x7f7e200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:37.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.932+0000 7f7e39aa4640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7e200a2800 0x7f7e200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:37.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.932+0000 7f7e2b7fe640 1 -- 172.21.15.116:0/1431191959 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f7e300f0840 con 0x7f7e34100750 2024-03-19T01:28:37.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.932+0000 7f7e39aa4640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7e200a2800 0x7f7e200a4cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009fc0 tx=0x7f7e24005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:37.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:37.940+0000 7f7e2b7fe640 1 -- 172.21.15.116:0/1431191959 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7e300bd270 con 0x7f7e34100750 2024-03-19T01:28:38.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.142+0000 7f7e3c509640 1 -- 172.21.15.116:0/1431191959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e341a03b0 con 0x7f7e34100750 2024-03-19T01:28:38.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.144+0000 7f7e2b7fe640 1 -- 172.21.15.116:0/1431191959 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22804 (secure 0 0 0) 0x7f7e300c1310 con 0x7f7e34100750 2024-03-19T01:28:38.144 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:28:38.145 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:11:28.144841+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:28:38.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.145+0000 7f7e3c509640 1 -- 172.21.15.116:0/1431191959 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7e200a2800 msgr2=0x7f7e200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.145+0000 7f7e3c509640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7e200a2800 0x7f7e200a4cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7e24009fc0 tx=0x7f7e24005f90 comp rx=0 tx=0).stop 2024-03-19T01:28:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.145+0000 7f7e3c509640 1 -- 172.21.15.116:0/1431191959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e34100750 msgr2=0x7f7e34197ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.146+0000 7f7e3c509640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e34100750 0x7f7e34197ca0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f7e30000c00 tx=0x7f7e3002c640 comp rx=0 tx=0).stop 2024-03-19T01:28:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.146+0000 7f7e3c509640 1 -- 172.21.15.116:0/1431191959 shutdown_connections 2024-03-19T01:28:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.146+0000 7f7e3c509640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f7e34102300 0x7f7e3419f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.146+0000 7f7e3c509640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f7e200a2800 0x7f7e200a4cc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.146+0000 7f7e3c509640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e34101940 0x7f7e341981e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.146+0000 7f7e3c509640 1 --2- 172.21.15.116:0/1431191959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e34100750 0x7f7e34197ca0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.146+0000 7f7e3c509640 1 -- 172.21.15.116:0/1431191959 >> 172.21.15.116:0/1431191959 conn(0x7f7e340fbd00 msgr2=0x7f7e340fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.146+0000 7f7e3c509640 1 -- 172.21.15.116:0/1431191959 shutdown_connections 2024-03-19T01:28:38.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.146+0000 7f7e3c509640 1 -- 172.21.15.116:0/1431191959 wait complete. 2024-03-19T01:28:38.157 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-19T01:05:53.301933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-19T01:10:31.028304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-19T01:10:31.704469+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-19T01:28:38.157 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-19T01:28:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.262+0000 7f20e595b640 1 -- 172.21.15.116:0/1303510117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20e0102a40 msgr2=0x7f20e0102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.262+0000 7f20e595b640 1 --2- 172.21.15.116:0/1303510117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20e0102a40 0x7f20e0102ec0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009900 tx=0x7f20c802bc30 comp rx=0 tx=0).stop 2024-03-19T01:28:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.262+0000 7f20e595b640 1 -- 172.21.15.116:0/1303510117 shutdown_connections 2024-03-19T01:28:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.262+0000 7f20e595b640 1 --2- 172.21.15.116:0/1303510117 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20e0103400 0x7f20e0109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.262+0000 7f20e595b640 1 --2- 172.21.15.116:0/1303510117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20e0102a40 0x7f20e0102ec0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.262+0000 7f20e595b640 1 --2- 172.21.15.116:0/1303510117 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20e0101850 0x7f20e0101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.262+0000 7f20e595b640 1 -- 172.21.15.116:0/1303510117 >> 172.21.15.116:0/1303510117 conn(0x7f20e00fcdc0 msgr2=0x7f20e00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.262+0000 7f20e595b640 1 -- 172.21.15.116:0/1303510117 shutdown_connections 2024-03-19T01:28:38.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.262+0000 7f20e595b640 1 -- 172.21.15.116:0/1303510117 wait complete. 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20e595b640 1 Processor -- start 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20e595b640 1 -- start start 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20e595b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20e0101850 0x7f20e0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20e595b640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20e0102a40 0x7f20e0198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20e595b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20e0103400 0x7f20e019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20e595b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f20e010d4d0 con 0x7f20e0103400 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20e595b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f20e010d350 con 0x7f20e0101850 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20e595b640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f20e010d650 con 0x7f20e0102a40 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20e0103400 0x7f20e019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20deffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20e0101850 0x7f20e0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20e0103400 0x7f20e019f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48850/0 (socket says 172.21.15.116:48850) 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20deffd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20e0101850 0x7f20e0197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:48584/0 (socket says 172.21.15.116:48584) 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20df7fe640 1 -- 172.21.15.116:0/396565375 learned_addr learned my addr 172.21.15.116:0/396565375 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:38.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20d7fff640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20e0102a40 0x7f20e0198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20df7fe640 1 -- 172.21.15.116:0/396565375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20e0102a40 msgr2=0x7f20e0198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20df7fe640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20e0102a40 0x7f20e0198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20df7fe640 1 -- 172.21.15.116:0/396565375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20e0101850 msgr2=0x7f20e0197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20df7fe640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20e0101850 0x7f20e0197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20df7fe640 1 -- 172.21.15.116:0/396565375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e019f760 con 0x7f20e0103400 2024-03-19T01:28:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.263+0000 7f20deffd640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20e0101850 0x7f20e0197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:28:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.264+0000 7f20df7fe640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20e0103400 0x7f20e019f0f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f20d800eb10 tx=0x7f20d800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.264+0000 7f20e4959640 1 -- 172.21.15.116:0/396565375 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d8004b20 con 0x7f20e0103400 2024-03-19T01:28:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.264+0000 7f20e4959640 1 -- 172.21.15.116:0/396565375 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20d8004cc0 con 0x7f20e0103400 2024-03-19T01:28:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.264+0000 7f20e4959640 1 -- 172.21.15.116:0/396565375 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d8010760 con 0x7f20e0103400 2024-03-19T01:28:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.264+0000 7f20e595b640 1 -- 172.21.15.116:0/396565375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20e019fa50 con 0x7f20e0103400 2024-03-19T01:28:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.264+0000 7f20e595b640 1 -- 172.21.15.116:0/396565375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20e019ff70 con 0x7f20e0103400 2024-03-19T01:28:38.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.265+0000 7f20e595b640 1 -- 172.21.15.116:0/396565375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e0108b70 con 0x7f20e0103400 2024-03-19T01:28:38.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.266+0000 7f20e4959640 1 -- 172.21.15.116:0/396565375 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f20d8005000 con 0x7f20e0103400 2024-03-19T01:28:38.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.266+0000 7f20e4959640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20cc0a2800 0x7f20cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:38.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.267+0000 7f20e4959640 1 -- 172.21.15.116:0/396565375 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f20d80cf4d0 con 0x7f20e0103400 2024-03-19T01:28:38.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.267+0000 7f20deffd640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20cc0a2800 0x7f20cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:38.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.267+0000 7f20deffd640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20cc0a2800 0x7f20cc0a4cc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f20e01028a0 tx=0x7f20d0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:38.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.270+0000 7f20e4959640 1 -- 172.21.15.116:0/396565375 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f20d809c550 con 0x7f20e0103400 2024-03-19T01:28:38.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.450+0000 7f20e595b640 1 -- 172.21.15.116:0/396565375 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f20e002d240 con 0x7f20e0103400 2024-03-19T01:28:38.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.451+0000 7f20e4959640 1 -- 172.21.15.116:0/396565375 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) ==== 93+0+10 (secure 0 0 0) 0x7f20d809be50 con 0x7f20e0103400 2024-03-19T01:28:38.451 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-03-19T01:28:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.453+0000 7f20e595b640 1 -- 172.21.15.116:0/396565375 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20cc0a2800 msgr2=0x7f20cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.453+0000 7f20e595b640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20cc0a2800 0x7f20cc0a4cc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f20e01028a0 tx=0x7f20d0005f90 comp rx=0 tx=0).stop 2024-03-19T01:28:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.453+0000 7f20e595b640 1 -- 172.21.15.116:0/396565375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20e0103400 msgr2=0x7f20e019f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.453+0000 7f20e595b640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20e0103400 0x7f20e019f0f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f20d800eb10 tx=0x7f20d800c2d0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.453+0000 7f20e595b640 1 -- 172.21.15.116:0/396565375 shutdown_connections 2024-03-19T01:28:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.453+0000 7f20e595b640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20e0103400 0x7f20e019f0f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.453+0000 7f20e595b640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f20e0102a40 0x7f20e0198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.453+0000 7f20e595b640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f20cc0a2800 0x7f20cc0a4cc0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.453+0000 7f20e595b640 1 --2- 172.21.15.116:0/396565375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20e0101850 0x7f20e0197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.453+0000 7f20e595b640 1 -- 172.21.15.116:0/396565375 >> 172.21.15.116:0/396565375 conn(0x7f20e00fcdc0 msgr2=0x7f20e0105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:38.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.453+0000 7f20e595b640 1 -- 172.21.15.116:0/396565375 shutdown_connections 2024-03-19T01:28:38.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.453+0000 7f20e595b640 1 -- 172.21.15.116:0/396565375 wait complete. 2024-03-19T01:28:38.465 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-19T01:28:38.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.571+0000 7f6ce6073640 1 -- 172.21.15.116:0/3458026254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ce0101910 msgr2=0x7f6ce0101d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.571+0000 7f6ce6073640 1 --2- 172.21.15.116:0/3458026254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ce0101910 0x7f6ce0101d90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009900 tx=0x7f6cd002bc30 comp rx=0 tx=0).stop 2024-03-19T01:28:38.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.571+0000 7f6ce6073640 1 -- 172.21.15.116:0/3458026254 shutdown_connections 2024-03-19T01:28:38.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.571+0000 7f6ce6073640 1 --2- 172.21.15.116:0/3458026254 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ce01022d0 0x7f6ce0108b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.571+0000 7f6ce6073640 1 --2- 172.21.15.116:0/3458026254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ce0101910 0x7f6ce0101d90 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.571+0000 7f6ce6073640 1 --2- 172.21.15.116:0/3458026254 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ce0100720 0x7f6ce0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.571+0000 7f6ce6073640 1 -- 172.21.15.116:0/3458026254 >> 172.21.15.116:0/3458026254 conn(0x7f6ce00fbcb0 msgr2=0x7f6ce00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:38.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.571+0000 7f6ce6073640 1 -- 172.21.15.116:0/3458026254 shutdown_connections 2024-03-19T01:28:38.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.571+0000 7f6ce6073640 1 -- 172.21.15.116:0/3458026254 wait complete. 2024-03-19T01:28:38.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.571+0000 7f6ce6073640 1 Processor -- start 2024-03-19T01:28:38.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6ce6073640 1 -- start start 2024-03-19T01:28:38.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6ce6073640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ce0100720 0x7f6ce0197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6ce6073640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ce0101910 0x7f6ce01980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6ce6073640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ce01022d0 0x7f6ce019f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6ce6073640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce010d4c0 con 0x7f6ce0100720 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6ce6073640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce010d340 con 0x7f6ce01022d0 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6ce6073640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce010d640 con 0x7f6ce0101910 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ce0100720 0x7f6ce0197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ce0100720 0x7f6ce0197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48870/0 (socket says 172.21.15.116:48870) 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdf7fe640 1 -- 172.21.15.116:0/3970966430 learned_addr learned my addr 172.21.15.116:0/3970966430 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdeffd640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ce0101910 0x7f6ce01980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdffff640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ce01022d0 0x7f6ce019f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdf7fe640 1 -- 172.21.15.116:0/3970966430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ce0101910 msgr2=0x7f6ce01980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdf7fe640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ce0101910 0x7f6ce01980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdf7fe640 1 -- 172.21.15.116:0/3970966430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ce01022d0 msgr2=0x7f6ce019f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdf7fe640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ce01022d0 0x7f6ce019f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdf7fe640 1 -- 172.21.15.116:0/3970966430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce019f780 con 0x7f6ce0100720 2024-03-19T01:28:38.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdf7fe640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ce0100720 0x7f6ce0197b60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400e9c0 tx=0x7f6cd400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:38.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdcff9640 1 -- 172.21.15.116:0/3970966430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd400ccc0 con 0x7f6ce0100720 2024-03-19T01:28:38.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdcff9640 1 -- 172.21.15.116:0/3970966430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cd40035f0 con 0x7f6ce0100720 2024-03-19T01:28:38.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6ce6073640 1 -- 172.21.15.116:0/3970966430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ce019fa70 con 0x7f6ce0100720 2024-03-19T01:28:38.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.572+0000 7f6cdcff9640 1 -- 172.21.15.116:0/3970966430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd400ce60 con 0x7f6ce0100720 2024-03-19T01:28:38.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.573+0000 7f6ce6073640 1 -- 172.21.15.116:0/3970966430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ce019ff90 con 0x7f6ce0100720 2024-03-19T01:28:38.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.573+0000 7f6ce6073640 1 -- 172.21.15.116:0/3970966430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ce0068610 con 0x7f6ce0100720 2024-03-19T01:28:38.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.575+0000 7f6cdcff9640 1 -- 172.21.15.116:0/3970966430 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f6cd40037a0 con 0x7f6ce0100720 2024-03-19T01:28:38.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.575+0000 7f6cdcff9640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6cc80a2800 0x7f6cc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:38.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.575+0000 7f6cdcff9640 1 -- 172.21.15.116:0/3970966430 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f6cd40cef00 con 0x7f6ce0100720 2024-03-19T01:28:38.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.575+0000 7f6cdeffd640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6cc80a2800 0x7f6cc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:38.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.576+0000 7f6cdeffd640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6cc80a2800 0x7f6cc80a4cc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6cd002c7b0 tx=0x7f6cd0005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:38.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.579+0000 7f6cdcff9640 1 -- 172.21.15.116:0/3970966430 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6cd409bf80 con 0x7f6ce0100720 2024-03-19T01:28:38.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.793+0000 7f6ce6073640 1 -- 172.21.15.116:0/3970966430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6ce002d1f0 con 0x7f6ce0100720 2024-03-19T01:28:38.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.794+0000 7f6cdcff9640 1 -- 172.21.15.116:0/3970966430 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) ==== 104+0+11 (secure 0 0 0) 0x7f6cd409b880 con 0x7f6ce0100720 2024-03-19T01:28:38.794 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-03-19T01:28:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.796+0000 7f6ce6073640 1 -- 172.21.15.116:0/3970966430 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6cc80a2800 msgr2=0x7f6cc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.796+0000 7f6ce6073640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6cc80a2800 0x7f6cc80a4cc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6cd002c7b0 tx=0x7f6cd0005fd0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.796+0000 7f6ce6073640 1 -- 172.21.15.116:0/3970966430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ce0100720 msgr2=0x7f6ce0197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.796+0000 7f6ce6073640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ce0100720 0x7f6ce0197b60 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400e9c0 tx=0x7f6cd400edc0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.796+0000 7f6ce6073640 1 -- 172.21.15.116:0/3970966430 shutdown_connections 2024-03-19T01:28:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.796+0000 7f6ce6073640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f6cc80a2800 0x7f6cc80a4cc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.796+0000 7f6ce6073640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ce01022d0 0x7f6ce019f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.796+0000 7f6ce6073640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f6ce0101910 0x7f6ce01980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.796+0000 7f6ce6073640 1 --2- 172.21.15.116:0/3970966430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ce0100720 0x7f6ce0197b60 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.796+0000 7f6ce6073640 1 -- 172.21.15.116:0/3970966430 >> 172.21.15.116:0/3970966430 conn(0x7f6ce00fbcb0 msgr2=0x7f6ce0104c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.796+0000 7f6ce6073640 1 -- 172.21.15.116:0/3970966430 shutdown_connections 2024-03-19T01:28:38.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.796+0000 7f6ce6073640 1 -- 172.21.15.116:0/3970966430 wait complete. 2024-03-19T01:28:38.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-19T01:28:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.912+0000 7f3f30636640 1 -- 172.21.15.116:0/1884884637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f281022c0 msgr2=0x7f3f281046b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.912+0000 7f3f30636640 1 --2- 172.21.15.116:0/1884884637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f281022c0 0x7f3f281046b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009900 tx=0x7f3f2402bc30 comp rx=0 tx=0).stop 2024-03-19T01:28:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.912+0000 7f3f30636640 1 -- 172.21.15.116:0/1884884637 shutdown_connections 2024-03-19T01:28:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.912+0000 7f3f30636640 1 --2- 172.21.15.116:0/1884884637 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f28107520 0x7f3f28109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.912+0000 7f3f30636640 1 --2- 172.21.15.116:0/1884884637 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f28104bf0 0x7f3f28106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.912+0000 7f3f30636640 1 --2- 172.21.15.116:0/1884884637 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f281022c0 0x7f3f281046b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.912+0000 7f3f30636640 1 -- 172.21.15.116:0/1884884637 >> 172.21.15.116:0/1884884637 conn(0x7f3f280fbcb0 msgr2=0x7f3f280fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.912+0000 7f3f30636640 1 -- 172.21.15.116:0/1884884637 shutdown_connections 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.912+0000 7f3f30636640 1 -- 172.21.15.116:0/1884884637 wait complete. 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f30636640 1 Processor -- start 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f30636640 1 -- start start 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f30636640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f281022c0 0x7f3f28197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f30636640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f28104bf0 0x7f3f281980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f30636640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f28107520 0x7f3f2819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f30636640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2810d4c0 con 0x7f3f281022c0 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f30636640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2810d340 con 0x7f3f28107520 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f30636640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2810d640 con 0x7f3f28104bf0 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f2e3d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f281022c0 0x7f3f28197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f2e3d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f281022c0 0x7f3f28197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48886/0 (socket says 172.21.15.116:48886) 2024-03-19T01:28:38.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f2e3d2640 1 -- 172.21.15.116:0/4143875938 learned_addr learned my addr 172.21.15.116:0/4143875938 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f2ebd3640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f28107520 0x7f3f2819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f2dbd1640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f28104bf0 0x7f3f281980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f2e3d2640 1 -- 172.21.15.116:0/4143875938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f28104bf0 msgr2=0x7f3f281980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f2e3d2640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f28104bf0 0x7f3f281980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f2e3d2640 1 -- 172.21.15.116:0/4143875938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f28107520 msgr2=0x7f3f2819f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f2e3d2640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f28107520 0x7f3f2819f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f2e3d2640 1 -- 172.21.15.116:0/4143875938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f2819f7b0 con 0x7f3f281022c0 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f2ebd3640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f28107520 0x7f3f2819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.913+0000 7f3f2e3d2640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f281022c0 0x7f3f28197b90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009230 tx=0x7f3f2402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.914+0000 7f3f1f7fe640 1 -- 172.21.15.116:0/4143875938 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f24041070 con 0x7f3f281022c0 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.914+0000 7f3f30636640 1 -- 172.21.15.116:0/4143875938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f2819fa40 con 0x7f3f281022c0 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.914+0000 7f3f1f7fe640 1 -- 172.21.15.116:0/4143875938 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f24004450 con 0x7f3f281022c0 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.914+0000 7f3f1f7fe640 1 -- 172.21.15.116:0/4143875938 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f24003e20 con 0x7f3f281022c0 2024-03-19T01:28:38.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.914+0000 7f3f30636640 1 -- 172.21.15.116:0/4143875938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f2819ff80 con 0x7f3f281022c0 2024-03-19T01:28:38.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.915+0000 7f3f30636640 1 -- 172.21.15.116:0/4143875938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f280fd7a0 con 0x7f3f281022c0 2024-03-19T01:28:38.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.916+0000 7f3f1f7fe640 1 -- 172.21.15.116:0/4143875938 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f3f24044070 con 0x7f3f281022c0 2024-03-19T01:28:38.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.917+0000 7f3f1f7fe640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3f140a2800 0x7f3f140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:38.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.917+0000 7f3f2dbd1640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3f140a2800 0x7f3f140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:38.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.917+0000 7f3f1f7fe640 1 -- 172.21.15.116:0/4143875938 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f3f240f09c0 con 0x7f3f281022c0 2024-03-19T01:28:38.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.917+0000 7f3f2dbd1640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3f140a2800 0x7f3f140a4cc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3f281990a0 tx=0x7f3f18005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:38.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:38.921+0000 7f3f1f7fe640 1 -- 172.21.15.116:0/4143875938 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3f240bd3f0 con 0x7f3f281022c0 2024-03-19T01:28:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.104+0000 7f3f30636640 1 -- 172.21.15.116:0/4143875938 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3f2802d3c0 con 0x7f3f281022c0 2024-03-19T01:28:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.104+0000 7f3f1f7fe640 1 -- 172.21.15.116:0/4143875938 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) ==== 100+0+11 (secure 0 0 0) 0x7f3f240c1490 con 0x7f3f281022c0 2024-03-19T01:28:39.105 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-03-19T01:28:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.106+0000 7f3f30636640 1 -- 172.21.15.116:0/4143875938 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3f140a2800 msgr2=0x7f3f140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.106+0000 7f3f30636640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3f140a2800 0x7f3f140a4cc0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3f281990a0 tx=0x7f3f18005f90 comp rx=0 tx=0).stop 2024-03-19T01:28:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.106+0000 7f3f30636640 1 -- 172.21.15.116:0/4143875938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f281022c0 msgr2=0x7f3f28197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:39.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.106+0000 7f3f30636640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f281022c0 0x7f3f28197b90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009230 tx=0x7f3f2402c650 comp rx=0 tx=0).stop 2024-03-19T01:28:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.106+0000 7f3f30636640 1 -- 172.21.15.116:0/4143875938 shutdown_connections 2024-03-19T01:28:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.106+0000 7f3f30636640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f3f140a2800 0x7f3f140a4cc0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.106+0000 7f3f30636640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f28107520 0x7f3f2819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.106+0000 7f3f30636640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f3f28104bf0 0x7f3f281980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.106+0000 7f3f30636640 1 --2- 172.21.15.116:0/4143875938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f281022c0 0x7f3f28197b90 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.106+0000 7f3f30636640 1 -- 172.21.15.116:0/4143875938 >> 172.21.15.116:0/4143875938 conn(0x7f3f280fbcb0 msgr2=0x7f3f28102cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.106+0000 7f3f30636640 1 -- 172.21.15.116:0/4143875938 shutdown_connections 2024-03-19T01:28:39.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.107+0000 7f3f30636640 1 -- 172.21.15.116:0/4143875938 wait complete. 2024-03-19T01:28:39.118 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-19T01:28:39.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.220+0000 7f85ecafe640 1 -- 172.21.15.116:0/1164832035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e81022d0 msgr2=0x7f85e8108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:39.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.220+0000 7f85ecafe640 1 --2- 172.21.15.116:0/1164832035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e81022d0 0x7f85e8108b60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f85e0009ca0 tx=0x7f85e002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.220+0000 7f85ecafe640 1 -- 172.21.15.116:0/1164832035 shutdown_connections 2024-03-19T01:28:39.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.220+0000 7f85ecafe640 1 --2- 172.21.15.116:0/1164832035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e81022d0 0x7f85e8108b60 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.220+0000 7f85ecafe640 1 --2- 172.21.15.116:0/1164832035 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85e8101910 0x7f85e8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.220+0000 7f85ecafe640 1 --2- 172.21.15.116:0/1164832035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85e8100720 0x7f85e8100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.220+0000 7f85ecafe640 1 -- 172.21.15.116:0/1164832035 >> 172.21.15.116:0/1164832035 conn(0x7f85e80fbcb0 msgr2=0x7f85e80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.220+0000 7f85ecafe640 1 -- 172.21.15.116:0/1164832035 shutdown_connections 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.221+0000 7f85ecafe640 1 -- 172.21.15.116:0/1164832035 wait complete. 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.221+0000 7f85ecafe640 1 Processor -- start 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.221+0000 7f85ecafe640 1 -- start start 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.221+0000 7f85ecafe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8100720 0x7f85e8197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.221+0000 7f85ecafe640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85e8101910 0x7f85e8198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.221+0000 7f85ecafe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85e81022d0 0x7f85e819f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.221+0000 7f85ecafe640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f85e810d4c0 con 0x7f85e8100720 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.221+0000 7f85ecafe640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f85e810d340 con 0x7f85e81022d0 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.221+0000 7f85ecafe640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f85e810d640 con 0x7f85e8101910 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.221+0000 7f85e659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8100720 0x7f85e8197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.221+0000 7f85e659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8100720 0x7f85e8197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48898/0 (socket says 172.21.15.116:48898) 2024-03-19T01:28:39.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.221+0000 7f85e659c640 1 -- 172.21.15.116:0/615043691 learned_addr learned my addr 172.21.15.116:0/615043691 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.222+0000 7f85dffff640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85e8101910 0x7f85e8198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.222+0000 7f85e6d9d640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85e81022d0 0x7f85e819f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.222+0000 7f85e659c640 1 -- 172.21.15.116:0/615043691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85e8101910 msgr2=0x7f85e8198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.222+0000 7f85e659c640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85e8101910 0x7f85e8198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.222+0000 7f85e659c640 1 -- 172.21.15.116:0/615043691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85e81022d0 msgr2=0x7f85e819f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.222+0000 7f85e659c640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85e81022d0 0x7f85e819f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.222+0000 7f85e659c640 1 -- 172.21.15.116:0/615043691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e819f730 con 0x7f85e8100720 2024-03-19T01:28:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.222+0000 7f85e659c640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8100720 0x7f85e8197b10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f85d800e9c0 tx=0x7f85d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.222+0000 7f85df7fe640 1 -- 172.21.15.116:0/615043691 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d800cd80 con 0x7f85e8100720 2024-03-19T01:28:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.222+0000 7f85df7fe640 1 -- 172.21.15.116:0/615043691 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85d80035f0 con 0x7f85e8100720 2024-03-19T01:28:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.222+0000 7f85ecafe640 1 -- 172.21.15.116:0/615043691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e819fa20 con 0x7f85e8100720 2024-03-19T01:28:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.222+0000 7f85df7fe640 1 -- 172.21.15.116:0/615043691 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8010690 con 0x7f85e8100720 2024-03-19T01:28:39.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.222+0000 7f85ecafe640 1 -- 172.21.15.116:0/615043691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e819ff70 con 0x7f85e8100720 2024-03-19T01:28:39.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.223+0000 7f85ecafe640 1 -- 172.21.15.116:0/615043691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85e8107a40 con 0x7f85e8100720 2024-03-19T01:28:39.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.225+0000 7f85df7fe640 1 -- 172.21.15.116:0/615043691 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f85d80037a0 con 0x7f85e8100720 2024-03-19T01:28:39.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.226+0000 7f85df7fe640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f85d40a2800 0x7f85d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.226+0000 7f85dffff640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f85d40a2800 0x7f85d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.226+0000 7f85df7fe640 1 -- 172.21.15.116:0/615043691 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f85d80cef10 con 0x7f85e8100720 2024-03-19T01:28:39.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.226+0000 7f85dffff640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f85d40a2800 0x7f85d40a4cc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f85e8199020 tx=0x7f85d0005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:39.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.228+0000 7f85df7fe640 1 -- 172.21.15.116:0/615043691 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f85d809c020 con 0x7f85e8100720 2024-03-19T01:28:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:39 smithi162 ceph-mon[67178]: pgmap v832: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.4 KiB/s wr, 12 op/s 2024-03-19T01:28:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:39 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1431191959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:28:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:39 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/396565375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:28:39.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:39 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3970966430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:28:39.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:39 smithi116 ceph-mon[62701]: pgmap v832: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.4 KiB/s wr, 12 op/s 2024-03-19T01:28:39.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:39 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1431191959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:28:39.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:39 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/396565375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:28:39.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:39 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3970966430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:28:39.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.455+0000 7f85ecafe640 1 -- 172.21.15.116:0/615043691 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f85e802d290 con 0x7f85e8100720 2024-03-19T01:28:39.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.455+0000 7f85df7fe640 1 -- 172.21.15.116:0/615043691 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) ==== 81+0+602 (secure 0 0 0) 0x7f85d809b920 con 0x7f85e8100720 2024-03-19T01:28:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/3676795487 2024-03-20T01:03:00.584768+0000 2024-03-19T01:28:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/1073894644 2024-03-20T01:03:26.974893+0000 2024-03-19T01:28:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6800/1107242393 2024-03-20T01:03:26.974893+0000 2024-03-19T01:28:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/3553505800 2024-03-20T01:03:00.584768+0000 2024-03-19T01:28:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/1107242393 2024-03-20T01:03:26.974893+0000 2024-03-19T01:28:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6801/3676795487 2024-03-20T01:03:00.584768+0000 2024-03-19T01:28:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2950253911 2024-03-20T01:03:26.974893+0000 2024-03-19T01:28:39.457 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2689675916 2024-03-20T01:03:00.584768+0000 2024-03-19T01:28:39.457 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/1322211514 2024-03-20T01:03:00.584768+0000 2024-03-19T01:28:39.457 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:0/2178441515 2024-03-20T01:03:26.974893+0000 2024-03-19T01:28:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:listed 10 entries 2024-03-19T01:28:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.457+0000 7f85ecafe640 1 -- 172.21.15.116:0/615043691 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f85d40a2800 msgr2=0x7f85d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.457+0000 7f85ecafe640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f85d40a2800 0x7f85d40a4cc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f85e8199020 tx=0x7f85d0005f90 comp rx=0 tx=0).stop 2024-03-19T01:28:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.457+0000 7f85ecafe640 1 -- 172.21.15.116:0/615043691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8100720 msgr2=0x7f85e8197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.457+0000 7f85ecafe640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8100720 0x7f85e8197b10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f85d800e9c0 tx=0x7f85d800ee80 comp rx=0 tx=0).stop 2024-03-19T01:28:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.457+0000 7f85ecafe640 1 -- 172.21.15.116:0/615043691 shutdown_connections 2024-03-19T01:28:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.457+0000 7f85ecafe640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f85d40a2800 0x7f85d40a4cc0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.457+0000 7f85ecafe640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f85e81022d0 0x7f85e819f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.457+0000 7f85ecafe640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f85e8101910 0x7f85e8198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.457+0000 7f85ecafe640 1 --2- 172.21.15.116:0/615043691 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85e8100720 0x7f85e8197b10 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.457+0000 7f85ecafe640 1 -- 172.21.15.116:0/615043691 >> 172.21.15.116:0/615043691 conn(0x7f85e80fbcb0 msgr2=0x7f85e8104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.457+0000 7f85ecafe640 1 -- 172.21.15.116:0/615043691 shutdown_connections 2024-03-19T01:28:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:39.457+0000 7f85ecafe640 1 -- 172.21.15.116:0/615043691 wait complete. 2024-03-19T01:28:39.469 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-19T01:28:39.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:28:39.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo umount /tmp/cephfs 2024-03-19T01:28:39.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:39 smithi156 ceph-mon[66799]: pgmap v832: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 5.4 KiB/s wr, 12 op/s 2024-03-19T01:28:39.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:39 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1431191959' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:28:39.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:39 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/396565375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-19T01:28:39.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:39 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3970966430' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-19T01:28:39.587 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T01:28:39.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:28:39.587 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /tmp/cephfs) 2024-03-19T01:28:39.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:28:39.646 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-03-19T01:28:39.685 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--tmp-cephfs (id: 0) 2024-03-19T01:28:39.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:28:39.686 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns delete ceph-ns--tmp-cephfs 2024-03-19T01:28:39.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-19T01:28:39.799 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-03-19T01:28:39.899 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-19T01:28:39.910 INFO:tasks.cephadm:Teardown begin 2024-03-19T01:28:39.910 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-19T01:28:39.910 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:28:40.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.099+0000 7f796ac9a640 1 -- 172.21.15.116:0/1122037499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7964108820 msgr2=0x7f796410ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:40.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.099+0000 7f796ac9a640 1 --2- 172.21.15.116:0/1122037499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7964108820 0x7f796410ac50 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f794c01a910 tx=0x7f794c03cca0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.100+0000 7f796ac9a640 1 -- 172.21.15.116:0/1122037499 shutdown_connections 2024-03-19T01:28:40.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.100+0000 7f796ac9a640 1 --2- 172.21.15.116:0/1122037499 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f796410db60 0x7f796410e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.100+0000 7f796ac9a640 1 --2- 172.21.15.116:0/1122037499 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f796410b190 0x7f796410d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.100+0000 7f796ac9a640 1 --2- 172.21.15.116:0/1122037499 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7964108820 0x7f796410ac50 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.100+0000 7f796ac9a640 1 -- 172.21.15.116:0/1122037499 >> 172.21.15.116:0/1122037499 conn(0x7f796406e690 msgr2=0x7f7964070ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:40.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.102+0000 7f796ac9a640 1 -- 172.21.15.116:0/1122037499 shutdown_connections 2024-03-19T01:28:40.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.102+0000 7f796ac9a640 1 -- 172.21.15.116:0/1122037499 wait complete. 2024-03-19T01:28:40.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.103+0000 7f796ac9a640 1 Processor -- start 2024-03-19T01:28:40.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.103+0000 7f796ac9a640 1 -- start start 2024-03-19T01:28:40.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.103+0000 7f796ac9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7964108820 0x7f796419e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:40.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.103+0000 7f796ac9a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f796410b190 0x7f796419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:40.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.103+0000 7f796ac9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796410db60 0x7f79641a5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:40.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.103+0000 7f796ac9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7964111b20 con 0x7f796410db60 2024-03-19T01:28:40.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.103+0000 7f796ac9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f79641119a0 con 0x7f7964108820 2024-03-19T01:28:40.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.103+0000 7f796ac9a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f7964111ca0 con 0x7f796410b190 2024-03-19T01:28:40.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.104+0000 7f7969237640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796410db60 0x7f79641a5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:40.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.104+0000 7f7969237640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796410db60 0x7f79641a5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48910/0 (socket says 172.21.15.116:48910) 2024-03-19T01:28:40.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.104+0000 7f7969237640 1 -- 172.21.15.116:0/2938465789 learned_addr learned my addr 172.21.15.116:0/2938465789 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:40.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.104+0000 7f7963fff640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f796410b190 0x7f796419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:40.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.104+0000 7f7968a36640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7964108820 0x7f796419e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:40.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.108+0000 7f7968a36640 1 -- 172.21.15.116:0/2938465789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f796410b190 msgr2=0x7f796419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:40.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.108+0000 7f7968a36640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f796410b190 0x7f796419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.108+0000 7f7968a36640 1 -- 172.21.15.116:0/2938465789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796410db60 msgr2=0x7f79641a5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:40.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.108+0000 7f7968a36640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796410db60 0x7f79641a5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.108+0000 7f7968a36640 1 -- 172.21.15.116:0/2938465789 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79641a5db0 con 0x7f7964108820 2024-03-19T01:28:40.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.108+0000 7f7963fff640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f796410b190 0x7f796419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:28:40.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.108+0000 7f7968a36640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7964108820 0x7f796419e190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f794c047040 tx=0x7f794c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:40.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.108+0000 7f7969237640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796410db60 0x7f79641a5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:28:40.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.109+0000 7f7961ffb640 1 -- 172.21.15.116:0/2938465789 <== mon.2 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c03d850 con 0x7f7964108820 2024-03-19T01:28:40.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.109+0000 7f7961ffb640 1 -- 172.21.15.116:0/2938465789 <== mon.2 v2:172.21.15.156:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f794c0035f0 con 0x7f7964108820 2024-03-19T01:28:40.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.111+0000 7f796ac9a640 1 -- 172.21.15.116:0/2938465789 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79641a6040 con 0x7f7964108820 2024-03-19T01:28:40.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.111+0000 7f796ac9a640 1 -- 172.21.15.116:0/2938465789 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79641a6500 con 0x7f7964108820 2024-03-19T01:28:40.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.111+0000 7f7961ffb640 1 -- 172.21.15.116:0/2938465789 <== mon.2 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f794c04e480 con 0x7f7964108820 2024-03-19T01:28:40.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.115+0000 7f796ac9a640 1 -- 172.21.15.116:0/2938465789 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79641a6820 con 0x7f7964108820 2024-03-19T01:28:40.119 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:40 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/4143875938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:28:40.119 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:40 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/615043691' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T01:28:40.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.127+0000 7f7961ffb640 1 -- 172.21.15.116:0/2938465789 <== mon.2 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f794c055060 con 0x7f7964108820 2024-03-19T01:28:40.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.127+0000 7f7961ffb640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f79500a2980 0x7f79500a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:40.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.128+0000 7f7961ffb640 1 -- 172.21.15.116:0/2938465789 <== mon.2 v2:172.21.15.156:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f794c101360 con 0x7f7964108820 2024-03-19T01:28:40.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.130+0000 7f7963fff640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f79500a2980 0x7f79500a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:40.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.136+0000 7f7961ffb640 1 -- 172.21.15.116:0/2938465789 <== mon.2 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f794c05c070 con 0x7f7964108820 2024-03-19T01:28:40.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.141+0000 7f7963fff640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f79500a2980 0x7f79500a4e40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f796419f6a0 tx=0x7f795400a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:40 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/4143875938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:28:40.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:40 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/615043691' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T01:28:40.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:40 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/4143875938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-19T01:28:40.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:40 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/615043691' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-19T01:28:40.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.569+0000 7f796ac9a640 1 -- 172.21.15.116:0/2938465789 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79641a69f0 con 0x7f7964108820 2024-03-19T01:28:40.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.570+0000 7f7961ffb640 1 -- 172.21.15.116:0/2938465789 <== mon.2 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11956 (secure 0 0 0) 0x7f794c0ce400 con 0x7f7964108820 2024-03-19T01:28:40.573 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:28:40.573 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T01:10:32.487607+0000","modified":"2024-03-19T01:28:32.411419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24488,"mds_2":24473,"mds_3":24460,"mds_4":24445},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1242639081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1242639081},{"type":"v1","addr":"172.21.15.116:6839","nonce":1242639081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24394":{"gid":24394,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":3,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.162:6839/441486392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":441486392},{"type":"v1","addr":"172.21.15.162:6839","nonce":441486392}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"a","rank":1,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6841/3343188504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3343188504},{"type":"v1","addr":"172.21.15.116:6841","nonce":3343188504}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1601537427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1601537427},{"type":"v1","addr":"172.21.15.156:6839","nonce":1601537427}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24473,24445,24500,24460,24488]},"id":1}]} 2024-03-19T01:28:40.573 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 56 2024-03-19T01:28:40.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.578+0000 7f796ac9a640 1 -- 172.21.15.116:0/2938465789 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f79500a2980 msgr2=0x7f79500a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:40.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.578+0000 7f796ac9a640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f79500a2980 0x7f79500a4e40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f796419f6a0 tx=0x7f795400a2d0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.578+0000 7f796ac9a640 1 -- 172.21.15.116:0/2938465789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7964108820 msgr2=0x7f796419e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:40.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.578+0000 7f796ac9a640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7964108820 0x7f796419e190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f794c047040 tx=0x7f794c004270 comp rx=0 tx=0).stop 2024-03-19T01:28:40.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.579+0000 7f796ac9a640 1 -- 172.21.15.116:0/2938465789 shutdown_connections 2024-03-19T01:28:40.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.579+0000 7f796ac9a640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f796410db60 0x7f79641a5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.579+0000 7f796ac9a640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f796410b190 0x7f796419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.579+0000 7f796ac9a640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f79500a2980 0x7f79500a4e40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.579+0000 7f796ac9a640 1 --2- 172.21.15.116:0/2938465789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7964108820 0x7f796419e190 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.579+0000 7f796ac9a640 1 -- 172.21.15.116:0/2938465789 >> 172.21.15.116:0/2938465789 conn(0x7f796406e690 msgr2=0x7f7964108fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.582+0000 7f796ac9a640 1 -- 172.21.15.116:0/2938465789 shutdown_connections 2024-03-19T01:28:40.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.582+0000 7f796ac9a640 1 -- 172.21.15.116:0/2938465789 wait complete. 2024-03-19T01:28:40.599 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-19T01:28:40.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.777+0000 7f4838fb9640 1 -- 172.21.15.116:0/2776751130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834105610 msgr2=0x7f483410bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:40.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.777+0000 7f4838fb9640 1 --2- 172.21.15.116:0/2776751130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834105610 0x7f483410bea0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f482c009ca0 tx=0x7f482c02c0f0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.777+0000 7f4838fb9640 1 -- 172.21.15.116:0/2776751130 shutdown_connections 2024-03-19T01:28:40.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.777+0000 7f4838fb9640 1 --2- 172.21.15.116:0/2776751130 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834105610 0x7f483410bea0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.777+0000 7f4838fb9640 1 --2- 172.21.15.116:0/2776751130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834104c50 0x7f48341050d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.777+0000 7f4838fb9640 1 --2- 172.21.15.116:0/2776751130 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4834103a60 0x7f4834103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.777+0000 7f4838fb9640 1 -- 172.21.15.116:0/2776751130 >> 172.21.15.116:0/2776751130 conn(0x7f48340fefd0 msgr2=0x7f4834101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:40.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.777+0000 7f4838fb9640 1 -- 172.21.15.116:0/2776751130 shutdown_connections 2024-03-19T01:28:40.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.777+0000 7f4838fb9640 1 -- 172.21.15.116:0/2776751130 wait complete. 2024-03-19T01:28:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.779+0000 7f4838fb9640 1 Processor -- start 2024-03-19T01:28:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.779+0000 7f4838fb9640 1 -- start start 2024-03-19T01:28:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.779+0000 7f4838fb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834103a60 0x7f4834199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.779+0000 7f4838fb9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4834104c50 0x7f483419a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.779+0000 7f4838fb9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834105610 0x7f48341a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.779+0000 7f4838fb9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f483410f6c0 con 0x7f4834103a60 2024-03-19T01:28:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.779+0000 7f4838fb9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f483410f540 con 0x7f4834105610 2024-03-19T01:28:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.779+0000 7f4838fb9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f483410f840 con 0x7f4834104c50 2024-03-19T01:28:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.779+0000 7f483259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834103a60 0x7f4834199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.779+0000 7f483259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834103a60 0x7f4834199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48918/0 (socket says 172.21.15.116:48918) 2024-03-19T01:28:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.779+0000 7f483259c640 1 -- 172.21.15.116:0/1514922290 learned_addr learned my addr 172.21.15.116:0/1514922290 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:40.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.779+0000 7f4831d9b640 1 --2- 172.21.15.116:0/1514922290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4834104c50 0x7f483419a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.780+0000 7f4831d9b640 1 -- 172.21.15.116:0/1514922290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834105610 msgr2=0x7f48341a1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.780+0000 7f4831d9b640 1 --2- 172.21.15.116:0/1514922290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834105610 0x7f48341a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.780+0000 7f4831d9b640 1 -- 172.21.15.116:0/1514922290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834103a60 msgr2=0x7f4834199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.780+0000 7f4831d9b640 1 --2- 172.21.15.116:0/1514922290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834103a60 0x7f4834199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.780+0000 7f4831d9b640 1 -- 172.21.15.116:0/1514922290 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48341a1780 con 0x7f4834104c50 2024-03-19T01:28:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.780+0000 7f4831d9b640 1 --2- 172.21.15.116:0/1514922290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4834104c50 0x7f483419a0a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f482800e9f0 tx=0x7f482800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.781+0000 7f48237fe640 1 -- 172.21.15.116:0/1514922290 <== mon.1 v2:172.21.15.162:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482800ccc0 con 0x7f4834104c50 2024-03-19T01:28:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.781+0000 7f48237fe640 1 -- 172.21.15.116:0/1514922290 <== mon.1 v2:172.21.15.162:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4828010910 con 0x7f4834104c50 2024-03-19T01:28:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.782+0000 7f48237fe640 1 -- 172.21.15.116:0/1514922290 <== mon.1 v2:172.21.15.162:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4828018980 con 0x7f4834104c50 2024-03-19T01:28:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.782+0000 7f4838fb9640 1 -- 172.21.15.116:0/1514922290 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48341a1a70 con 0x7f4834104c50 2024-03-19T01:28:40.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.782+0000 7f4838fb9640 1 -- 172.21.15.116:0/1514922290 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48341a1e80 con 0x7f4834104c50 2024-03-19T01:28:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.784+0000 7f4838fb9640 1 -- 172.21.15.116:0/1514922290 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f483410acb0 con 0x7f4834104c50 2024-03-19T01:28:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.785+0000 7f48237fe640 1 -- 172.21.15.116:0/1514922290 <== mon.1 v2:172.21.15.162:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f4828010ac0 con 0x7f4834104c50 2024-03-19T01:28:40.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.786+0000 7f48237fe640 1 --2- 172.21.15.116:0/1514922290 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f481c0a2770 0x7f481c0a4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.787+0000 7f48237fe640 1 -- 172.21.15.116:0/1514922290 <== mon.1 v2:172.21.15.162:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f48280cefb0 con 0x7f4834104c50 2024-03-19T01:28:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.787+0000 7f483259c640 1 --2- 172.21.15.116:0/1514922290 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f481c0a2770 0x7f481c0a4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:40.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.787+0000 7f483259c640 1 --2- 172.21.15.116:0/1514922290 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f481c0a2770 0x7f481c0a4c30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f48240058c0 tx=0x7f4824005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:40.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:40.800+0000 7f48237fe640 1 -- 172.21.15.116:0/1514922290 <== mon.1 v2:172.21.15.162:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f482809c030 con 0x7f4834104c50 2024-03-19T01:28:41.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.079+0000 7f4838fb9640 1 -- 172.21.15.116:0/1514922290 --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4834062420 con 0x7f4834104c50 2024-03-19T01:28:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.080+0000 7f48237fe640 1 -- 172.21.15.116:0/1514922290 <== mon.1 v2:172.21.15.162:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+11956 (secure 0 0 0) 0x7f482809b930 con 0x7f4834104c50 2024-03-19T01:28:41.080 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:28:41.081 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14535,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/909235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":909235},{"type":"v1","addr":"172.21.15.116:6835","nonce":909235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6837/3764874477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6836","nonce":3764874477},{"type":"v1","addr":"172.21.15.116:6837","nonce":3764874477}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":56,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-19T01:10:32.487607+0000","modified":"2024-03-19T01:28:32.411419+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24500,"mds_1":24488,"mds_2":24473,"mds_3":24460,"mds_4":24445},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.116:6839/1242639081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6838","nonce":1242639081},{"type":"v1","addr":"172.21.15.116:6839","nonce":1242639081}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24394":{"gid":24394,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6833/3113382416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6832","nonce":3113382416},{"type":"v1","addr":"172.21.15.162:6833","nonce":3113382416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24410":{"gid":24410,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6833/1647822466","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1647822466},{"type":"v1","addr":"172.21.15.156:6833","nonce":1647822466}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.162:6835/1592939032","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6834","nonce":1592939032},{"type":"v1","addr":"172.21.15.162:6835","nonce":1592939032}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24434":{"gid":24434,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6835/1738017780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1738017780},{"type":"v1","addr":"172.21.15.156:6835","nonce":1738017780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"c","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.162:6837/585664575","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6836","nonce":585664575},{"type":"v1","addr":"172.21.15.162:6837","nonce":585664575}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"i","rank":3,"incarnation":20,"state":"up:active","state_seq":6,"addr":"172.21.15.162:6839/441486392","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6838","nonce":441486392},{"type":"v1","addr":"172.21.15.162:6839","nonce":441486392}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"h","rank":2,"incarnation":18,"state":"up:active","state_seq":7,"addr":"172.21.15.156:6837/4147060396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":4147060396},{"type":"v1","addr":"172.21.15.156:6837","nonce":4147060396}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"a","rank":1,"incarnation":16,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6841/3343188504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6840","nonce":3343188504},{"type":"v1","addr":"172.21.15.116:6841","nonce":3343188504}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6839/1601537427","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":1601537427},{"type":"v1","addr":"172.21.15.156:6839","nonce":1601537427}]},"join_fscid":-1,"export_targets":[2,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24500,"qdb_cluster":[24473,24445,24500,24460,24488]},"id":1}]} 2024-03-19T01:28:41.081 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 56 2024-03-19T01:28:41.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.082+0000 7f4838fb9640 1 -- 172.21.15.116:0/1514922290 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f481c0a2770 msgr2=0x7f481c0a4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.082+0000 7f4838fb9640 1 --2- 172.21.15.116:0/1514922290 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f481c0a2770 0x7f481c0a4c30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f48240058c0 tx=0x7f4824005f90 comp rx=0 tx=0).stop 2024-03-19T01:28:41.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.082+0000 7f4838fb9640 1 -- 172.21.15.116:0/1514922290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4834104c50 msgr2=0x7f483419a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.082+0000 7f4838fb9640 1 --2- 172.21.15.116:0/1514922290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4834104c50 0x7f483419a0a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f482800e9f0 tx=0x7f482800eeb0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.082+0000 7f4838fb9640 1 -- 172.21.15.116:0/1514922290 shutdown_connections 2024-03-19T01:28:41.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.082+0000 7f4838fb9640 1 --2- 172.21.15.116:0/1514922290 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f481c0a2770 0x7f481c0a4c30 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.082+0000 7f4838fb9640 1 --2- 172.21.15.116:0/1514922290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834105610 0x7f48341a1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.082+0000 7f4838fb9640 1 --2- 172.21.15.116:0/1514922290 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4834104c50 0x7f483419a0a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.082+0000 7f4838fb9640 1 --2- 172.21.15.116:0/1514922290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4834103a60 0x7f4834199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.082+0000 7f4838fb9640 1 -- 172.21.15.116:0/1514922290 >> 172.21.15.116:0/1514922290 conn(0x7f48340fefd0 msgr2=0x7f4834107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.082+0000 7f4838fb9640 1 -- 172.21.15.116:0/1514922290 shutdown_connections 2024-03-19T01:28:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.082+0000 7f4838fb9640 1 -- 172.21.15.116:0/1514922290 wait complete. 2024-03-19T01:28:41.095 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-19T01:28:41.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.239+0000 7f1f5fadc640 1 -- 172.21.15.116:0/3203149855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58101930 msgr2=0x7f1f58101db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.239+0000 7f1f5fadc640 1 --2- 172.21.15.116:0/3203149855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58101930 0x7f1f58101db0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009900 tx=0x7f1f4802bc30 comp rx=0 tx=0).stop 2024-03-19T01:28:41.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.239+0000 7f1f5fadc640 1 -- 172.21.15.116:0/3203149855 shutdown_connections 2024-03-19T01:28:41.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.239+0000 7f1f5fadc640 1 --2- 172.21.15.116:0/3203149855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f581022f0 0x7f1f58108b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.239+0000 7f1f5fadc640 1 --2- 172.21.15.116:0/3203149855 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f58101930 0x7f1f58101db0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.239+0000 7f1f5fadc640 1 --2- 172.21.15.116:0/3203149855 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f58100740 0x7f1f58100b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.239+0000 7f1f5fadc640 1 -- 172.21.15.116:0/3203149855 >> 172.21.15.116:0/3203149855 conn(0x7f1f580fbcb0 msgr2=0x7f1f580fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:41.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.239+0000 7f1f5fadc640 1 -- 172.21.15.116:0/3203149855 shutdown_connections 2024-03-19T01:28:41.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.239+0000 7f1f5fadc640 1 -- 172.21.15.116:0/3203149855 wait complete. 2024-03-19T01:28:41.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.240+0000 7f1f5fadc640 1 Processor -- start 2024-03-19T01:28:41.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.240+0000 7f1f5fadc640 1 -- start start 2024-03-19T01:28:41.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.240+0000 7f1f5fadc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f58100740 0x7f1f58069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.240+0000 7f1f5fadc640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f58101930 0x7f1f5819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.240+0000 7f1f5fadc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f581022f0 0x7f1f5819eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.240+0000 7f1f5fadc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5810c3c0 con 0x7f1f581022f0 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.240+0000 7f1f5fadc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5810c240 con 0x7f1f58100740 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.240+0000 7f1f5fadc640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5810c540 con 0x7f1f58101930 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.240+0000 7f1f5e079640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f581022f0 0x7f1f5819eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.240+0000 7f1f5d878640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f58100740 0x7f1f58069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.240+0000 7f1f5d878640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f58100740 0x7f1f58069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.116:48678/0 (socket says 172.21.15.116:48678) 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.240+0000 7f1f5d077640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f58101930 0x7f1f5819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.240+0000 7f1f5d878640 1 -- 172.21.15.116:0/1964067972 learned_addr learned my addr 172.21.15.116:0/1964067972 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.241+0000 7f1f5e079640 1 -- 172.21.15.116:0/1964067972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f58101930 msgr2=0x7f1f5819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.241+0000 7f1f5e079640 1 --2- 172.21.15.116:0/1964067972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f58101930 0x7f1f5819c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.241+0000 7f1f5e079640 1 -- 172.21.15.116:0/1964067972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f58100740 msgr2=0x7f1f58069510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.241+0000 7f1f5e079640 1 --2- 172.21.15.116:0/1964067972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f58100740 0x7f1f58069510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.241+0000 7f1f5e079640 1 -- 172.21.15.116:0/1964067972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5819f770 con 0x7f1f581022f0 2024-03-19T01:28:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.241+0000 7f1f5d878640 1 --2- 172.21.15.116:0/1964067972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f58100740 0x7f1f58069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:28:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.241+0000 7f1f5e079640 1 --2- 172.21.15.116:0/1964067972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f581022f0 0x7f1f5819eeb0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1f50009fd0 tx=0x7f1f5000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.241+0000 7f1f4effd640 1 -- 172.21.15.116:0/1964067972 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5000ce20 con 0x7f1f581022f0 2024-03-19T01:28:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.241+0000 7f1f5fadc640 1 -- 172.21.15.116:0/1964067972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f5819fa60 con 0x7f1f581022f0 2024-03-19T01:28:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.241+0000 7f1f4effd640 1 -- 172.21.15.116:0/1964067972 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f50005020 con 0x7f1f581022f0 2024-03-19T01:28:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.241+0000 7f1f4effd640 1 -- 172.21.15.116:0/1964067972 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f50010720 con 0x7f1f581022f0 2024-03-19T01:28:41.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.241+0000 7f1f5fadc640 1 -- 172.21.15.116:0/1964067972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f5819ff30 con 0x7f1f581022f0 2024-03-19T01:28:41.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.242+0000 7f1f5fadc640 1 -- 172.21.15.116:0/1964067972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f58107ae0 con 0x7f1f581022f0 2024-03-19T01:28:41.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.243+0000 7f1f4effd640 1 -- 172.21.15.116:0/1964067972 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f1f50005540 con 0x7f1f581022f0 2024-03-19T01:28:41.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.244+0000 7f1f4effd640 1 --2- 172.21.15.116:0/1964067972 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1f440a2800 0x7f1f440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:41.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.244+0000 7f1f4effd640 1 -- 172.21.15.116:0/1964067972 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f1f500cf180 con 0x7f1f581022f0 2024-03-19T01:28:41.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.244+0000 7f1f5d878640 1 --2- 172.21.15.116:0/1964067972 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1f440a2800 0x7f1f440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:41.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.245+0000 7f1f5d878640 1 --2- 172.21.15.116:0/1964067972 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1f440a2800 0x7f1f440a4cc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1f540058c0 tx=0x7f1f54005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:41.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.252+0000 7f1f4effd640 1 -- 172.21.15.116:0/1964067972 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1f5009c200 con 0x7f1f581022f0 2024-03-19T01:28:41.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-mon[67178]: pgmap v833: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2024-03-19T01:28:41.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/2938465789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:28:41.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1514922290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:28:41.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:41 smithi116 ceph-mon[62701]: pgmap v833: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2024-03-19T01:28:41.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:41 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/2938465789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:28:41.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:41 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1514922290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:28:41.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.456+0000 7f1f5fadc640 1 -- 172.21.15.116:0/1964067972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f5802d1f0 con 0x7f1f581022f0 2024-03-19T01:28:41.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.458+0000 7f1f4effd640 1 -- 172.21.15.116:0/1964067972 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) ==== 74+0+22804 (secure 0 0 0) 0x7f1f5009bb00 con 0x7f1f581022f0 2024-03-19T01:28:41.459 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-03-19T01:28:41.459 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"fsid":"35912c14-e58c-11ee-95c9-87774f69a715","created":"2024-03-19T01:02:30.471059+0000","modified":"2024-03-19T01:11:28.144841+0000","last_up_change":"2024-03-19T01:09:43.111191+0000","last_in_change":"2024-03-19T01:09:22.775870+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-19T01:05:53.301933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-19T01:10:31.028304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-19T01:10:31.704469+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"95728d5b-46c8-41c0-830a-7602d8b251fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6803","nonce":1461395811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6805","nonce":1461395811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6809","nonce":1461395811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":1461395811},{"type":"v1","addr":"172.21.15.116:6807","nonce":1461395811}]},"public_addr":"172.21.15.116:6803/1461395811","cluster_addr":"172.21.15.116:6805/1461395811","heartbeat_back_addr":"172.21.15.116:6809/1461395811","heartbeat_front_addr":"172.21.15.116:6807/1461395811","state":["exists","up"]},{"osd":1,"uuid":"6ab4837e-b52f-4696-9361-2395e835e740","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6801","nonce":2727091908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6803","nonce":2727091908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6807","nonce":2727091908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":2727091908},{"type":"v1","addr":"172.21.15.156:6805","nonce":2727091908}]},"public_addr":"172.21.15.156:6801/2727091908","cluster_addr":"172.21.15.156:6803/2727091908","heartbeat_back_addr":"172.21.15.156:6807/2727091908","heartbeat_front_addr":"172.21.15.156:6805/2727091908","state":["exists","up"]},{"osd":2,"uuid":"ef1c978e-270f-4b74-844d-2fe4c09f2ef5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6801","nonce":2120908758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6803","nonce":2120908758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6807","nonce":2120908758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":2120908758},{"type":"v1","addr":"172.21.15.162:6805","nonce":2120908758}]},"public_addr":"172.21.15.162:6801/2120908758","cluster_addr":"172.21.15.162:6803/2120908758","heartbeat_back_addr":"172.21.15.162:6807/2120908758","heartbeat_front_addr":"172.21.15.162:6805/2120908758","state":["exists","up"]},{"osd":3,"uuid":"944f6747-a95f-49ae-8b3b-d7c48dc15059","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6811","nonce":2026775607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6813","nonce":2026775607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6817","nonce":2026775607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":2026775607},{"type":"v1","addr":"172.21.15.116:6815","nonce":2026775607}]},"public_addr":"172.21.15.116:6811/2026775607","cluster_addr":"172.21.15.116:6813/2026775607","heartbeat_back_addr":"172.21.15.116:6817/2026775607","heartbeat_front_addr":"172.21.15.116:6815/2026775607","state":["exists","up"]},{"osd":4,"uuid":"1bd47f0a-a0d1-4402-8de0-940cd4f6a69b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6809","nonce":1262076382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6811","nonce":1262076382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6815","nonce":1262076382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":1262076382},{"type":"v1","addr":"172.21.15.156:6813","nonce":1262076382}]},"public_addr":"172.21.15.156:6809/1262076382","cluster_addr":"172.21.15.156:6811/1262076382","heartbeat_back_addr":"172.21.15.156:6815/1262076382","heartbeat_front_addr":"172.21.15.156:6813/1262076382","state":["exists","up"]},{"osd":5,"uuid":"457a29a6-6e9d-406a-95f6-d34e1881e6ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6809","nonce":2786247633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6811","nonce":2786247633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6815","nonce":2786247633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":2786247633},{"type":"v1","addr":"172.21.15.162:6813","nonce":2786247633}]},"public_addr":"172.21.15.162:6809/2786247633","cluster_addr":"172.21.15.162:6811/2786247633","heartbeat_back_addr":"172.21.15.162:6815/2786247633","heartbeat_front_addr":"172.21.15.162:6813/2786247633","state":["exists","up"]},{"osd":6,"uuid":"7db79fde-7c61-4d3e-adb5-78b906eb3173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6819","nonce":2037449241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6821","nonce":2037449241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6825","nonce":2037449241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":2037449241},{"type":"v1","addr":"172.21.15.116:6823","nonce":2037449241}]},"public_addr":"172.21.15.116:6819/2037449241","cluster_addr":"172.21.15.116:6821/2037449241","heartbeat_back_addr":"172.21.15.116:6825/2037449241","heartbeat_front_addr":"172.21.15.116:6823/2037449241","state":["exists","up"]},{"osd":7,"uuid":"8e27814d-4a85-4dcd-8640-a09b2e55df23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6817","nonce":3540949514}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6819","nonce":3540949514}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6823","nonce":3540949514}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":3540949514},{"type":"v1","addr":"172.21.15.156:6821","nonce":3540949514}]},"public_addr":"172.21.15.156:6817/3540949514","cluster_addr":"172.21.15.156:6819/3540949514","heartbeat_back_addr":"172.21.15.156:6823/3540949514","heartbeat_front_addr":"172.21.15.156:6821/3540949514","state":["exists","up"]},{"osd":8,"uuid":"9660b3b0-ed99-4050-a339-1beea2450c79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6817","nonce":1669615340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6819","nonce":1669615340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6823","nonce":1669615340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":1669615340},{"type":"v1","addr":"172.21.15.162:6821","nonce":1669615340}]},"public_addr":"172.21.15.162:6817/1669615340","cluster_addr":"172.21.15.162:6819/1669615340","heartbeat_back_addr":"172.21.15.162:6823/1669615340","heartbeat_front_addr":"172.21.15.162:6821/1669615340","state":["exists","up"]},{"osd":9,"uuid":"59d2e3c8-0173-48eb-8eed-303c1dc31822","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6827","nonce":96304116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6829","nonce":96304116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6833","nonce":96304116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":96304116},{"type":"v1","addr":"172.21.15.116:6831","nonce":96304116}]},"public_addr":"172.21.15.116:6827/96304116","cluster_addr":"172.21.15.116:6829/96304116","heartbeat_back_addr":"172.21.15.116:6833/96304116","heartbeat_front_addr":"172.21.15.116:6831/96304116","state":["exists","up"]},{"osd":10,"uuid":"7abd5362-d32e-48a0-9de5-d23ea5ba4391","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6825","nonce":3316972631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6827","nonce":3316972631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6831","nonce":3316972631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":3316972631},{"type":"v1","addr":"172.21.15.156:6829","nonce":3316972631}]},"public_addr":"172.21.15.156:6825/3316972631","cluster_addr":"172.21.15.156:6827/3316972631","heartbeat_back_addr":"172.21.15.156:6831/3316972631","heartbeat_front_addr":"172.21.15.156:6829/3316972631","state":["exists","up"]},{"osd":11,"uuid":"b4fafaa0-173c-4b54-b5b1-10b925873fa0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6825","nonce":246086941}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6827","nonce":246086941}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6831","nonce":246086941}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":246086941},{"type":"v1","addr":"172.21.15.162:6829","nonce":246086941}]},"public_addr":"172.21.15.162:6825/246086941","cluster_addr":"172.21.15.162:6827/246086941","heartbeat_back_addr":"172.21.15.162:6831/246086941","heartbeat_front_addr":"172.21.15.162:6829/246086941","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:03.223167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:26.705783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:05:50.626554+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:15.351946+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:06:40.674442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:06.049961+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:31.890323+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:07:57.610486+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:22.408179+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:08:49.317772+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:15.210649+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-19T01:09:41.194687+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":1}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.c","mappings":[{"from":5,"to":6}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6800/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1073894644":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6800/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/3553505800":"2024-03-20T01:03:00.584768+0000","172.21.15.116:6801/1107242393":"2024-03-20T01:03:26.974893+0000","172.21.15.116:6801/3676795487":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2950253911":"2024-03-20T01:03:26.974893+0000","172.21.15.116:0/2689675916":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/1322211514":"2024-03-20T01:03:00.584768+0000","172.21.15.116:0/2178441515":"2024-03-20T01:03:26.974893+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-19T01:28:41.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.460+0000 7f1f5fadc640 1 -- 172.21.15.116:0/1964067972 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1f440a2800 msgr2=0x7f1f440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.460+0000 7f1f5fadc640 1 --2- 172.21.15.116:0/1964067972 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1f440a2800 0x7f1f440a4cc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1f540058c0 tx=0x7f1f54005f90 comp rx=0 tx=0).stop 2024-03-19T01:28:41.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.460+0000 7f1f5fadc640 1 -- 172.21.15.116:0/1964067972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f581022f0 msgr2=0x7f1f5819eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.460+0000 7f1f5fadc640 1 --2- 172.21.15.116:0/1964067972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f581022f0 0x7f1f5819eeb0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1f50009fd0 tx=0x7f1f5000ef10 comp rx=0 tx=0).stop 2024-03-19T01:28:41.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.460+0000 7f1f5fadc640 1 -- 172.21.15.116:0/1964067972 shutdown_connections 2024-03-19T01:28:41.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.460+0000 7f1f5fadc640 1 --2- 172.21.15.116:0/1964067972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f581022f0 0x7f1f5819eeb0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.460+0000 7f1f5fadc640 1 --2- 172.21.15.116:0/1964067972 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f1f58101930 0x7f1f5819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.460+0000 7f1f5fadc640 1 --2- 172.21.15.116:0/1964067972 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f1f440a2800 0x7f1f440a4cc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.460+0000 7f1f5fadc640 1 --2- 172.21.15.116:0/1964067972 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f58100740 0x7f1f58069510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.460+0000 7f1f5fadc640 1 -- 172.21.15.116:0/1964067972 >> 172.21.15.116:0/1964067972 conn(0x7f1f580fbcb0 msgr2=0x7f1f58104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:41.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.460+0000 7f1f5fadc640 1 -- 172.21.15.116:0/1964067972 shutdown_connections 2024-03-19T01:28:41.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.460+0000 7f1f5fadc640 1 -- 172.21.15.116:0/1964067972 wait complete. 2024-03-19T01:28:41.472 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-19T01:28:41.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-mon[66799]: pgmap v833: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2024-03-19T01:28:41.473 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/2938465789' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:28:41.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1514922290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-19T01:28:41.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.576+0000 7f8def5a9640 1 -- 172.21.15.116:0/3344508118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8de810de70 msgr2=0x7f8de8110260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.576+0000 7f8def5a9640 1 --2- 172.21.15.116:0/3344508118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8de810de70 0x7f8de8110260 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009cd0 tx=0x7f8de002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.576+0000 7f8def5a9640 1 -- 172.21.15.116:0/3344508118 shutdown_connections 2024-03-19T01:28:41.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.576+0000 7f8def5a9640 1 --2- 172.21.15.116:0/3344508118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8de810de70 0x7f8de8110260 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.576+0000 7f8def5a9640 1 --2- 172.21.15.116:0/3344508118 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8de8074280 0x7f8de8074700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.576+0000 7f8def5a9640 1 --2- 172.21.15.116:0/3344508118 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8de8075860 0x7f8de8073cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.576+0000 7f8def5a9640 1 -- 172.21.15.116:0/3344508118 >> 172.21.15.116:0/3344508118 conn(0x7f8de80fce30 msgr2=0x7f8de80ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:41.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.576+0000 7f8def5a9640 1 -- 172.21.15.116:0/3344508118 shutdown_connections 2024-03-19T01:28:41.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.576+0000 7f8def5a9640 1 -- 172.21.15.116:0/3344508118 wait complete. 2024-03-19T01:28:41.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8def5a9640 1 Processor -- start 2024-03-19T01:28:41.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8def5a9640 1 -- start start 2024-03-19T01:28:41.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8def5a9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8de8074280 0x7f8de819e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:41.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8def5a9640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8de8075860 0x7f8de819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:41.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8def5a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8de810de70 0x7f8de81a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:41.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8def5a9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8de8113a80 con 0x7f8de810de70 2024-03-19T01:28:41.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8def5a9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8de8113900 con 0x7f8de8074280 2024-03-19T01:28:41.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8def5a9640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f8de8113c00 con 0x7f8de8075860 2024-03-19T01:28:41.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8dedb46640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8de810de70 0x7f8de81a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:41.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8ded345640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8de8074280 0x7f8de819e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:41.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8dedb46640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8de810de70 0x7f8de81a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48970/0 (socket says 172.21.15.116:48970) 2024-03-19T01:28:41.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8dedb46640 1 -- 172.21.15.116:0/1088387026 learned_addr learned my addr 172.21.15.116:0/1088387026 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:41.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8decb44640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8de8075860 0x7f8de819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:41.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8dedb46640 1 -- 172.21.15.116:0/1088387026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8de8075860 msgr2=0x7f8de819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8dedb46640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8de8075860 0x7f8de819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8dedb46640 1 -- 172.21.15.116:0/1088387026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8de8074280 msgr2=0x7f8de819e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8dedb46640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8de8074280 0x7f8de819e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8dedb46640 1 -- 172.21.15.116:0/1088387026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de81a5d30 con 0x7f8de810de70 2024-03-19T01:28:41.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.577+0000 7f8ded345640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8de8074280 0x7f8de819e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-19T01:28:41.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.578+0000 7f8dedb46640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8de810de70 0x7f8de81a56c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009ca0 tx=0x7f8de0032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:41.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.578+0000 7f8dde7fc640 1 -- 172.21.15.116:0/1088387026 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0041020 con 0x7f8de810de70 2024-03-19T01:28:41.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.578+0000 7f8def5a9640 1 -- 172.21.15.116:0/1088387026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de81a5fc0 con 0x7f8de810de70 2024-03-19T01:28:41.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.578+0000 7f8dde7fc640 1 -- 172.21.15.116:0/1088387026 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8de0034cd0 con 0x7f8de810de70 2024-03-19T01:28:41.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.578+0000 7f8dde7fc640 1 -- 172.21.15.116:0/1088387026 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0039040 con 0x7f8de810de70 2024-03-19T01:28:41.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.578+0000 7f8def5a9640 1 -- 172.21.15.116:0/1088387026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de81a6480 con 0x7f8de810de70 2024-03-19T01:28:41.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.579+0000 7f8def5a9640 1 -- 172.21.15.116:0/1088387026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de8104040 con 0x7f8de810de70 2024-03-19T01:28:41.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.580+0000 7f8dde7fc640 1 -- 172.21.15.116:0/1088387026 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f8de0044020 con 0x7f8de810de70 2024-03-19T01:28:41.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.580+0000 7f8dde7fc640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8dd40a2800 0x7f8dd40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:41.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.581+0000 7f8dde7fc640 1 -- 172.21.15.116:0/1088387026 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8269+0+0 (secure 0 0 0) 0x7f8de00f0c90 con 0x7f8de810de70 2024-03-19T01:28:41.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.581+0000 7f8ded345640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8dd40a2800 0x7f8dd40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:41.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.581+0000 7f8ded345640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8dd40a2800 0x7f8dd40a4cc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009fc0 tx=0x7f8de4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:41.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.584+0000 7f8dde7fc640 1 -- 172.21.15.116:0/1088387026 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8de004b020 con 0x7f8de810de70 2024-03-19T01:28:41.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.804+0000 7f8def5a9640 1 -- 172.21.15.116:0/1088387026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f8de802d3f0 con 0x7f8de810de70 2024-03-19T01:28:41.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.806+0000 7f8dde7fc640 1 -- 172.21.15.116:0/1088387026 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v57) ==== 161+0+0 (secure 0 0 0) 0x7f8de00bdda0 con 0x7f8de810de70 2024-03-19T01:28:41.806 INFO:teuthology.orchestra.run.smithi116.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-19T01:28:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.809+0000 7f8def5a9640 1 -- 172.21.15.116:0/1088387026 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8dd40a2800 msgr2=0x7f8dd40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.809+0000 7f8def5a9640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8dd40a2800 0x7f8dd40a4cc0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009fc0 tx=0x7f8de4005f90 comp rx=0 tx=0).stop 2024-03-19T01:28:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.809+0000 7f8def5a9640 1 -- 172.21.15.116:0/1088387026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8de810de70 msgr2=0x7f8de81a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.809+0000 7f8def5a9640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8de810de70 0x7f8de81a56c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009ca0 tx=0x7f8de0032f20 comp rx=0 tx=0).stop 2024-03-19T01:28:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.809+0000 7f8def5a9640 1 -- 172.21.15.116:0/1088387026 shutdown_connections 2024-03-19T01:28:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.809+0000 7f8def5a9640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8de810de70 0x7f8de81a56c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.809+0000 7f8def5a9640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f8de8075860 0x7f8de819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.809+0000 7f8def5a9640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f8dd40a2800 0x7f8dd40a4cc0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.809+0000 7f8def5a9640 1 --2- 172.21.15.116:0/1088387026 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8de8074280 0x7f8de819e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.809+0000 7f8def5a9640 1 -- 172.21.15.116:0/1088387026 >> 172.21.15.116:0/1088387026 conn(0x7f8de80fce30 msgr2=0x7f8de8101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:41.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.809+0000 7f8def5a9640 1 -- 172.21.15.116:0/1088387026 shutdown_connections 2024-03-19T01:28:41.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.809+0000 7f8def5a9640 1 -- 172.21.15.116:0/1088387026 wait complete. 2024-03-19T01:28:41.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-19T01:28:41.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.938+0000 7f5c41110640 1 -- 172.21.15.116:0/2174264827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c1022d0 msgr2=0x7f5c3c108b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.938+0000 7f5c41110640 1 --2- 172.21.15.116:0/2174264827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c1022d0 0x7f5c3c108b60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5c34009ca0 tx=0x7f5c3402c0f0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.938+0000 7f5c41110640 1 -- 172.21.15.116:0/2174264827 shutdown_connections 2024-03-19T01:28:41.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.938+0000 7f5c41110640 1 --2- 172.21.15.116:0/2174264827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c1022d0 0x7f5c3c108b60 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.938+0000 7f5c41110640 1 --2- 172.21.15.116:0/2174264827 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c3c101910 0x7f5c3c101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.938+0000 7f5c41110640 1 --2- 172.21.15.116:0/2174264827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c3c100720 0x7f5c3c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.938+0000 7f5c41110640 1 -- 172.21.15.116:0/2174264827 >> 172.21.15.116:0/2174264827 conn(0x7f5c3c0fbcb0 msgr2=0x7f5c3c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:41.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.938+0000 7f5c41110640 1 -- 172.21.15.116:0/2174264827 shutdown_connections 2024-03-19T01:28:41.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.938+0000 7f5c41110640 1 -- 172.21.15.116:0/2174264827 wait complete. 2024-03-19T01:28:41.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.938+0000 7f5c41110640 1 Processor -- start 2024-03-19T01:28:41.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c41110640 1 -- start start 2024-03-19T01:28:41.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c41110640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100720 0x7f5c3c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:41.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c41110640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c3c101910 0x7f5c3c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:41.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c41110640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c3c1022d0 0x7f5c3c19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:41.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c41110640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3c10d4c0 con 0x7f5c3c100720 2024-03-19T01:28:41.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c41110640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3c10d340 con 0x7f5c3c1022d0 2024-03-19T01:28:41.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c41110640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3c10d640 con 0x7f5c3c101910 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c3ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100720 0x7f5c3c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c3ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100720 0x7f5c3c197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49036/0 (socket says 172.21.15.116:49036) 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c3ad9d640 1 -- 172.21.15.116:0/3407218951 learned_addr learned my addr 172.21.15.116:0/3407218951 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c3a59c640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c3c101910 0x7f5c3c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c3b59e640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c3c1022d0 0x7f5c3c19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c3ad9d640 1 -- 172.21.15.116:0/3407218951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c3c101910 msgr2=0x7f5c3c198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c3ad9d640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c3c101910 0x7f5c3c198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c3ad9d640 1 -- 172.21.15.116:0/3407218951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c3c1022d0 msgr2=0x7f5c3c19f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c3ad9d640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c3c1022d0 0x7f5c3c19f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c3ad9d640 1 -- 172.21.15.116:0/3407218951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3c19f760 con 0x7f5c3c100720 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c3b59e640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c3c1022d0 0x7f5c3c19f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.939+0000 7f5c3ad9d640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100720 0x7f5c3c197b40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000e9c0 tx=0x7f5c3000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.940+0000 7f5c23fff640 1 -- 172.21.15.116:0/3407218951 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3000cd80 con 0x7f5c3c100720 2024-03-19T01:28:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.940+0000 7f5c41110640 1 -- 172.21.15.116:0/3407218951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c3c19fa50 con 0x7f5c3c100720 2024-03-19T01:28:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.940+0000 7f5c23fff640 1 -- 172.21.15.116:0/3407218951 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c300035f0 con 0x7f5c3c100720 2024-03-19T01:28:41.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.940+0000 7f5c41110640 1 -- 172.21.15.116:0/3407218951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c3c19fff0 con 0x7f5c3c100720 2024-03-19T01:28:41.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.940+0000 7f5c23fff640 1 -- 172.21.15.116:0/3407218951 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c30010690 con 0x7f5c3c100720 2024-03-19T01:28:41.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.941+0000 7f5c41110640 1 -- 172.21.15.116:0/3407218951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c3c107a40 con 0x7f5c3c100720 2024-03-19T01:28:41.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.942+0000 7f5c23fff640 1 -- 172.21.15.116:0/3407218951 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f5c30004020 con 0x7f5c3c100720 2024-03-19T01:28:41.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.942+0000 7f5c23fff640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5c240a2800 0x7f5c240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:41.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.943+0000 7f5c3a59c640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5c240a2800 0x7f5c240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:41.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.943+0000 7f5c23fff640 1 -- 172.21.15.116:0/3407218951 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8699+0+0 (secure 0 0 0) 0x7f5c300cf0b0 con 0x7f5c3c100720 2024-03-19T01:28:41.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.943+0000 7f5c3a59c640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5c240a2800 0x7f5c240a4cc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c199050 tx=0x7f5c2c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:41.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:41.947+0000 7f5c23fff640 1 -- 172.21.15.116:0/3407218951 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5c3009bf80 con 0x7f5c3c100720 2024-03-19T01:28:42.120 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-a[62697]: 2024-03-19T01:28:41.804+0000 7f395d7eb700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T01:28:42.121 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:28:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-a[104332]: ignoring --setuser ceph since I am not root 2024-03-19T01:28:42.121 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:28:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-a[104332]: ignoring --setgroup ceph since I am not root 2024-03-19T01:28:42.121 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:28:41 smithi116 ceph-mds[104338]: -- 172.21.15.116:0/527909772 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557a95c3da40 con 0x557a95c10c00 2024-03-19T01:28:42.121 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:28:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-a[104332]: starting mds.a at 2024-03-19T01:28:42.121 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:28:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-g[103976]: ignoring --setuser ceph since I am not root 2024-03-19T01:28:42.121 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:28:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-g[103976]: ignoring --setgroup ceph since I am not root 2024-03-19T01:28:42.122 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:28:41 smithi116 ceph-mds[103980]: -- 172.21.15.116:0/1802983796 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ceaef99a40 con 0x55ceaef6d000 2024-03-19T01:28:42.122 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:28:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-g[103976]: starting mds.g at 2024-03-19T01:28:42.122 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-l[97350]: ignoring --setuser ceph since I am not root 2024-03-19T01:28:42.122 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-l[97350]: ignoring --setgroup ceph since I am not root 2024-03-19T01:28:42.123 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-mds[97354]: -- 172.21.15.162:0/3901802190 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5632057aba40 con 0x56320577e800 2024-03-19T01:28:42.123 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-l[97350]: starting mds.l at 2024-03-19T01:28:42.123 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-e[96690]: ignoring --setuser ceph since I am not root 2024-03-19T01:28:42.123 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-e[96690]: ignoring --setgroup ceph since I am not root 2024-03-19T01:28:42.123 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-mds[96694]: -- 172.21.15.156:0/3773958165 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5654d2c8ba40 con 0x5654d2c5ec00 2024-03-19T01:28:42.123 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-e[96690]: starting mds.e at 2024-03-19T01:28:42.123 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-c[97982]: ignoring --setuser ceph since I am not root 2024-03-19T01:28:42.123 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-c[97982]: ignoring --setgroup ceph since I am not root 2024-03-19T01:28:42.124 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-mds[97986]: -- 172.21.15.162:0/2145189681 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f6b2667a40 con 0x55f6b263ac00 2024-03-19T01:28:42.124 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-c[97982]: starting mds.c at 2024-03-19T01:28:42.124 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b[97322]: ignoring --setuser ceph since I am not root 2024-03-19T01:28:42.124 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b[97322]: ignoring --setgroup ceph since I am not root 2024-03-19T01:28:42.124 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-mds[97326]: -- 172.21.15.156:0/1694773975 <== mon.0 v2:172.21.15.116:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557043ebda40 con 0x557043e90c00 2024-03-19T01:28:42.124 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b[97322]: starting mds.b at 2024-03-19T01:28:42.124 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-i[98298]: ignoring --setuser ceph since I am not root 2024-03-19T01:28:42.125 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-i[98298]: ignoring --setgroup ceph since I am not root 2024-03-19T01:28:42.125 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-mds[98302]: -- 172.21.15.162:0/3535201434 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5582ac2cfa40 con 0x5582ac2a2800 2024-03-19T01:28:42.125 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-i[98298]: starting mds.i at 2024-03-19T01:28:42.125 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-k[96373]: ignoring --setuser ceph since I am not root 2024-03-19T01:28:42.125 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-k[96373]: ignoring --setgroup ceph since I am not root 2024-03-19T01:28:42.125 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-mds[96377]: -- 172.21.15.156:0/2828391210 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d58a0cda40 con 0x55d58a0a1000 2024-03-19T01:28:42.125 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-k[96373]: starting mds.k at 2024-03-19T01:28:42.125 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-f[97676]: ignoring --setuser ceph since I am not root 2024-03-19T01:28:42.125 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-f[97676]: ignoring --setgroup ceph since I am not root 2024-03-19T01:28:42.126 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-mds[97680]: -- 172.21.15.162:0/2644290444 <== mon.1 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56341fb95a40 con 0x56341fb68c00 2024-03-19T01:28:42.126 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:28:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-f[97676]: starting mds.f at 2024-03-19T01:28:42.126 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-h[97011]: ignoring --setuser ceph since I am not root 2024-03-19T01:28:42.126 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-h[97011]: ignoring --setgroup ceph since I am not root 2024-03-19T01:28:42.126 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-mds[97015]: -- 172.21.15.156:0/3673645047 <== mon.2 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5619959d9a40 con 0x5619959ad000 2024-03-19T01:28:42.126 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:28:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-h[97011]: starting mds.h at 2024-03-19T01:28:42.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.164+0000 7f5c41110640 1 -- 172.21.15.116:0/3407218951 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f5c3c02d1f0 con 0x7f5c3c100720 2024-03-19T01:28:42.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:42 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1964067972' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:28:42.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:42 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1088387026' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T01:28:42.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:42 smithi162 ceph-mon[67178]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-19T01:28:42.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:42 smithi162 ceph-mon[67178]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T01:28:42.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:42 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/1088387026' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-19T01:28:42.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:42 smithi162 ceph-mon[67178]: osdmap e88: 12 total, 12 up, 12 in 2024-03-19T01:28:42.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:42 smithi162 ceph-mon[67178]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-19T01:28:42.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:42 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1964067972' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:28:42.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:42 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1088387026' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T01:28:42.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:42 smithi116 ceph-mon[62701]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-19T01:28:42.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:42 smithi116 ceph-mon[62701]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T01:28:42.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:42 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/1088387026' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-19T01:28:42.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:42 smithi116 ceph-mon[62701]: osdmap e88: 12 total, 12 up, 12 in 2024-03-19T01:28:42.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:42 smithi116 ceph-mon[62701]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-19T01:28:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:42 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1964067972' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-19T01:28:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:42 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1088387026' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-19T01:28:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:42 smithi156 ceph-mon[66799]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-19T01:28:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:42 smithi156 ceph-mon[66799]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-19T01:28:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:42 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/1088387026' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-19T01:28:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:42 smithi156 ceph-mon[66799]: osdmap e88: 12 total, 12 up, 12 in 2024-03-19T01:28:42.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:42 smithi156 ceph-mon[66799]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-19T01:28:42.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.807+0000 7f5c23fff640 1 -- 172.21.15.116:0/3407218951 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v58) ==== 104+0+0 (secure 0 0 0) 0x7f5c3c02d1f0 con 0x7f5c3c100720 2024-03-19T01:28:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.810+0000 7f5c41110640 1 -- 172.21.15.116:0/3407218951 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5c240a2800 msgr2=0x7f5c240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.810+0000 7f5c41110640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5c240a2800 0x7f5c240a4cc0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c199050 tx=0x7f5c2c005f90 comp rx=0 tx=0).stop 2024-03-19T01:28:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.810+0000 7f5c41110640 1 -- 172.21.15.116:0/3407218951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100720 msgr2=0x7f5c3c197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.810+0000 7f5c41110640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100720 0x7f5c3c197b40 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000e9c0 tx=0x7f5c3000ee80 comp rx=0 tx=0).stop 2024-03-19T01:28:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.811+0000 7f5c41110640 1 -- 172.21.15.116:0/3407218951 shutdown_connections 2024-03-19T01:28:42.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.811+0000 7f5c41110640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f5c240a2800 0x7f5c240a4cc0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.811+0000 7f5c41110640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5c3c1022d0 0x7f5c3c19f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.811+0000 7f5c41110640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f5c3c101910 0x7f5c3c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.811+0000 7f5c41110640 1 --2- 172.21.15.116:0/3407218951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c3c100720 0x7f5c3c197b40 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.811+0000 7f5c41110640 1 -- 172.21.15.116:0/3407218951 >> 172.21.15.116:0/3407218951 conn(0x7f5c3c0fbcb0 msgr2=0x7f5c3c104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.811+0000 7f5c41110640 1 -- 172.21.15.116:0/3407218951 shutdown_connections 2024-03-19T01:28:42.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.811+0000 7f5c41110640 1 -- 172.21.15.116:0/3407218951 wait complete. 2024-03-19T01:28:42.828 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-19T01:28:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.938+0000 7f4df528a640 1 -- 172.21.15.116:0/1295184756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df00ffe90 msgr2=0x7f4df01002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:42.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.938+0000 7f4df528a640 1 --2- 172.21.15.116:0/1295184756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df00ffe90 0x7f4df01002b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009ca0 tx=0x7f4de002c0f0 comp rx=0 tx=0).stop 2024-03-19T01:28:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 -- 172.21.15.116:0/1295184756 shutdown_connections 2024-03-19T01:28:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 --2- 172.21.15.116:0/1295184756 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4df0101a30 0x7f4df0108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 --2- 172.21.15.116:0/1295184756 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4df0101070 0x7f4df01014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 --2- 172.21.15.116:0/1295184756 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df00ffe90 0x7f4df01002b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 -- 172.21.15.116:0/1295184756 >> 172.21.15.116:0/1295184756 conn(0x7f4df00fb450 msgr2=0x7f4df00fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 -- 172.21.15.116:0/1295184756 shutdown_connections 2024-03-19T01:28:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 -- 172.21.15.116:0/1295184756 wait complete. 2024-03-19T01:28:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 Processor -- start 2024-03-19T01:28:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 -- start start 2024-03-19T01:28:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df00ffe90 0x7f4df0197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4df0101070 0x7f4df0198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4df0101a30 0x7f4df019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:42.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4df010c410 con 0x7f4df00ffe90 2024-03-19T01:28:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4df010c290 con 0x7f4df0101a30 2024-03-19T01:28:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.939+0000 7f4df528a640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7f4df010c590 con 0x7f4df0101070 2024-03-19T01:28:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4deeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df00ffe90 0x7f4df0197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4deeffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df00ffe90 0x7f4df0197c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49054/0 (socket says 172.21.15.116:49054) 2024-03-19T01:28:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4deeffd640 1 -- 172.21.15.116:0/651594371 learned_addr learned my addr 172.21.15.116:0/651594371 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4def7fe640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4df0101a30 0x7f4df019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4dee7fc640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4df0101070 0x7f4df0198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4deeffd640 1 -- 172.21.15.116:0/651594371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4df0101070 msgr2=0x7f4df0198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4deeffd640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4df0101070 0x7f4df0198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4deeffd640 1 -- 172.21.15.116:0/651594371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4df0101a30 msgr2=0x7f4df019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4deeffd640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4df0101a30 0x7f4df019f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4deeffd640 1 -- 172.21.15.116:0/651594371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df019f810 con 0x7f4df00ffe90 2024-03-19T01:28:42.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4dee7fc640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4df0101070 0x7f4df0198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:28:42.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4def7fe640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4df0101a30 0x7f4df019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-19T01:28:42.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4deeffd640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df00ffe90 0x7f4df0197c40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4de00324c0 tx=0x7f4de0032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:42.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4dd7fff640 1 -- 172.21.15.116:0/651594371 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de0039040 con 0x7f4df00ffe90 2024-03-19T01:28:42.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4df528a640 1 -- 172.21.15.116:0/651594371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df019faa0 con 0x7f4df00ffe90 2024-03-19T01:28:42.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4dd7fff640 1 -- 172.21.15.116:0/651594371 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4de0044070 con 0x7f4df00ffe90 2024-03-19T01:28:42.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4dd7fff640 1 -- 172.21.15.116:0/651594371 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de00042a0 con 0x7f4df00ffe90 2024-03-19T01:28:42.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.940+0000 7f4df528a640 1 -- 172.21.15.116:0/651594371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df019ff90 con 0x7f4df00ffe90 2024-03-19T01:28:42.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.941+0000 7f4df528a640 1 -- 172.21.15.116:0/651594371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df01a01c0 con 0x7f4df00ffe90 2024-03-19T01:28:42.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.942+0000 7f4dd7fff640 1 -- 172.21.15.116:0/651594371 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7f4de0003970 con 0x7f4df00ffe90 2024-03-19T01:28:42.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.943+0000 7f4dd7fff640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4dd80a2800 0x7f4dd80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:42.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.943+0000 7f4dee7fc640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4dd80a2800 0x7f4dd80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:42.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.943+0000 7f4dd7fff640 1 -- 172.21.15.116:0/651594371 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8699+0+0 (secure 0 0 0) 0x7f4de00f1140 con 0x7f4df00ffe90 2024-03-19T01:28:42.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.943+0000 7f4dee7fc640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4dd80a2800 0x7f4dd80a4cc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4df0199100 tx=0x7f4de4005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:42.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:42.947+0000 7f4dd7fff640 1 -- 172.21.15.116:0/651594371 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4de003e3a0 con 0x7f4df00ffe90 2024-03-19T01:28:43.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.123+0000 7f4df528a640 1 -- 172.21.15.116:0/651594371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4df01a08e0 con 0x7f4df00ffe90 2024-03-19T01:28:43.127 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: pgmap v834: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-03-19T01:28:43.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: pgmap v834: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-03-19T01:28:43.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3407218951' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T01:28:43.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-19T01:28:43.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T01:28:43.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: Cluster is now healthy 2024-03-19T01:28:43.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3407218951' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-19T01:28:43.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.162:6837/3542103096,v1:172.21.15.162:6839/3542103096] up:boot 2024-03-19T01:28:43.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.116:6838/1669336799,v1:172.21.15.116:6839/1669336799] up:boot 2024-03-19T01:28:43.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.156:6832/507692267,v1:172.21.15.156:6833/507692267] up:boot 2024-03-19T01:28:43.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.162:6832/633717121,v1:172.21.15.162:6834/633717121] up:boot 2024-03-19T01:28:43.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.162:6833/3163588832,v1:172.21.15.162:6835/3163588832] up:boot 2024-03-19T01:28:43.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.156:6836/856443881,v1:172.21.15.156:6837/856443881] up:boot 2024-03-19T01:28:43.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.116:6840/3830019959,v1:172.21.15.116:6841/3830019959] up:boot 2024-03-19T01:28:43.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.162:6836/3016477558,v1:172.21.15.162:6838/3016477558] up:boot 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.156:6834/4040107248,v1:172.21.15.156:6835/4040107248] up:boot 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: mds.? [v2:172.21.15.156:6838/249051508,v1:172.21.15.156:6839/249051508] up:boot 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: fsmap 12 up:standby 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T01:28:43.401 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:43 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/651594371' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T01:28:43.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3407218951' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T01:28:43.405 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: Cluster is now healthy 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3407218951' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.162:6837/3542103096,v1:172.21.15.162:6839/3542103096] up:boot 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.116:6838/1669336799,v1:172.21.15.116:6839/1669336799] up:boot 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.156:6832/507692267,v1:172.21.15.156:6833/507692267] up:boot 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.162:6832/633717121,v1:172.21.15.162:6834/633717121] up:boot 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.162:6833/3163588832,v1:172.21.15.162:6835/3163588832] up:boot 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.156:6836/856443881,v1:172.21.15.156:6837/856443881] up:boot 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.116:6840/3830019959,v1:172.21.15.116:6841/3830019959] up:boot 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.162:6836/3016477558,v1:172.21.15.162:6838/3016477558] up:boot 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.156:6834/4040107248,v1:172.21.15.156:6835/4040107248] up:boot 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: mds.? [v2:172.21.15.156:6838/249051508,v1:172.21.15.156:6839/249051508] up:boot 2024-03-19T01:28:43.406 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: fsmap 12 up:standby 2024-03-19T01:28:43.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T01:28:43.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T01:28:43.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T01:28:43.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T01:28:43.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T01:28:43.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T01:28:43.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T01:28:43.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T01:28:43.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T01:28:43.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T01:28:43.407 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:43 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/651594371' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T01:28:43.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: pgmap v834: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 10 KiB/s wr, 14 op/s 2024-03-19T01:28:43.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3407218951' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-19T01:28:43.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: Cluster is now healthy 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3407218951' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.162:6837/3542103096,v1:172.21.15.162:6839/3542103096] up:boot 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.116:6838/1669336799,v1:172.21.15.116:6839/1669336799] up:boot 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.156:6832/507692267,v1:172.21.15.156:6833/507692267] up:boot 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.162:6832/633717121,v1:172.21.15.162:6834/633717121] up:boot 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.162:6833/3163588832,v1:172.21.15.162:6835/3163588832] up:boot 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.156:6836/856443881,v1:172.21.15.156:6837/856443881] up:boot 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.116:6840/3830019959,v1:172.21.15.116:6841/3830019959] up:boot 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.162:6836/3016477558,v1:172.21.15.162:6838/3016477558] up:boot 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.156:6834/4040107248,v1:172.21.15.156:6835/4040107248] up:boot 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: mds.? [v2:172.21.15.156:6838/249051508,v1:172.21.15.156:6839/249051508] up:boot 2024-03-19T01:28:43.475 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: fsmap 12 up:standby 2024-03-19T01:28:43.476 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-19T01:28:43.476 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-19T01:28:43.476 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-19T01:28:43.476 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-19T01:28:43.476 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-19T01:28:43.476 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-19T01:28:43.476 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-19T01:28:43.476 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-19T01:28:43.476 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-19T01:28:43.476 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: from='mgr.14150 172.21.15.116:0/2411172647' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-19T01:28:43.476 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:43 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/651594371' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T01:28:43.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.809+0000 7f4dd7fff640 1 -- 172.21.15.116:0/651594371 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v89) ==== 181+0+0 (secure 0 0 0) 0x7f4de00be0a0 con 0x7f4df00ffe90 2024-03-19T01:28:43.810 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_metadata' removed 2024-03-19T01:28:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.813+0000 7f4df528a640 1 -- 172.21.15.116:0/651594371 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4dd80a2800 msgr2=0x7f4dd80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.813+0000 7f4df528a640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4dd80a2800 0x7f4dd80a4cc0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f4df0199100 tx=0x7f4de4005e50 comp rx=0 tx=0).stop 2024-03-19T01:28:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.813+0000 7f4df528a640 1 -- 172.21.15.116:0/651594371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df00ffe90 msgr2=0x7f4df0197c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.813+0000 7f4df528a640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df00ffe90 0x7f4df0197c40 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4de00324c0 tx=0x7f4de0032f20 comp rx=0 tx=0).stop 2024-03-19T01:28:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.813+0000 7f4df528a640 1 -- 172.21.15.116:0/651594371 shutdown_connections 2024-03-19T01:28:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.813+0000 7f4df528a640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7f4dd80a2800 0x7f4dd80a4cc0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.813+0000 7f4df528a640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4df0101a30 0x7f4df019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.813+0000 7f4df528a640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7f4df0101070 0x7f4df0198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.813+0000 7f4df528a640 1 --2- 172.21.15.116:0/651594371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4df00ffe90 0x7f4df0197c40 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:43.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.813+0000 7f4df528a640 1 -- 172.21.15.116:0/651594371 >> 172.21.15.116:0/651594371 conn(0x7f4df00fb450 msgr2=0x7f4df0108d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:43.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.813+0000 7f4df528a640 1 -- 172.21.15.116:0/651594371 shutdown_connections 2024-03-19T01:28:43.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.813+0000 7f4df528a640 1 -- 172.21.15.116:0/651594371 wait complete. 2024-03-19T01:28:43.828 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-19T01:28:43.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.938+0000 7fb681ec6640 1 -- 172.21.15.116:0/1420128486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67c102300 msgr2=0x7fb67c1046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:43.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.938+0000 7fb681ec6640 1 --2- 172.21.15.116:0/1420128486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67c102300 0x7fb67c1046f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fb670009900 tx=0x7fb67002bc30 comp rx=0 tx=0).stop 2024-03-19T01:28:43.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.938+0000 7fb681ec6640 1 -- 172.21.15.116:0/1420128486 shutdown_connections 2024-03-19T01:28:43.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.938+0000 7fb681ec6640 1 --2- 172.21.15.116:0/1420128486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb67c107560 0x7fb67c109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:43.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.938+0000 7fb681ec6640 1 --2- 172.21.15.116:0/1420128486 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb67c104c30 0x7fb67c107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:43.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.938+0000 7fb681ec6640 1 --2- 172.21.15.116:0/1420128486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67c102300 0x7fb67c1046f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:43.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.938+0000 7fb681ec6640 1 -- 172.21.15.116:0/1420128486 >> 172.21.15.116:0/1420128486 conn(0x7fb67c0fbcb0 msgr2=0x7fb67c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:43.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.938+0000 7fb681ec6640 1 -- 172.21.15.116:0/1420128486 shutdown_connections 2024-03-19T01:28:43.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.938+0000 7fb681ec6640 1 -- 172.21.15.116:0/1420128486 wait complete. 2024-03-19T01:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb681ec6640 1 Processor -- start 2024-03-19T01:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb681ec6640 1 -- start start 2024-03-19T01:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb681ec6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67c102300 0x7fb67c197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb681ec6640 1 --2- >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb67c104c30 0x7fb67c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb681ec6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb67c107560 0x7fb67c19f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb681ec6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c10d4c0 con 0x7fb67c102300 2024-03-19T01:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb681ec6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c10d340 con 0x7fb67c107560 2024-03-19T01:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb681ec6640 1 -- --> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c10d640 con 0x7fb67c104c30 2024-03-19T01:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb67b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67c102300 0x7fb67c197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb67b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67c102300 0x7fb67c197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49066/0 (socket says 172.21.15.116:49066) 2024-03-19T01:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb67b7fe640 1 -- 172.21.15.116:0/3530248861 learned_addr learned my addr 172.21.15.116:0/3530248861 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-03-19T01:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb67affd640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb67c104c30 0x7fb67c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:43.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb67bfff640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb67c107560 0x7fb67c19f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb67b7fe640 1 -- 172.21.15.116:0/3530248861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb67c104c30 msgr2=0x7fb67c1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb67b7fe640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb67c104c30 0x7fb67c1980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb67b7fe640 1 -- 172.21.15.116:0/3530248861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb67c107560 msgr2=0x7fb67c19f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb67b7fe640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb67c107560 0x7fb67c19f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb67b7fe640 1 -- 172.21.15.116:0/3530248861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67c19f7b0 con 0x7fb67c102300 2024-03-19T01:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb67bfff640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb67c107560 0x7fb67c19f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-19T01:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.939+0000 7fb67b7fe640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67c102300 0x7fb67c197b90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb670009230 tx=0x7fb670005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.940+0000 7fb678ff9640 1 -- 172.21.15.116:0/3530248861 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670041070 con 0x7fb67c102300 2024-03-19T01:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.940+0000 7fb678ff9640 1 -- 172.21.15.116:0/3530248861 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6700054e0 con 0x7fb67c102300 2024-03-19T01:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.940+0000 7fb681ec6640 1 -- 172.21.15.116:0/3530248861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67c19fa40 con 0x7fb67c102300 2024-03-19T01:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.940+0000 7fb678ff9640 1 -- 172.21.15.116:0/3530248861 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670034440 con 0x7fb67c102300 2024-03-19T01:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.940+0000 7fb681ec6640 1 -- 172.21.15.116:0/3530248861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb67c19ff30 con 0x7fb67c102300 2024-03-19T01:28:43.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.941+0000 7fb681ec6640 1 -- 172.21.15.116:0/3530248861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb67c0fd7a0 con 0x7fb67c102300 2024-03-19T01:28:43.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.942+0000 7fb678ff9640 1 -- 172.21.15.116:0/3530248861 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 18) ==== 137291+0+0 (secure 0 0 0) 0x7fb670044070 con 0x7fb67c102300 2024-03-19T01:28:43.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.942+0000 7fb678ff9640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6640a2800 0x7fb6640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-19T01:28:43.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.942+0000 7fb678ff9640 1 -- 172.21.15.116:0/3530248861 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8145+0+0 (secure 0 0 0) 0x7fb6700f0780 con 0x7fb67c102300 2024-03-19T01:28:43.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.942+0000 7fb67affd640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6640a2800 0x7fb6640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-19T01:28:43.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.943+0000 7fb67affd640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6640a2800 0x7fb6640a4cc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb66c005fd0 tx=0x7fb66c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-19T01:28:43.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:43.947+0000 7fb678ff9640 1 -- 172.21.15.116:0/3530248861 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb6700bd8c0 con 0x7fb67c102300 2024-03-19T01:28:44.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.120+0000 7fb681ec6640 1 -- 172.21.15.116:0/3530248861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb67c02d1f0 con 0x7fb67c102300 2024-03-19T01:28:44.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.810+0000 7fb678ff9640 1 -- 172.21.15.116:0/3530248861 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v90) ==== 169+0+0 (secure 0 0 0) 0x7fb6700bd1c0 con 0x7fb67c102300 2024-03-19T01:28:44.811 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'cephfs_data' removed 2024-03-19T01:28:44.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.814+0000 7fb681ec6640 1 -- 172.21.15.116:0/3530248861 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6640a2800 msgr2=0x7fb6640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:44.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.814+0000 7fb681ec6640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6640a2800 0x7fb6640a4cc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb66c005fd0 tx=0x7fb66c005850 comp rx=0 tx=0).stop 2024-03-19T01:28:44.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.814+0000 7fb681ec6640 1 -- 172.21.15.116:0/3530248861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67c102300 msgr2=0x7fb67c197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-19T01:28:44.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.814+0000 7fb681ec6640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67c102300 0x7fb67c197b90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fb670009230 tx=0x7fb670005dd0 comp rx=0 tx=0).stop 2024-03-19T01:28:44.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.814+0000 7fb681ec6640 1 -- 172.21.15.116:0/3530248861 shutdown_connections 2024-03-19T01:28:44.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.814+0000 7fb681ec6640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.116:6800/2389438183,v1:172.21.15.116:6801/2389438183] conn(0x7fb6640a2800 0x7fb6640a4cc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:44.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.814+0000 7fb681ec6640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb67c107560 0x7fb67c19f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:44.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.814+0000 7fb681ec6640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] conn(0x7fb67c104c30 0x7fb67c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:44.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.814+0000 7fb681ec6640 1 --2- 172.21.15.116:0/3530248861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67c102300 0x7fb67c197b90 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-19T01:28:44.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.814+0000 7fb681ec6640 1 -- 172.21.15.116:0/3530248861 >> 172.21.15.116:0/3530248861 conn(0x7fb67c0fbcb0 msgr2=0x7fb67c102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-19T01:28:44.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.814+0000 7fb681ec6640 1 -- 172.21.15.116:0/3530248861 shutdown_connections 2024-03-19T01:28:44.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-03-19T01:28:44.814+0000 7fb681ec6640 1 -- 172.21.15.116:0/3530248861 wait complete. 2024-03-19T01:28:44.835 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T01:28:44.869 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T01:28:44.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T01:28:44.941 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-19T01:28:44.941 DEBUG:teuthology.orchestra.run.smithi116:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-19T01:28:44.958 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-19T01:28:44.974 DEBUG:teuthology.orchestra.run.smithi162:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-19T01:28:45.006 INFO:tasks.cephadm:Stopping all daemons... 2024-03-19T01:28:45.007 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-19T01:28:45.007 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.a 2024-03-19T01:28:45.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:44 smithi162 ceph-mon[67178]: pgmap v836: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-03-19T01:28:45.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:44 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/651594371' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T01:28:45.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:44 smithi162 ceph-mon[67178]: osdmap e89: 12 total, 12 up, 12 in 2024-03-19T01:28:45.149 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:44 smithi162 ceph-mon[67178]: from='client.? 172.21.15.116:0/3530248861' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T01:28:45.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:44 smithi116 ceph-mon[62701]: pgmap v836: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-03-19T01:28:45.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:44 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/651594371' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T01:28:45.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:44 smithi116 ceph-mon[62701]: osdmap e89: 12 total, 12 up, 12 in 2024-03-19T01:28:45.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:44 smithi116 ceph-mon[62701]: from='client.? 172.21.15.116:0/3530248861' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T01:28:45.155 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:45 smithi116 systemd[1]: Stopping Ceph mon.a for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:28:45.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:44 smithi156 ceph-mon[66799]: pgmap v836: 97 pgs: 97 active+clean; 1.6 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 13 op/s 2024-03-19T01:28:45.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:44 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/651594371' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-19T01:28:45.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:44 smithi156 ceph-mon[66799]: osdmap e89: 12 total, 12 up, 12 in 2024-03-19T01:28:45.224 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:44 smithi156 ceph-mon[66799]: from='client.? 172.21.15.116:0/3530248861' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-19T01:28:45.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:45 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-a[62697]: 2024-03-19T01:28:45.573+0000 7f3962ff6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:28:45.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:45 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-a[62697]: 2024-03-19T01:28:45.573+0000 7f3962ff6700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-19T01:28:45.905 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:45 smithi116 podman[123329]: 2024-03-19 01:28:45.625788583 +0000 UTC m=+0.396223950 container died a41987fc403771ac6802512e0eec1abdf17d68d4c71ee454a69989dd7de95e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T01:28:46.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:46 smithi116 podman[123329]: 2024-03-19 01:28:46.048434794 +0000 UTC m=+0.818870168 container cleanup a41987fc403771ac6802512e0eec1abdf17d68d4c71ee454a69989dd7de95e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, ceph=True) 2024-03-19T01:28:46.266 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:46 smithi116 bash[123329]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-a 2024-03-19T01:28:46.491 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.a.service' 2024-03-19T01:28:46.517 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:46 smithi116 podman[123340]: 2024-03-19 01:28:46.266747431 +0000 UTC m=+0.638901661 container remove a41987fc403771ac6802512e0eec1abdf17d68d4c71ee454a69989dd7de95e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-03-19T01:28:46.517 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:46 smithi116 systemd[1]: ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-19T01:28:46.517 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:46 smithi116 systemd[1]: Stopped Ceph mon.a for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:28:46.517 INFO:journalctl@ceph.mon.a.smithi116.stdout:Mar 19 01:28:46 smithi116 systemd[1]: ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.a.service: Consumed 22.175s CPU time. 2024-03-19T01:28:46.606 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:28:46.607 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-19T01:28:46.607 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-19T01:28:46.607 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.b 2024-03-19T01:28:46.965 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:46 smithi156 systemd[1]: Stopping Ceph mon.b for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:28:47.220 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:46 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-b[66794]: 2024-03-19T01:28:46.963+0000 7fef226fc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:28:47.221 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:46 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-b[66794]: 2024-03-19T01:28:46.963+0000 7fef226fc700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-19T01:28:47.474 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:47 smithi156 podman[102696]: 2024-03-19 01:28:47.220106465 +0000 UTC m=+0.418081479 container died bc093f4fc87f0f42ca4e5e1c5b753cd3a1ad4a6d7bb669acc01d00edc87be279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-b, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:28:48.183 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:47 smithi156 podman[102696]: 2024-03-19 01:28:47.948379044 +0000 UTC m=+1.146354016 container cleanup bc093f4fc87f0f42ca4e5e1c5b753cd3a1ad4a6d7bb669acc01d00edc87be279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T01:28:48.184 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:47 smithi156 bash[102696]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-b 2024-03-19T01:28:48.442 INFO:journalctl@ceph.mon.b.smithi156.stdout:Mar 19 01:28:48 smithi156 podman[102710]: 2024-03-19 01:28:48.183346795 +0000 UTC m=+0.955004629 container remove bc093f4fc87f0f42ca4e5e1c5b753cd3a1ad4a6d7bb669acc01d00edc87be279 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T01:28:48.453 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.b.service' 2024-03-19T01:28:48.503 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:28:48.504 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-19T01:28:48.504 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-19T01:28:48.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.c 2024-03-19T01:28:48.899 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:48 smithi162 systemd[1]: Stopping Ceph mon.c for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:28:49.399 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:48 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-c[67174]: 2024-03-19T01:28:48.966+0000 7f8d81bd1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:28:49.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:48 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-c[67174]: 2024-03-19T01:28:48.966+0000 7f8d81bd1700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-19T01:28:49.400 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:49 smithi162 podman[103797]: 2024-03-19 01:28:49.160220656 +0000 UTC m=+0.462946862 container died 8c9c409520b397746bd6310fadbae6ea8be3a5532c1ca84b396b0296047b898a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux ) 2024-03-19T01:28:49.827 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:49 smithi162 podman[103797]: 2024-03-19 01:28:49.567203075 +0000 UTC m=+0.869929260 container cleanup 8c9c409520b397746bd6310fadbae6ea8be3a5532c1ca84b396b0296047b898a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T01:28:49.828 INFO:journalctl@ceph.mon.c.smithi162.stdout:Mar 19 01:28:49 smithi162 bash[103797]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mon-c 2024-03-19T01:28:50.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mon.c.service' 2024-03-19T01:28:50.065 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:28:50.065 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-19T01:28:50.065 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-19T01:28:50.065 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.x 2024-03-19T01:28:50.370 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:28:50 smithi116 systemd[1]: Stopping Ceph mgr.x for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:28:50.655 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:28:50 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:28:50.369+0000 7f5d0e3e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:28:50.656 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:28:50 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x[62936]: 2024-03-19T01:28:50.369+0000 7f5d0e3e0700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-19T01:28:50.656 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:28:50 smithi116 podman[123445]: 2024-03-19 01:28:50.401960489 +0000 UTC m=+0.143235635 container died 1e8e8ed90d700f79cd94f118089968db39a86a6092d556ff6d425d3c49cce8c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux ) 2024-03-19T01:28:51.155 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:28:50 smithi116 podman[123445]: 2024-03-19 01:28:50.795940072 +0000 UTC m=+0.537215217 container cleanup 1e8e8ed90d700f79cd94f118089968db39a86a6092d556ff6d425d3c49cce8c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:b6ae24918b03b1b8a19c9263a50655bba80bb810, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-19T01:28:51.155 INFO:journalctl@ceph.mgr.x.smithi116.stdout:Mar 19 01:28:50 smithi116 bash[123445]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-x 2024-03-19T01:28:51.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-19T01:28:51.402 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:28:51.402 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-19T01:28:51.402 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-19T01:28:51.403 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.y 2024-03-19T01:28:51.724 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:28:51 smithi156 systemd[1]: Stopping Ceph mgr.y for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:28:52.045 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:28:51 smithi156 podman[102830]: 2024-03-19 01:28:51.723956098 +0000 UTC m=+0.130695597 container died a73594c62eb8b699027bf61b2853a17a9444e6ac4264069c4a22fc486f80744f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:28:52.045 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:28:52 smithi156 podman[102830]: 2024-03-19 01:28:52.045094301 +0000 UTC m=+0.451833839 container cleanup a73594c62eb8b699027bf61b2853a17a9444e6ac4264069c4a22fc486f80744f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-19T01:28:52.454 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:28:52 smithi156 bash[102830]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y 2024-03-19T01:28:52.454 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:28:52 smithi156 podman[102845]: 2024-03-19 01:28:52.263352585 +0000 UTC m=+0.537067481 container remove a73594c62eb8b699027bf61b2853a17a9444e6ac4264069c4a22fc486f80744f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-y, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:28:52.454 INFO:journalctl@ceph.mgr.y.smithi156.stdout:Mar 19 01:28:52 smithi156 systemd[1]: ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-19T01:28:52.467 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-19T01:28:52.504 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:28:52.504 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-19T01:28:52.505 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-19T01:28:52.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.z 2024-03-19T01:28:52.843 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:28:52 smithi162 systemd[1]: Stopping Ceph mgr.z for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:28:53.141 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:28:52 smithi162 podman[103929]: 2024-03-19 01:28:52.84329118 +0000 UTC m=+0.148129610 container died edc47797286761a05e808a613debfcd3c8b4059aa927ea614ba487f97085292a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:28:53.402 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:28:53 smithi162 podman[103929]: 2024-03-19 01:28:53.231664952 +0000 UTC m=+0.536503373 container cleanup edc47797286761a05e808a613debfcd3c8b4059aa927ea614ba487f97085292a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:28:53.402 INFO:journalctl@ceph.mgr.z.smithi162.stdout:Mar 19 01:28:53 smithi162 bash[103929]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mgr-z 2024-03-19T01:28:53.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-19T01:28:53.694 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:28:53.694 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-19T01:28:53.694 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-19T01:28:53.694 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.0 2024-03-19T01:28:53.984 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:28:53 smithi116 systemd[1]: Stopping Ceph osd.0 for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:28:54.405 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:28:53 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0[77431]: 2024-03-19T01:28:53.983+0000 7f5036758700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:28:54.405 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:28:53 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0[77431]: 2024-03-19T01:28:53.983+0000 7f5036758700 -1 osd.0 90 *** Got signal Terminated *** 2024-03-19T01:28:54.405 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:28:53 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0[77431]: 2024-03-19T01:28:53.983+0000 7f5036758700 -1 osd.0 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T01:28:59.901 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:28:59 smithi116 podman[123567]: 2024-03-19 01:28:59.601325487 +0000 UTC m=+5.716758422 container died 0b756cb5b23a071f047dc6f7485d5298a6c0c5affab8abde2fe94cc22edfe0b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-19T01:29:00.155 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:29:00 smithi116 podman[123567]: 2024-03-19 01:29:00.000678914 +0000 UTC m=+6.116111860 container cleanup 0b756cb5b23a071f047dc6f7485d5298a6c0c5affab8abde2fe94cc22edfe0b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-19T01:29:00.155 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:29:00 smithi116 bash[123567]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0 2024-03-19T01:29:00.406 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:29:00 smithi116 podman[123584]: 2024-03-19 01:29:00.218880123 +0000 UTC m=+0.612148143 container remove 0b756cb5b23a071f047dc6f7485d5298a6c0c5affab8abde2fe94cc22edfe0b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-19T01:29:00.948 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:29:00 smithi116 podman[123674]: 2024-03-19 01:29:00.567696494 +0000 UTC m=+0.019255537 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:29:00.949 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:29:00 smithi116 podman[123674]: 2024-03-19 01:29:00.71206328 +0000 UTC m=+0.163622331 container create 603d0cc707ce7d2b36790f544b7a0087fcdbbd560e73544fb9fbbd0d1861f87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-19T01:29:01.244 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:29:00 smithi116 podman[123674]: 2024-03-19 01:29:00.94854847 +0000 UTC m=+0.400107498 container init 603d0cc707ce7d2b36790f544b7a0087fcdbbd560e73544fb9fbbd0d1861f87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-03-19T01:29:01.244 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:29:00 smithi116 podman[123674]: 2024-03-19 01:29:00.952154939 +0000 UTC m=+0.403713944 container start 603d0cc707ce7d2b36790f544b7a0087fcdbbd560e73544fb9fbbd0d1861f87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:29:01.244 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:29:01 smithi116 podman[123674]: 2024-03-19 01:29:01.048206 +0000 UTC m=+0.499765017 container attach 603d0cc707ce7d2b36790f544b7a0087fcdbbd560e73544fb9fbbd0d1861f87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T01:29:01.519 INFO:journalctl@ceph.osd.0.smithi116.stdout:Mar 19 01:29:01 smithi116 podman[123674]: 2024-03-19 01:29:01.244078467 +0000 UTC m=+0.695637472 container died 603d0cc707ce7d2b36790f544b7a0087fcdbbd560e73544fb9fbbd0d1861f87a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-19T01:29:01.903 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.0.service' 2024-03-19T01:29:01.945 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:29:01.945 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-19T01:29:01.945 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-19T01:29:01.946 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.1 2024-03-19T01:29:02.267 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:01 smithi156 systemd[1]: Stopping Ceph osd.1 for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:29:02.724 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:02 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1[71462]: 2024-03-19T01:29:02.265+0000 7fa764087700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:29:02.724 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:02 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1[71462]: 2024-03-19T01:29:02.265+0000 7fa764087700 -1 osd.1 90 *** Got signal Terminated *** 2024-03-19T01:29:02.724 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:02 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1[71462]: 2024-03-19T01:29:02.265+0000 7fa764087700 -1 osd.1 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T01:29:08.035 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:07 smithi156 podman[102956]: 2024-03-19 01:29:07.739170688 +0000 UTC m=+5.606217838 container died 9ba4ea87209c557e9b458cd6a6c1db8f339807f0b704c1e5b7adb85823fd8b8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-19T01:29:08.369 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:08 smithi156 podman[102956]: 2024-03-19 01:29:08.134066976 +0000 UTC m=+6.001114121 container cleanup 9ba4ea87209c557e9b458cd6a6c1db8f339807f0b704c1e5b7adb85823fd8b8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-03-19T01:29:08.369 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:08 smithi156 bash[102956]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1 2024-03-19T01:29:08.724 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:08 smithi156 podman[102974]: 2024-03-19 01:29:08.368903398 +0000 UTC m=+0.626477266 container remove 9ba4ea87209c557e9b458cd6a6c1db8f339807f0b704c1e5b7adb85823fd8b8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-19T01:29:09.039 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:08 smithi156 podman[103068]: 2024-03-19 01:29:08.67733048 +0000 UTC m=+0.018252681 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:29:09.040 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:08 smithi156 podman[103068]: 2024-03-19 01:29:08.803246929 +0000 UTC m=+0.144169140 container create 1833818d331a2e7e042aab5c298457e20c092735486dc8252ac7e3883af2f9b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:29:09.040 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:09 smithi156 podman[103068]: 2024-03-19 01:29:09.039315519 +0000 UTC m=+0.380237715 container init 1833818d331a2e7e042aab5c298457e20c092735486dc8252ac7e3883af2f9b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-19T01:29:09.326 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:09 smithi156 podman[103068]: 2024-03-19 01:29:09.042963036 +0000 UTC m=+0.383885228 container start 1833818d331a2e7e042aab5c298457e20c092735486dc8252ac7e3883af2f9b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:29:09.326 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:09 smithi156 podman[103068]: 2024-03-19 01:29:09.131124929 +0000 UTC m=+0.472047123 container attach 1833818d331a2e7e042aab5c298457e20c092735486dc8252ac7e3883af2f9b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-19T01:29:09.711 INFO:journalctl@ceph.osd.1.smithi156.stdout:Mar 19 01:29:09 smithi156 podman[103068]: 2024-03-19 01:29:09.346618544 +0000 UTC m=+0.687540745 container died 1833818d331a2e7e042aab5c298457e20c092735486dc8252ac7e3883af2f9b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-19T01:29:10.032 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.1.service' 2024-03-19T01:29:10.073 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:29:10.073 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-19T01:29:10.074 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-19T01:29:10.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.2 2024-03-19T01:29:10.385 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:10 smithi162 systemd[1]: Stopping Ceph osd.2 for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:29:10.649 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:10 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2[72346]: 2024-03-19T01:29:10.383+0000 7fa7d13e3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:29:10.649 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:10 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2[72346]: 2024-03-19T01:29:10.383+0000 7fa7d13e3700 -1 osd.2 90 *** Got signal Terminated *** 2024-03-19T01:29:10.649 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:10 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2[72346]: 2024-03-19T01:29:10.383+0000 7fa7d13e3700 -1 osd.2 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T01:29:16.311 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:16 smithi162 podman[104054]: 2024-03-19 01:29:16.001937684 +0000 UTC m=+5.743489310 container died d214d261da72ec8a254a5a29704ca381f30fa491277ceaf0a244e94ddaf7bc9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-03-19T01:29:16.649 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:16 smithi162 podman[104054]: 2024-03-19 01:29:16.392969582 +0000 UTC m=+6.134521208 container cleanup d214d261da72ec8a254a5a29704ca381f30fa491277ceaf0a244e94ddaf7bc9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-19T01:29:16.649 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:16 smithi162 bash[104054]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2 2024-03-19T01:29:17.002 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:16 smithi162 podman[104071]: 2024-03-19 01:29:16.661059706 +0000 UTC m=+0.655740019 container remove d214d261da72ec8a254a5a29704ca381f30fa491277ceaf0a244e94ddaf7bc9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:29:17.273 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:17 smithi162 podman[104162]: 2024-03-19 01:29:16.904422196 +0000 UTC m=+0.017743618 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:29:17.273 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:17 smithi162 podman[104162]: 2024-03-19 01:29:17.03703096 +0000 UTC m=+0.150352376 container create 7ad2365d10daf3040c3467493dc23eb159c3975b03e82650d7f0332115d437ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T01:29:17.273 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:17 smithi162 podman[104162]: 2024-03-19 01:29:17.272764484 +0000 UTC m=+0.386085901 container init 7ad2365d10daf3040c3467493dc23eb159c3975b03e82650d7f0332115d437ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:29:17.564 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:17 smithi162 podman[104162]: 2024-03-19 01:29:17.276476816 +0000 UTC m=+0.389798235 container start 7ad2365d10daf3040c3467493dc23eb159c3975b03e82650d7f0332115d437ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-03-19T01:29:17.564 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:17 smithi162 podman[104162]: 2024-03-19 01:29:17.356178879 +0000 UTC m=+0.469500295 container attach 7ad2365d10daf3040c3467493dc23eb159c3975b03e82650d7f0332115d437ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-19T01:29:17.899 INFO:journalctl@ceph.osd.2.smithi162.stdout:Mar 19 01:29:17 smithi162 podman[104162]: 2024-03-19 01:29:17.563910096 +0000 UTC m=+0.677231516 container died 7ad2365d10daf3040c3467493dc23eb159c3975b03e82650d7f0332115d437ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T01:29:18.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.2.service' 2024-03-19T01:29:18.293 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:29:18.293 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-19T01:29:18.293 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-19T01:29:18.293 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.3 2024-03-19T01:29:18.605 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:18 smithi116 systemd[1]: Stopping Ceph osd.3 for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:29:18.905 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:18 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3[83944]: 2024-03-19T01:29:18.604+0000 7f6d4854c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:29:18.905 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:18 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3[83944]: 2024-03-19T01:29:18.604+0000 7f6d4854c700 -1 osd.3 90 *** Got signal Terminated *** 2024-03-19T01:29:18.905 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:18 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3[83944]: 2024-03-19T01:29:18.604+0000 7f6d4854c700 -1 osd.3 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T01:29:21.844 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:21 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:21.462+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:22.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:21 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:21.844+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:22.655 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:22 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3[83944]: 2024-03-19T01:29:22.394+0000 7f6d40b38700 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.847412+0000 front 2024-03-19T01:28:55.847439+0000 (oldest deadline 2024-03-19T01:29:21.648175+0000) 2024-03-19T01:29:22.655 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:22 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:22.509+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:23.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:22 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:22.818+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:23.224 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:22 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:29:22.882+0000 7f0efca2c700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:57.339900+0000 front 2024-03-19T01:28:57.339733+0000 (oldest deadline 2024-03-19T01:29:22.540496+0000) 2024-03-19T01:29:23.809 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:23 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:23.507+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:23.810 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:23 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3[83944]: 2024-03-19T01:29:23.410+0000 7f6d40b38700 -1 osd.3 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.847412+0000 front 2024-03-19T01:28:55.847439+0000 (oldest deadline 2024-03-19T01:29:21.648175+0000) 2024-03-19T01:29:24.149 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:23 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:23.857+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:24.155 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:24 smithi116 podman[123791]: 2024-03-19 01:29:24.015719234 +0000 UTC m=+5.524266291 container died 9dd4351123502cc05acbb5f2d22b05804f279880a6648565e0800a068b4453c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:29:24.156 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:23 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:23.808+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:24.224 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:23 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:29:23.850+0000 7f0efca2c700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:57.339900+0000 front 2024-03-19T01:28:57.339733+0000 (oldest deadline 2024-03-19T01:29:22.540496+0000) 2024-03-19T01:29:24.655 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:24 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:24.496+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:24.655 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:24 smithi116 podman[123791]: 2024-03-19 01:29:24.429540346 +0000 UTC m=+5.938087402 container cleanup 9dd4351123502cc05acbb5f2d22b05804f279880a6648565e0800a068b4453c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T01:29:24.655 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:24 smithi116 bash[123791]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3 2024-03-19T01:29:24.887 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:24 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:24.511+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:25.039 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:24 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:24.781+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:25.039 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:24 smithi116 podman[123807]: 2024-03-19 01:29:24.691706384 +0000 UTC m=+0.672045244 container remove 9dd4351123502cc05acbb5f2d22b05804f279880a6648565e0800a068b4453c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-03-19T01:29:25.149 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:24 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:24.886+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:25.224 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:24 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:29:24.841+0000 7f0efca2c700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:57.339900+0000 front 2024-03-19T01:28:57.339733+0000 (oldest deadline 2024-03-19T01:29:22.540496+0000) 2024-03-19T01:29:25.294 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:25 smithi116 podman[123898]: 2024-03-19 01:29:24.941888348 +0000 UTC m=+0.017930891 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:29:25.294 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:25 smithi116 podman[123898]: 2024-03-19 01:29:25.040305242 +0000 UTC m=+0.116347802 container create aacf931c99aaee3ab9d89fbbd6c4a0d396c98eef7b3d3308bfed7c70306c00e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-19T01:29:25.584 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:25 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:25.457+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:25.584 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:25 smithi116 podman[123898]: 2024-03-19 01:29:25.294414219 +0000 UTC m=+0.370456751 container init aacf931c99aaee3ab9d89fbbd6c4a0d396c98eef7b3d3308bfed7c70306c00e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-19T01:29:25.584 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:25 smithi116 podman[123898]: 2024-03-19 01:29:25.298013864 +0000 UTC m=+0.374056380 container start aacf931c99aaee3ab9d89fbbd6c4a0d396c98eef7b3d3308bfed7c70306c00e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-19T01:29:25.584 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:25 smithi116 podman[123898]: 2024-03-19 01:29:25.359807813 +0000 UTC m=+0.435850338 container attach aacf931c99aaee3ab9d89fbbd6c4a0d396c98eef7b3d3308bfed7c70306c00e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-19T01:29:25.850 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:25 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:25.530+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:25.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:25 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:25.775+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:25.905 INFO:journalctl@ceph.osd.3.smithi116.stdout:Mar 19 01:29:25 smithi116 podman[123898]: 2024-03-19 01:29:25.584464097 +0000 UTC m=+0.660506620 container died aacf931c99aaee3ab9d89fbbd6c4a0d396c98eef7b3d3308bfed7c70306c00e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-19T01:29:26.149 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:25 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:25.849+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:26.224 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:25 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:29:25.869+0000 7f0efca2c700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:57.339900+0000 front 2024-03-19T01:28:57.339733+0000 (oldest deadline 2024-03-19T01:29:22.540496+0000) 2024-03-19T01:29:26.284 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.3.service' 2024-03-19T01:29:26.325 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:29:26.325 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-19T01:29:26.325 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-19T01:29:26.325 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.4 2024-03-19T01:29:26.535 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:26 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:26.488+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:26.631 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:26 smithi156 systemd[1]: Stopping Ceph osd.4 for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:29:26.869 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:26 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:26.566+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:26.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:26 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:26.726+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:26.974 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:26 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:29:26.629+0000 7f0f04440700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:29:26.974 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:26 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:29:26.630+0000 7f0f04440700 -1 osd.4 90 *** Got signal Terminated *** 2024-03-19T01:29:26.974 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:26 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:29:26.630+0000 7f0f04440700 -1 osd.4 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T01:29:26.974 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:26 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:29:26.828+0000 7f0efca2c700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:57.339900+0000 front 2024-03-19T01:28:57.339733+0000 (oldest deadline 2024-03-19T01:29:22.540496+0000) 2024-03-19T01:29:27.149 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:26 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:26.868+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:27.726 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:27 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:27.489+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:27.828 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:27 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:27.571+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:28.149 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:27 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:27.827+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:28.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:27 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:27.726+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:28.224 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:27 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:29:27.817+0000 7f0efca2c700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:57.339900+0000 front 2024-03-19T01:28:57.339733+0000 (oldest deadline 2024-03-19T01:29:22.540496+0000) 2024-03-19T01:29:28.869 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:28 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:28.597+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:28.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:28 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:28.521+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:28.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:28 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:28.677+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:29.149 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:28 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:28.868+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:29.224 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:28 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:29:28.786+0000 7f0efca2c700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:57.339900+0000 front 2024-03-19T01:28:57.339733+0000 (oldest deadline 2024-03-19T01:29:22.540496+0000) 2024-03-19T01:29:29.724 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:29 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:29.258+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:29.870 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:29 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:29.569+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:29.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:29 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:29.667+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:29.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:29 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:29.478+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:30.149 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:29 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:29.868+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:30.149 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:29 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:29.938+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:30.224 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:29 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:29:29.769+0000 7f0efca2c700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:57.339900+0000 front 2024-03-19T01:28:57.339733+0000 (oldest deadline 2024-03-19T01:29:22.540496+0000) 2024-03-19T01:29:30.724 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:30 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:30.278+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:30.805 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:30 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:30.716+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:30.806 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:30 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:30.495+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:30.806 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:30 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:30.495+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:30.877 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:30 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:30.589+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:30.877 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:30 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:30.589+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:31.081 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:30 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4[78411]: 2024-03-19T01:29:30.804+0000 7f0efca2c700 -1 osd.4 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:57.339900+0000 front 2024-03-19T01:28:57.339733+0000 (oldest deadline 2024-03-19T01:29:22.540496+0000) 2024-03-19T01:29:31.149 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:30 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:30.909+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:31.149 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:30 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:30.876+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:31.149 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:30 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:30.876+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:04.045239+0000 front 2024-03-19T01:29:04.045211+0000 (oldest deadline 2024-03-19T01:29:29.945206+0000) 2024-03-19T01:29:31.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:31 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:31.258+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:31.817 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:31 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:31.729+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:31.817 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:31 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:31.518+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:31.818 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:31 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:31.518+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:31.899 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:31 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:31.871+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:31.899 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:31 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:31.871+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:04.045239+0000 front 2024-03-19T01:29:04.045211+0000 (oldest deadline 2024-03-19T01:29:29.945206+0000) 2024-03-19T01:29:31.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:31 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:31.624+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:31.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:31 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:31.624+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:32.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:31 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:31.945+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:32.474 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:32 smithi156 podman[103190]: 2024-03-19 01:29:32.208314009 +0000 UTC m=+5.683445456 container died 50fa4a6d94065392fdfcfdc144a6982cc2ef4a54256d07612632657a3d647b09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:29:32.475 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:32 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:32.229+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:32.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:32 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:32.684+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:32.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:32 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:32.684+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:32.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:32 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:32.539+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:32.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:32 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:32.539+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:32.932 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:32 smithi156 podman[103190]: 2024-03-19 01:29:32.530078581 +0000 UTC m=+6.005210030 container cleanup 50fa4a6d94065392fdfcfdc144a6982cc2ef4a54256d07612632657a3d647b09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-19T01:29:32.933 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:32 smithi156 bash[103190]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4 2024-03-19T01:29:32.933 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:32 smithi156 podman[103208]: 2024-03-19 01:29:32.739834494 +0000 UTC m=+0.529883484 container remove 50fa4a6d94065392fdfcfdc144a6982cc2ef4a54256d07612632657a3d647b09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-19T01:29:33.149 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:32 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:32.913+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:33.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:32 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:32.667+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:33.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:32 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:32.667+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:33.150 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:32 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:32.867+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:33.150 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:32 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:32.867+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:04.045239+0000 front 2024-03-19T01:29:04.045211+0000 (oldest deadline 2024-03-19T01:29:29.945206+0000) 2024-03-19T01:29:33.190 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:32 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:32.931+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:33.190 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:33 smithi156 podman[103298]: 2024-03-19 01:29:33.03518933 +0000 UTC m=+0.017883657 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:29:33.190 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:33 smithi156 podman[103298]: 2024-03-19 01:29:33.157502376 +0000 UTC m=+0.140196703 container create 6e9608528fef7edf05ca5bd583e77ab114cac2fa00d4179cefdaffde7e3c74d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T01:29:33.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:33 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:33.189+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:33.474 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:33 smithi156 podman[103298]: 2024-03-19 01:29:33.359847554 +0000 UTC m=+0.342541887 container init 6e9608528fef7edf05ca5bd583e77ab114cac2fa00d4179cefdaffde7e3c74d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-19T01:29:33.475 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:33 smithi156 podman[103298]: 2024-03-19 01:29:33.362852008 +0000 UTC m=+0.345546328 container start 6e9608528fef7edf05ca5bd583e77ab114cac2fa00d4179cefdaffde7e3c74d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-19T01:29:33.475 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:33 smithi156 podman[103298]: 2024-03-19 01:29:33.434765325 +0000 UTC m=+0.417459670 container attach 6e9608528fef7edf05ca5bd583e77ab114cac2fa00d4179cefdaffde7e3c74d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=) 2024-03-19T01:29:33.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:33 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:33.652+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:33.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:33 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:33.652+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:33.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:33 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:33.536+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:33.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:33 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:33.536+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:33.938 INFO:journalctl@ceph.osd.4.smithi156.stdout:Mar 19 01:29:33 smithi156 podman[103298]: 2024-03-19 01:29:33.656053957 +0000 UTC m=+0.638748303 container died 6e9608528fef7edf05ca5bd583e77ab114cac2fa00d4179cefdaffde7e3c74d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-4-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-19T01:29:33.961 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:33 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:33.703+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:33.961 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:33 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:33.703+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:33.962 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:33 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:33.885+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:33.962 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:33 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:33.885+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:04.045239+0000 front 2024-03-19T01:29:04.045211+0000 (oldest deadline 2024-03-19T01:29:29.945206+0000) 2024-03-19T01:29:34.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:33 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:33.961+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:34.224 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:34 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:34.163+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:34.261 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.4.service' 2024-03-19T01:29:34.302 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:29:34.302 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-19T01:29:34.302 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-19T01:29:34.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.5 2024-03-19T01:29:34.331 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:33 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:33.960+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:34.629 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:34 smithi162 systemd[1]: Stopping Ceph osd.5 for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:29:34.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:34 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:34.661+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:34.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:34 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:34.661+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:34.900 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:34 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:34.628+0000 7f20ffc4e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:29:34.900 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:34 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:34.628+0000 7f20ffc4e700 -1 osd.5 90 *** Got signal Terminated *** 2024-03-19T01:29:34.900 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:34 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:34.628+0000 7f20ffc4e700 -1 osd.5 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T01:29:34.900 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:34 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:34.869+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:34.900 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:34 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:34.869+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:04.045239+0000 front 2024-03-19T01:29:04.045211+0000 (oldest deadline 2024-03-19T01:29:29.945206+0000) 2024-03-19T01:29:34.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:34 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:34.688+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:34.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:34 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:34.688+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:34.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:34 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:34.534+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:34.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:34 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:34.534+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:35.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:34 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:34.955+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:35.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:35 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:35.011+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:35.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:35 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:35.195+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:35.899 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:35 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:35.831+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:35.899 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:35 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:35.831+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:04.045239+0000 front 2024-03-19T01:29:04.045211+0000 (oldest deadline 2024-03-19T01:29:29.945206+0000) 2024-03-19T01:29:35.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:35 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:35.621+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:35.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:35 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:35.621+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:35.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:35 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:35.709+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:35.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:35 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:35.709+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:35.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:35 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:35.490+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:35.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:35 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:35.490+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:36.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:35 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:35.961+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:36.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:36 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:36.162+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:36.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:36 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:36.034+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:36.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:36 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:36.745+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:36.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:36 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:36.745+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:36.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:36 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:36.517+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:36.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:36 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:36.517+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:36.959 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:36 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:36.667+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:36.959 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:36 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:36.667+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:36.960 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:36 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:36.785+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:36.960 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:36 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:36.785+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:04.045239+0000 front 2024-03-19T01:29:04.045211+0000 (oldest deadline 2024-03-19T01:29:29.945206+0000) 2024-03-19T01:29:37.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:36 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:36.958+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:37.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:37 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:37.069+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:37.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:37 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:37.147+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:37.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:37.717+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:37.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:37.717+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:37.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:37.493+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:37.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:37.493+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:37.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:37.493+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:37.955 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:37 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:37.816+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:37.955 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:37 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:37.816+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:04.045239+0000 front 2024-03-19T01:29:04.045211+0000 (oldest deadline 2024-03-19T01:29:29.945206+0000) 2024-03-19T01:29:37.955 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:37 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:37.687+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:37.955 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:37 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:37.687+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:38.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:37 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:37.953+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:38.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:38 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:38.182+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:38.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:38 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:38.118+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:38.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:38 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:38.673+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:38.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:38 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:38.673+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:38.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:38 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:38.533+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:38.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:38 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:38.533+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:38.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:38 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:38.533+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:38.963 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:38 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:38.665+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:38.963 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:38 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:38.665+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:38.963 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:38 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:38.771+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:55.343578+0000 front 2024-03-19T01:28:55.343635+0000 (oldest deadline 2024-03-19T01:29:23.544060+0000) 2024-03-19T01:29:38.963 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:38 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:38.771+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:04.045239+0000 front 2024-03-19T01:29:04.045211+0000 (oldest deadline 2024-03-19T01:29:29.945206+0000) 2024-03-19T01:29:38.963 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:38 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5[79030]: 2024-03-19T01:29:38.771+0000 7f20f823a700 -1 osd.5 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:11.046169+0000 front 2024-03-19T01:29:11.046217+0000 (oldest deadline 2024-03-19T01:29:38.646766+0000) 2024-03-19T01:29:39.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:38 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:38.962+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:39.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:39 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:39.187+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:39.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:39 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:39.070+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:39.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:39 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:39.719+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:39.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:39 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:39.719+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:39.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:39 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:39.542+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:39.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:39 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:39.542+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:39.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:39 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:39.542+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:40.003 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:39 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:39.699+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:40.003 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:39 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:39.699+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:40.399 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:40 smithi162 podman[104281]: 2024-03-19 01:29:40.235956125 +0000 UTC m=+5.733190423 container died 4da553d7f7eb0265632b0901eb4598429e2726fa1015a7569f62e57047b00243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-19T01:29:40.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:40 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:40.002+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:40.400 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:40 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:40.002+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:40.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:40 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:40.140+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:40.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:40 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:40.140+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:40.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:40 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:40.067+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:40.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:40 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:40.067+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:40.888 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:40 smithi162 podman[104281]: 2024-03-19 01:29:40.670241526 +0000 UTC m=+6.167475816 container cleanup 4da553d7f7eb0265632b0901eb4598429e2726fa1015a7569f62e57047b00243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-19T01:29:40.888 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:40 smithi162 bash[104281]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5 2024-03-19T01:29:40.889 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:40 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:40.702+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:40.889 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:40 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:40.702+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:40.889 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:40 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:40.702+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:40.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:40 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:40.692+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:40.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:40 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:40.692+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:40.905 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:40 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:40.692+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.392016+0000 front 2024-03-19T01:29:14.392003+0000 (oldest deadline 2024-03-19T01:29:40.291942+0000) 2024-03-19T01:29:40.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:40 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:40.508+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:40.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:40 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:40.508+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:40.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:40 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:40.508+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:41.149 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:40 smithi162 podman[104298]: 2024-03-19 01:29:40.888348033 +0000 UTC m=+0.650669446 container remove 4da553d7f7eb0265632b0901eb4598429e2726fa1015a7569f62e57047b00243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-19T01:29:41.149 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:40 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:40.992+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:41.149 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:40 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:40.993+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:41.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:41.164+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:41.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:41.164+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:41.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:41.062+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:41.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:41 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:41.062+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:41.617 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:41 smithi162 podman[104388]: 2024-03-19 01:29:41.224317579 +0000 UTC m=+0.017692837 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:29:41.617 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:41 smithi162 podman[104388]: 2024-03-19 01:29:41.372512676 +0000 UTC m=+0.165887927 container create b05702953b635abd6513f7ea5043817997db146495d68b16294fdb79d4d32401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T01:29:41.860 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:41.666+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:41.860 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:41.666+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:41.860 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:41.666+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.392016+0000 front 2024-03-19T01:29:14.392003+0000 (oldest deadline 2024-03-19T01:29:40.291942+0000) 2024-03-19T01:29:41.861 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:41.467+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:41.861 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:41.467+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:41.861 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:41 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:41.467+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:41.900 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:41 smithi162 podman[104388]: 2024-03-19 01:29:41.617167137 +0000 UTC m=+0.410542381 container init b05702953b635abd6513f7ea5043817997db146495d68b16294fdb79d4d32401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-19T01:29:41.900 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:41 smithi162 podman[104388]: 2024-03-19 01:29:41.620366022 +0000 UTC m=+0.413741265 container start b05702953b635abd6513f7ea5043817997db146495d68b16294fdb79d4d32401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-19T01:29:41.900 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:41 smithi162 podman[104388]: 2024-03-19 01:29:41.700329672 +0000 UTC m=+0.493704934 container attach b05702953b635abd6513f7ea5043817997db146495d68b16294fdb79d4d32401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=) 2024-03-19T01:29:41.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:41.738+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:41.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:41.738+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:41.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:41 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:41.738+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:42.262 INFO:journalctl@ceph.osd.5.smithi162.stdout:Mar 19 01:29:41 smithi162 podman[104388]: 2024-03-19 01:29:41.903536891 +0000 UTC m=+0.696912135 container died b05702953b635abd6513f7ea5043817997db146495d68b16294fdb79d4d32401 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-19T01:29:42.262 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:42 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:42.027+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:42.262 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:42 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:42.027+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:42.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:42.145+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:42.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:42.145+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:42.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:42.038+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:42.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:42.038+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:42.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.5.service' 2024-03-19T01:29:42.670 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:42 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:42.419+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:42.670 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:42 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:42.419+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:42.670 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:42 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:42.419+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:42.673 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:29:42.673 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-19T01:29:42.673 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-19T01:29:42.674 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.6 2024-03-19T01:29:42.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:42 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:42.718+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:42.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:42 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:42.718+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:42.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:42 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:42.718+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:42.999 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:42 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:42.669+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:43.000 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:42 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:42.669+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:43.000 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:42 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:42.669+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.392016+0000 front 2024-03-19T01:29:14.392003+0000 (oldest deadline 2024-03-19T01:29:40.291942+0000) 2024-03-19T01:29:43.000 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:42 smithi116 systemd[1]: Stopping Ceph osd.6 for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:29:43.000 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:43 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:42.999+0000 7fb3275c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:29:43.000 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:43 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:42.999+0000 7fb3275c9700 -1 osd.6 90 *** Got signal Terminated *** 2024-03-19T01:29:43.000 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:43 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:42.999+0000 7fb3275c9700 -1 osd.6 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T01:29:43.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:43 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:43.010+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:43.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:43 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:43.010+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:43.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:43 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:43.100+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:43.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:43 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:43.100+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:43.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:42.993+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:43.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:42 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:42.993+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:43.719 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:43 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:43.431+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:43.719 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:43 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:43.431+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:43.719 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:43 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:43.431+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:44.000 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:43 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:43.730+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:44.000 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:43 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:43.730+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:44.000 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:43 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:43.730+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:44.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:43 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:43.719+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:44.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:43 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:43.719+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:44.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:43 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:43.719+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.392016+0000 front 2024-03-19T01:29:14.392003+0000 (oldest deadline 2024-03-19T01:29:40.291942+0000) 2024-03-19T01:29:44.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:43 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:43.963+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:44.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:43 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:43.963+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:44.224 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:44 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:44.092+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:44.224 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:44 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:44.092+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:44.400 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:44 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:43.999+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:44.400 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:44 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:43.999+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:44.744 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:44 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:44.434+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:44.745 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:44 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:44.434+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:44.745 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:44 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:44.434+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:44.996 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:44 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:44.706+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:44.997 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:44 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:44.706+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:44.997 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:44 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:44.706+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:45.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:44 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:44.744+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:45.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:44 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:44.744+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:45.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:44 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:44.744+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.392016+0000 front 2024-03-19T01:29:14.392003+0000 (oldest deadline 2024-03-19T01:29:40.291942+0000) 2024-03-19T01:29:45.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:44 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:44.744+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.292358+0000 front 2024-03-19T01:29:20.292385+0000 (oldest deadline 2024-03-19T01:29:44.392554+0000) 2024-03-19T01:29:45.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:44 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:44.995+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:45.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:44 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:44.995+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:45.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:45 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:45.113+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:45.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:45 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:45.113+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:45.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:45 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:45.113+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:19.351979+0000 front 2024-03-19T01:29:19.352013+0000 (oldest deadline 2024-03-19T01:29:44.651964+0000) 2024-03-19T01:29:45.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:45 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:45.006+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:45.475 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:45 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:45.006+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:45.475 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:45 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:45.006+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:45.781 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:45 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:45.431+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:45.781 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:45 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:45.431+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:45.781 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:45 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:45.431+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:45.971 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:45 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:45.719+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:45.971 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:45 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:45.719+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:45.971 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:45 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:45.719+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:46.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:45 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:45.781+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:46.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:45 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:45.781+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:46.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:45 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:45.781+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.392016+0000 front 2024-03-19T01:29:14.392003+0000 (oldest deadline 2024-03-19T01:29:40.291942+0000) 2024-03-19T01:29:46.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:45 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:45.781+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.292358+0000 front 2024-03-19T01:29:20.292385+0000 (oldest deadline 2024-03-19T01:29:44.392554+0000) 2024-03-19T01:29:46.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:45 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:45.970+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:46.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:45 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:45.970+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:46.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:46 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:46.136+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:46.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:46 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:46.136+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:46.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:46 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:46.136+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:19.351979+0000 front 2024-03-19T01:29:19.352013+0000 (oldest deadline 2024-03-19T01:29:44.651964+0000) 2024-03-19T01:29:46.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:46 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:46.036+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:46.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:46 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:46.036+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:46.475 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:46 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:46.036+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:46.747 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:46 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:46.455+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:46.747 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:46 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:46.455+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:46.747 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:46 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:46.455+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:46.747 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:46 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:46.455+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:29:47.020 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:46 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:46.728+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:47.020 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:46 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:46.728+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:47.020 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:46 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:46.728+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:47.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:46 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:46.746+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:47.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:46 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:46.746+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:47.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:46 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:46.746+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.392016+0000 front 2024-03-19T01:29:14.392003+0000 (oldest deadline 2024-03-19T01:29:40.291942+0000) 2024-03-19T01:29:47.155 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:46 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:46.746+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.292358+0000 front 2024-03-19T01:29:20.292385+0000 (oldest deadline 2024-03-19T01:29:44.392554+0000) 2024-03-19T01:29:47.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:47 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:47.019+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:47.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:47 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:47.019+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:47.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:47 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:47.019+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:29:47.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:47 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:47.104+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:47.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:47 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:47.104+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:47.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:47 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:47.104+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:19.351979+0000 front 2024-03-19T01:29:19.352013+0000 (oldest deadline 2024-03-19T01:29:44.651964+0000) 2024-03-19T01:29:47.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:47 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:47.028+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:47.475 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:47 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:47.028+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:47.475 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:47 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:47.028+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:47.747 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:47 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:47.745+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:54.971635+0000 front 2024-03-19T01:28:54.971677+0000 (oldest deadline 2024-03-19T01:29:20.889984+0000) 2024-03-19T01:29:47.747 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:47 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:47.745+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.190907+0000 front 2024-03-19T01:29:06.190876+0000 (oldest deadline 2024-03-19T01:29:32.090946+0000) 2024-03-19T01:29:47.747 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:47 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:47.745+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.392016+0000 front 2024-03-19T01:29:14.392003+0000 (oldest deadline 2024-03-19T01:29:40.291942+0000) 2024-03-19T01:29:47.747 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:47 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6[90894]: 2024-03-19T01:29:47.745+0000 7fb323bbd700 -1 osd.6 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.292358+0000 front 2024-03-19T01:29:20.292385+0000 (oldest deadline 2024-03-19T01:29:44.392554+0000) 2024-03-19T01:29:47.748 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:47 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:47.496+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:47.748 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:47 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:47.496+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:47.748 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:47 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:47.496+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:47.748 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:47 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:47.496+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:29:48.027 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:47 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:47.746+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:48.027 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:47 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:47.746+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:48.027 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:47 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:47.746+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:48.299 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:48 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:48.026+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:48.299 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:48 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:48.026+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:48.299 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:48 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:48.026+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:29:48.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:48 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:48.137+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:48.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:48 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:48.137+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:48.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:48 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:48.137+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:19.351979+0000 front 2024-03-19T01:29:19.352013+0000 (oldest deadline 2024-03-19T01:29:44.651964+0000) 2024-03-19T01:29:48.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:48 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:48.029+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:48.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:48 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:48.029+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:48.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:48 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:48.029+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:48.824 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:48 smithi116 podman[124013]: 2024-03-19 01:29:48.567011701 +0000 UTC m=+5.691032359 container died d04294efa9a281df73e2b6a9a00ebc2a997132b4f9af84abfc371477628c1c06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:29:48.825 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:48 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:48.484+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:48.825 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:48 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:48.484+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:48.825 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:48 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:48.484+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:48.825 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:48 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:48.484+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:29:49.036 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:48 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:48.711+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:49.037 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:48 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:48.711+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:49.037 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:48 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:48.711+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:49.125 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:48 smithi116 podman[124013]: 2024-03-19 01:29:48.907669506 +0000 UTC m=+6.031690166 container cleanup d04294efa9a281df73e2b6a9a00ebc2a997132b4f9af84abfc371477628c1c06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-19T01:29:49.125 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:48 smithi116 bash[124013]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6 2024-03-19T01:29:49.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:49 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:49.034+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:49.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:49 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:49.034+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:49.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:49 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:49.034+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:29:49.405 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:49 smithi116 podman[124032]: 2024-03-19 01:29:49.125606662 +0000 UTC m=+0.553154875 container remove d04294efa9a281df73e2b6a9a00ebc2a997132b4f9af84abfc371477628c1c06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T01:29:49.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:49 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:49.150+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:49.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:49 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:49.150+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:49.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:49 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:49.150+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:19.351979+0000 front 2024-03-19T01:29:19.352013+0000 (oldest deadline 2024-03-19T01:29:44.651964+0000) 2024-03-19T01:29:49.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:49 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:49.003+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:49.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:49 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:49.003+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:49.475 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:49 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:49.003+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:49.795 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:49 smithi116 podman[124124]: 2024-03-19 01:29:49.424808392 +0000 UTC m=+0.019179007 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:29:49.796 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:49 smithi116 podman[124124]: 2024-03-19 01:29:49.551501064 +0000 UTC m=+0.145871717 container create bf6f49d0a2da99db2849ebb383d7670436b0a7140e5752302d363b8902846af0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-19T01:29:49.796 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:49 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:49.529+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:49.796 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:49 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:49.529+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:49.796 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:49 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:49.529+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:49.796 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:49 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:49.529+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:29:50.022 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:49 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:49.753+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:50.022 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:49 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:49.753+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:50.023 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:49 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:49.753+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:50.092 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:49 smithi116 podman[124124]: 2024-03-19 01:29:49.79591259 +0000 UTC m=+0.390283202 container init bf6f49d0a2da99db2849ebb383d7670436b0a7140e5752302d363b8902846af0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-19T01:29:50.093 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:49 smithi116 podman[124124]: 2024-03-19 01:29:49.799069007 +0000 UTC m=+0.393439623 container start bf6f49d0a2da99db2849ebb383d7670436b0a7140e5752302d363b8902846af0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-19T01:29:50.093 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:49 smithi116 podman[124124]: 2024-03-19 01:29:49.895536898 +0000 UTC m=+0.489907521 container attach bf6f49d0a2da99db2849ebb383d7670436b0a7140e5752302d363b8902846af0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-19T01:29:50.375 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:49 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:49.993+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:50.376 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:49 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:49.993+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:50.376 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:49 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:49.993+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:50.376 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:50 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:50.147+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:50.376 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:50 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:50.147+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:50.376 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:50 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:50.147+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:19.351979+0000 front 2024-03-19T01:29:19.352013+0000 (oldest deadline 2024-03-19T01:29:44.651964+0000) 2024-03-19T01:29:50.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:50 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:50.022+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:50.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:50 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:50.022+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:50.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:50 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:50.022+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:29:50.405 INFO:journalctl@ceph.osd.6.smithi116.stdout:Mar 19 01:29:50 smithi116 podman[124124]: 2024-03-19 01:29:50.093144242 +0000 UTC m=+0.687514858 container died bf6f49d0a2da99db2849ebb383d7670436b0a7140e5752302d363b8902846af0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-19T01:29:50.732 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:50 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:50.542+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:50.732 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:50 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:50.542+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:50.732 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:50 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:50.542+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:50.732 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:50 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:50.542+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:29:50.752 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.6.service' 2024-03-19T01:29:50.796 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:29:50.797 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-19T01:29:50.798 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-19T01:29:50.798 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.7 2024-03-19T01:29:50.974 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:50 smithi156 systemd[1]: Stopping Ceph osd.7 for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:29:51.066 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:50 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:50.730+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:51.066 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:50 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:50.730+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:51.066 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:50 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:50.730+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:51.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:51 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:51.065+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:51.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:51 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:51.065+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:51.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:51 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:51.065+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:29:51.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:51 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:51.034+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:51.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:51 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:51.034+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:51.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:51 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:51.034+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:51.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:51 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:51.090+0000 7fa815d9d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:29:51.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:51 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:51.090+0000 7fa815d9d700 -1 osd.7 90 *** Got signal Terminated *** 2024-03-19T01:29:51.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:51 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:51.090+0000 7fa815d9d700 -1 osd.7 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T01:29:51.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:51 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:51.162+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:51.475 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:51 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:51.162+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:51.475 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:51 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:51.162+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:19.351979+0000 front 2024-03-19T01:29:19.352013+0000 (oldest deadline 2024-03-19T01:29:44.651964+0000) 2024-03-19T01:29:51.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:51 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:51.554+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:51.935 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:51 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:51.554+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:51.935 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:51 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:51.554+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:51.936 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:51 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:51.554+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:29:52.024 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:51 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:51.740+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:52.025 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:51 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:51.740+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:52.025 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:51 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:51.740+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:52.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:52 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:52.024+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:52.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:52 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:52.024+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:52.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:52 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:52.024+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:29:52.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:52 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:52.015+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:52.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:52 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:52.015+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:52.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:52 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:52.015+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:52.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:52 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:52.152+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:52.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:52 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:52.152+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:52.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:52 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:52.152+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:19.351979+0000 front 2024-03-19T01:29:19.352013+0000 (oldest deadline 2024-03-19T01:29:44.651964+0000) 2024-03-19T01:29:52.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:52 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:52.595+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:52.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:52 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:52.595+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:52.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:52 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:52.595+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:52.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:52 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:52.595+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:29:53.041 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:52 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:52.706+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:53.042 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:52 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:52.706+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:53.042 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:52 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:52.706+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:53.042 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:52 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:52.706+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:29:53.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:53 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:53.041+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:53.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:53 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:53.041+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:53.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:53 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:53.041+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:29:53.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:53 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:53.188+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:53.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:53 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:53.188+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:53.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:53 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:53.188+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:19.351979+0000 front 2024-03-19T01:29:19.352013+0000 (oldest deadline 2024-03-19T01:29:44.651964+0000) 2024-03-19T01:29:53.475 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:53 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:53.022+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:53.475 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:53 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:53.022+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:53.475 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:53 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:53.022+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:53.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:53 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:53.632+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:53.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:53 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:53.632+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:53.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:53 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:53.632+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:53.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:53 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:53.632+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:29:54.077 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:53 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:53.687+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:54.077 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:53 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:53.687+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:54.077 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:53 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:53.687+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:54.077 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:53 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:53.687+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:29:54.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:54 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:54.075+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:54.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:54 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:54.075+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:54.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:54 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:54.075+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:29:54.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:54 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:54.236+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:54.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:54 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:54.236+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:54.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:54 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:54.236+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:19.351979+0000 front 2024-03-19T01:29:19.352013+0000 (oldest deadline 2024-03-19T01:29:44.651964+0000) 2024-03-19T01:29:54.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:54 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:54.003+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:54.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:54 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:54.003+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:54.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:54 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:54.003+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:54.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:54 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:54.643+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:54.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:54 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:54.643+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:54.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:54 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:54.643+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:54.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:54 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:54.643+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:29:54.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:54 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:54.609+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:54.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:54 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:54.609+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:54.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:54 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:54.609+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:54.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:54 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:54.609+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:29:55.275 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:55 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:55.015+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:55.275 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:55 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:55.015+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:55.275 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:55 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:55.015+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:55.275 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:55 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:55.015+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:29:55.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:55 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:55.089+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:55.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:55 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:55.089+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:55.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:55 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:55.089+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:29:55.724 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:55 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:55.273+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:03.048806+0000 front 2024-03-19T01:29:03.049034+0000 (oldest deadline 2024-03-19T01:29:28.349046+0000) 2024-03-19T01:29:55.724 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:55 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:55.274+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.251555+0000 front 2024-03-19T01:29:15.251455+0000 (oldest deadline 2024-03-19T01:29:39.351348+0000) 2024-03-19T01:29:55.724 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:55 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7[85262]: 2024-03-19T01:29:55.274+0000 7fa80e389700 -1 osd.7 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:19.351979+0000 front 2024-03-19T01:29:19.352013+0000 (oldest deadline 2024-03-19T01:29:44.651964+0000) 2024-03-19T01:29:55.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:55 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:55.621+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:55.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:55 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:55.621+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:55.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:55 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:55.621+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:55.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:55 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:55.621+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:29:55.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:55 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:55.606+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:55.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:55 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:55.606+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:55.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:55 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:55.606+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:55.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:55 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:55.606+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:29:56.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:56 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:56.072+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:56.400 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:56 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:56.072+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:56.400 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:56 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:56.072+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:29:56.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:56 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:56.009+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:56.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:56 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:56.009+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:56.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:56 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:56.009+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:56.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:56 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:56.009+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:29:56.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:56 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:56.615+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:56.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:56 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:56.615+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:56.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:56 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:56.615+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:56.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:56 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:56.615+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:29:56.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:56 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:56.631+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:56.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:56 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:56.631+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:56.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:56 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:56.631+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:56.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:56 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:56.631+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:29:56.965 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:56 smithi156 podman[103412]: 2024-03-19 01:29:56.697438082 +0000 UTC m=+5.724012619 container died aacc5bb634330e5a286feac6ed30a8c6fbf96e54eb51a143afdde7c4b4b3b7da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-19T01:29:57.216 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:56 smithi156 podman[103412]: 2024-03-19 01:29:56.973276619 +0000 UTC m=+5.999851136 container cleanup aacc5bb634330e5a286feac6ed30a8c6fbf96e54eb51a143afdde7c4b4b3b7da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-19T01:29:57.216 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:56 smithi156 bash[103412]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7 2024-03-19T01:29:57.216 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:56 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:56.964+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:57.217 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:56 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:56.964+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:57.217 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:56 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:56.964+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:57.217 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:56 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:56.964+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:29:57.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:57 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:57.105+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:57.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:57 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:57.105+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:57.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:57 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:57.105+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:29:57.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:57 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:57.105+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:26.677007+0000 front 2024-03-19T01:29:26.677021+0000 (oldest deadline 2024-03-19T01:29:56.077543+0000) 2024-03-19T01:29:57.474 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:57 smithi156 podman[103432]: 2024-03-19 01:29:57.216115487 +0000 UTC m=+0.520484647 container remove aacc5bb634330e5a286feac6ed30a8c6fbf96e54eb51a143afdde7c4b4b3b7da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:29:57.786 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:57 smithi156 podman[103529]: 2024-03-19 01:29:57.43590269 +0000 UTC m=+0.017976000 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:29:57.786 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:57 smithi156 podman[103529]: 2024-03-19 01:29:57.533564041 +0000 UTC m=+0.115637348 container create 55fdb3ccf05fb58aaa8ad0c426ecd4be77d82c2d594153ea6338756ac175882b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T01:29:57.786 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:57 smithi156 podman[103529]: 2024-03-19 01:29:57.785982504 +0000 UTC m=+0.368055813 container init 55fdb3ccf05fb58aaa8ad0c426ecd4be77d82c2d594153ea6338756ac175882b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=) 2024-03-19T01:29:57.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:57 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:57.574+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:57.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:57 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:57.574+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:57.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:57 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:57.574+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:57.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:57 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:57.574+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:29:57.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:57 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:57.622+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:57.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:57 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:57.622+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:57.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:57 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:57.622+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:57.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:57 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:57.622+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:29:58.077 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:57 smithi156 podman[103529]: 2024-03-19 01:29:57.789437809 +0000 UTC m=+0.371511116 container start 55fdb3ccf05fb58aaa8ad0c426ecd4be77d82c2d594153ea6338756ac175882b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-19T01:29:58.077 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:57 smithi156 podman[103529]: 2024-03-19 01:29:57.903179896 +0000 UTC m=+0.485253211 container attach 55fdb3ccf05fb58aaa8ad0c426ecd4be77d82c2d594153ea6338756ac175882b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-19T01:29:58.077 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:57 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:57.946+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:58.077 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:57 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:57.946+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:58.077 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:57 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:57.946+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:58.077 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:57 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:57.946+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:29:58.381 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:58 smithi156 podman[103529]: 2024-03-19 01:29:58.076902718 +0000 UTC m=+0.658976038 container died 55fdb3ccf05fb58aaa8ad0c426ecd4be77d82c2d594153ea6338756ac175882b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T01:29:58.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:58 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:58.116+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:58.400 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:58 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:58.116+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:58.400 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:58 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:58.116+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:29:58.400 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:58 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:58.116+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:26.677007+0000 front 2024-03-19T01:29:26.677021+0000 (oldest deadline 2024-03-19T01:29:56.077543+0000) 2024-03-19T01:29:58.633 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:58 smithi156 podman[103555]: 2024-03-19 01:29:58.61496487 +0000 UTC m=+0.532583324 container remove 55fdb3ccf05fb58aaa8ad0c426ecd4be77d82c2d594153ea6338756ac175882b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-7-deactivate, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-19T01:29:58.633 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:58 smithi156 systemd[1]: ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-19T01:29:58.633 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:58 smithi156 systemd[1]: Stopped Ceph osd.7 for 35912c14-e58c-11ee-95c9-87774f69a715. 2024-03-19T01:29:58.633 INFO:journalctl@ceph.osd.7.smithi156.stdout:Mar 19 01:29:58 smithi156 systemd[1]: ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.7.service: Consumed 10min 24.932s CPU time. 2024-03-19T01:29:58.635 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.7.service' 2024-03-19T01:29:58.675 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:29:58.676 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-19T01:29:58.676 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-19T01:29:58.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.8 2024-03-19T01:29:58.899 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:58 smithi162 systemd[1]: Stopping Ceph osd.8 for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:29:58.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:58 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:58.540+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:58.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:58 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:58.540+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:58.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:58 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:58.540+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:58.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:58 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:58.540+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:29:58.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:58 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:58.629+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:58.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:58 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:58.629+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:58.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:58 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:58.629+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:58.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:58 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:58.629+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:29:59.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:58 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:58.921+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:29:59.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:58 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:58.921+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:29:59.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:58 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:58.921+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:29:59.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:58 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:58.921+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:29:59.284 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:58 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:58.980+0000 7f6f8e952700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:29:59.285 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:58 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:58.980+0000 7f6f8e952700 -1 osd.8 90 *** Got signal Terminated *** 2024-03-19T01:29:59.285 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:58 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:58.980+0000 7f6f8e952700 -1 osd.8 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T01:29:59.285 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:59 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:59.101+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:29:59.285 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:59 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:59.101+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:29:59.285 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:59 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:59.101+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:29:59.285 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:29:59 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:29:59.101+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:26.677007+0000 front 2024-03-19T01:29:26.677021+0000 (oldest deadline 2024-03-19T01:29:56.077543+0000) 2024-03-19T01:29:59.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:59 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:59.587+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:29:59.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:59 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:59.587+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:29:59.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:59 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:59.587+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:29:59.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:29:59 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:29:59.587+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:29:59.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:59 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:59.650+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:29:59.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:59 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:59.650+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:29:59.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:59 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:59.650+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:29:59.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:29:59 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:29:59.650+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:30:00.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:59 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:59.961+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:00.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:59 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:59.961+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:00.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:59 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:59.962+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:00.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:29:59 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:29:59.962+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:00.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:00 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:00.084+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:30:00.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:00 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:00.084+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:30:00.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:00 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:00.084+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:30:00.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:00 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:00.084+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:26.677007+0000 front 2024-03-19T01:29:26.677021+0000 (oldest deadline 2024-03-19T01:29:56.077543+0000) 2024-03-19T01:30:00.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:00 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:00.593+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:00.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:00 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:00.593+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:00.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:00 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:00.593+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:00.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:00 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:00.593+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:01.077 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:00 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:00.691+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:30:01.077 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:00 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:00.691+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:30:01.077 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:00 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:00.691+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:30:01.077 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:00 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:00.691+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:30:01.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:01 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:01.039+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:30:01.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:01 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:01.039+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:30:01.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:01 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:01.039+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:30:01.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:01 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:01.039+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:26.677007+0000 front 2024-03-19T01:29:26.677021+0000 (oldest deadline 2024-03-19T01:29:56.077543+0000) 2024-03-19T01:30:01.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:00 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:00.990+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:01.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:00 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:00.990+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:01.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:00 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:00.990+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:01.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:00 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:00.990+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:01.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:01 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:01.609+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:01.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:01 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:01.609+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:01.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:01 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:01.609+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:01.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:01 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:01.609+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:01.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:01 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:01.653+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:30:01.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:01 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:01.653+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:30:01.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:01 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:01.653+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:30:01.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:01 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:01.653+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:30:02.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:01 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:01.989+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:30:02.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:01 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:01.989+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:30:02.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:01 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:01.989+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:30:02.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:01 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:01.989+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:26.677007+0000 front 2024-03-19T01:29:26.677021+0000 (oldest deadline 2024-03-19T01:29:56.077543+0000) 2024-03-19T01:30:02.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:02 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:02.014+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:02.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:02 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:02.014+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:02.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:02 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:02.014+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:02.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:02 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:02.014+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:02.475 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:02 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:02.014+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:02.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:02 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:02.645+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:02.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:02 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:02.645+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:02.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:02 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:02.645+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:02.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:02 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:02.645+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:02.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:02 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:02.611+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:30:02.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:02 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:02.611+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:30:02.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:02 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:02.611+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:30:02.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:02 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:02.611+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:30:03.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:02 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:02.963+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:30:03.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:02 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:02.963+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:30:03.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:02 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:02.963+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:30:03.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:02 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:02.963+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:26.677007+0000 front 2024-03-19T01:29:26.677021+0000 (oldest deadline 2024-03-19T01:29:56.077543+0000) 2024-03-19T01:30:03.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:02 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:02.974+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:03.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:02 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:02.974+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:03.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:02 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:02.974+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:03.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:02 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:02.974+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:03.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:02 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:02.974+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:03.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:03 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:03.646+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:03.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:03 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:03.646+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:03.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:03 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:03.646+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:03.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:03 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:03.646+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:03.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:03 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:03.646+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:03.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:03 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:03.565+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:30:03.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:03 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:03.565+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:30:03.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:03 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:03.565+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:30:03.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:03 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:03.565+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:30:03.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:03 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:03.565+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.053271+0000 front 2024-03-19T01:29:38.053345+0000 (oldest deadline 2024-03-19T01:30:02.653780+0000) 2024-03-19T01:30:04.400 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:03 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:03.973+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.372974+0000 front 2024-03-19T01:29:06.372765+0000 (oldest deadline 2024-03-19T01:29:29.872772+0000) 2024-03-19T01:30:04.400 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:03 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:03.973+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.975135+0000 front 2024-03-19T01:29:14.975168+0000 (oldest deadline 2024-03-19T01:29:39.675261+0000) 2024-03-19T01:30:04.400 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:03 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:03.973+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:21.976260+0000 front 2024-03-19T01:29:21.976468+0000 (oldest deadline 2024-03-19T01:29:46.676311+0000) 2024-03-19T01:30:04.400 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:03 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:03.973+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:26.677007+0000 front 2024-03-19T01:29:26.677021+0000 (oldest deadline 2024-03-19T01:29:56.077543+0000) 2024-03-19T01:30:04.400 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:03 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8[85597]: 2024-03-19T01:30:03.973+0000 7f6f86f3e700 -1 osd.8 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.378558+0000 front 2024-03-19T01:29:38.378589+0000 (oldest deadline 2024-03-19T01:30:03.078760+0000) 2024-03-19T01:30:04.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:04 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:04.004+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:04.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:04 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:04.004+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:04.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:04 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:04.004+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:04.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:04 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:04.004+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:04.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:04 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:04.004+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:04.714 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:04 smithi162 podman[104505]: 2024-03-19 01:30:04.446002245 +0000 UTC m=+5.572253688 container died 203f3c6d51be5642ff013cc6dfbf2afc7debe94a5cd7e6f99d7ae42625fd5477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS) 2024-03-19T01:30:04.715 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:04 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:04.615+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:04.715 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:04 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:04.615+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:04.715 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:04 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:04.615+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:04.715 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:04 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:04.615+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:04.715 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:04 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:04.615+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:04.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:04 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:04.589+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:30:04.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:04 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:04.589+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:30:04.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:04 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:04.589+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:30:04.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:04 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:04.589+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:30:04.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:04 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:04.589+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.053271+0000 front 2024-03-19T01:29:38.053345+0000 (oldest deadline 2024-03-19T01:30:02.653780+0000) 2024-03-19T01:30:05.081 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:04 smithi162 podman[104505]: 2024-03-19 01:30:04.81351552 +0000 UTC m=+5.939766972 container cleanup 203f3c6d51be5642ff013cc6dfbf2afc7debe94a5cd7e6f99d7ae42625fd5477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:30:05.082 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:04 smithi162 bash[104505]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8 2024-03-19T01:30:05.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:04 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:04.972+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:05.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:04 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:04.972+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:05.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:04 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:04.972+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:05.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:04 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:04.972+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:05.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:04 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:04.972+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:05.399 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:05 smithi162 podman[104524]: 2024-03-19 01:30:05.081519938 +0000 UTC m=+0.632653544 container remove 203f3c6d51be5642ff013cc6dfbf2afc7debe94a5cd7e6f99d7ae42625fd5477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-19T01:30:05.676 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:05 smithi162 podman[104620]: 2024-03-19 01:30:05.327236584 +0000 UTC m=+0.018092411 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:30:05.677 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:05 smithi162 podman[104620]: 2024-03-19 01:30:05.432294631 +0000 UTC m=+0.123150458 container create 5525e0d3702c1d77ddd937a8b5157716a39027715f05d58fd02585b893377ec4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-19T01:30:05.677 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:05 smithi162 podman[104620]: 2024-03-19 01:30:05.676538343 +0000 UTC m=+0.367394159 container init 5525e0d3702c1d77ddd937a8b5157716a39027715f05d58fd02585b893377ec4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T01:30:05.677 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:05 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:05.652+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:05.677 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:05 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:05.652+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:05.677 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:05 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:05.652+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:05.677 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:05 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:05.652+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:05.677 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:05 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:05.652+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:05.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:05.553+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:30:05.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:05.553+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:30:05.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:05.553+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:30:05.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:05.553+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:30:05.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:05 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:05.553+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.053271+0000 front 2024-03-19T01:29:38.053345+0000 (oldest deadline 2024-03-19T01:30:02.653780+0000) 2024-03-19T01:30:05.969 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:05 smithi162 podman[104620]: 2024-03-19 01:30:05.680186376 +0000 UTC m=+0.371042207 container start 5525e0d3702c1d77ddd937a8b5157716a39027715f05d58fd02585b893377ec4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-19T01:30:05.969 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:05 smithi162 podman[104620]: 2024-03-19 01:30:05.784797048 +0000 UTC m=+0.475652880 container attach 5525e0d3702c1d77ddd937a8b5157716a39027715f05d58fd02585b893377ec4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-19T01:30:06.304 INFO:journalctl@ceph.osd.8.smithi162.stdout:Mar 19 01:30:05 smithi162 podman[104620]: 2024-03-19 01:30:05.969306483 +0000 UTC m=+0.660162319 container died 5525e0d3702c1d77ddd937a8b5157716a39027715f05d58fd02585b893377ec4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-19T01:30:06.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:05 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:05.981+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:06.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:05 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:05.981+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:06.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:05 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:05.981+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:06.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:05 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:05.981+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:06.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:05 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:05.981+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:06.612 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.8.service' 2024-03-19T01:30:06.655 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:30:06.655 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-19T01:30:06.655 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-19T01:30:06.655 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.9 2024-03-19T01:30:06.865 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:06 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:06.679+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:06.865 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:06 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:06.679+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:06.865 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:06 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:06.679+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:06.865 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:06 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:06.679+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:06.865 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:06 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:06.679+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:06.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:06 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:06.510+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:30:06.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:06 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:06.510+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:30:06.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:06 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:06.510+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:30:06.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:06 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:06.510+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:30:06.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:06 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:06.510+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.053271+0000 front 2024-03-19T01:29:38.053345+0000 (oldest deadline 2024-03-19T01:30:02.653780+0000) 2024-03-19T01:30:06.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:06 smithi116 systemd[1]: Stopping Ceph osd.9 for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:30:07.405 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:06 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:06.976+0000 7fd930176700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:30:07.405 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:06 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:06.976+0000 7fd930176700 -1 osd.9 90 *** Got signal Terminated *** 2024-03-19T01:30:07.405 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:06 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:06.976+0000 7fd930176700 -1 osd.9 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T01:30:07.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:06 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:06.988+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:07.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:06 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:06.988+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:07.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:06 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:06.988+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:07.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:06 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:06.988+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:07.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:06 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:06.988+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:07.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:07 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:07.491+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:30:07.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:07 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:07.491+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:30:07.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:07 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:07.491+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:30:07.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:07 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:07.491+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:30:07.906 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:07 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:07.491+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.053271+0000 front 2024-03-19T01:29:38.053345+0000 (oldest deadline 2024-03-19T01:30:02.653780+0000) 2024-03-19T01:30:08.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:07 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:07.655+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:08.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:07 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:07.655+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:08.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:07 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:07.655+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:08.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:07 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:07.655+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:08.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:07 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:07.655+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:08.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:07 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:07.967+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:08.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:07 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:07.967+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:08.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:07 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:07.967+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:08.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:07 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:07.967+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:08.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:07 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:07.967+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:08.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:08 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:08.624+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:08.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:08 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:08.624+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:08.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:08 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:08.624+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:08.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:08 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:08.624+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:08.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:08 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:08.624+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:08.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:08 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:08.514+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:30:08.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:08 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:08.514+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:30:08.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:08 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:08.514+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:30:08.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:08 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:08.514+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:30:08.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:08 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:08.514+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.053271+0000 front 2024-03-19T01:29:38.053345+0000 (oldest deadline 2024-03-19T01:30:02.653780+0000) 2024-03-19T01:30:09.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:08 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:08.997+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:09.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:08 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:08.997+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:09.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:08 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:08.997+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:09.475 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:08 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:08.997+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:09.475 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:08 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:08.997+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:09.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:09 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:09.625+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:09.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:09 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:09.625+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:09.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:09 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:09.625+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:09.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:09 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:09.625+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:09.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:09 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:09.625+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:09.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:09 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:09.528+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:30:09.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:09 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:09.528+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:30:09.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:09 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:09.528+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:30:09.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:09 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:09.528+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:30:09.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:09 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:09.528+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.053271+0000 front 2024-03-19T01:29:38.053345+0000 (oldest deadline 2024-03-19T01:30:02.653780+0000) 2024-03-19T01:30:10.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:10 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:10.009+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:10.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:10 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:10.009+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:10.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:10 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:10.009+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:10.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:10 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:10.009+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:10.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:10 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:10.009+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:10.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:10 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:10.009+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:43.916011+0000 front 2024-03-19T01:29:43.916188+0000 (oldest deadline 2024-03-19T01:30:09.215957+0000) 2024-03-19T01:30:10.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:10 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:10.536+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:30:10.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:10 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:10.536+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:30:10.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:10 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:10.536+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:30:10.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:10 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:10.536+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:30:10.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:10 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:10.536+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.053271+0000 front 2024-03-19T01:29:38.053345+0000 (oldest deadline 2024-03-19T01:30:02.653780+0000) 2024-03-19T01:30:11.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:10 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:10.656+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:11.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:10 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:10.656+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:11.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:10 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:10.656+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:11.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:10 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:10.656+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:11.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:10 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:10.656+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:11.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:10 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:10.977+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:11.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:10 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:10.977+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:11.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:10 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:10.977+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:11.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:10 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:10.977+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:11.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:10 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:10.977+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:11.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:10 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:10.977+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:43.916011+0000 front 2024-03-19T01:29:43.916188+0000 (oldest deadline 2024-03-19T01:30:09.215957+0000) 2024-03-19T01:30:11.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:11 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:11.632+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:11.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:11 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:11.632+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:11.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:11 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:11.632+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:11.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:11 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:11.632+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:11.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:11 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:11.632+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:11.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:11 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:11.632+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:11.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:11 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:11.557+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:56.845745+0000 front 2024-03-19T01:28:56.845721+0000 (oldest deadline 2024-03-19T01:29:20.945725+0000) 2024-03-19T01:30:11.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:11 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:11.557+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.047609+0000 front 2024-03-19T01:29:06.047540+0000 (oldest deadline 2024-03-19T01:29:29.547623+0000) 2024-03-19T01:30:11.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:11 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:11.557+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:09.548333+0000 front 2024-03-19T01:29:09.548286+0000 (oldest deadline 2024-03-19T01:29:36.548828+0000) 2024-03-19T01:30:11.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:11 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:11.557+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:23.550437+0000 front 2024-03-19T01:29:23.550387+0000 (oldest deadline 2024-03-19T01:29:46.450607+0000) 2024-03-19T01:30:11.905 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:11 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9[97487]: 2024-03-19T01:30:11.557+0000 7fd92c76a700 -1 osd.9 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.053271+0000 front 2024-03-19T01:29:38.053345+0000 (oldest deadline 2024-03-19T01:30:02.653780+0000) 2024-03-19T01:30:12.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:11 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:11.943+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:12.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:11 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:11.943+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:12.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:11 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:11.943+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:12.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:11 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:11.943+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:12.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:11 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:11.943+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:12.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:11 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:11.943+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:43.916011+0000 front 2024-03-19T01:29:43.916188+0000 (oldest deadline 2024-03-19T01:30:09.215957+0000) 2024-03-19T01:30:12.810 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:12 smithi116 podman[124243]: 2024-03-19 01:30:12.546031858 +0000 UTC m=+5.690845474 container died f5159dbddb5ba261954ba44ee8a73448fff8e40e35abda9563cc6b36ab1f4193 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:30:12.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:12 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:12.618+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:12.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:12 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:12.618+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:12.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:12 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:12.618+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:12.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:12 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:12.618+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:12.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:12 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:12.618+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:12.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:12 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:12.618+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:13.155 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:12 smithi116 podman[124243]: 2024-03-19 01:30:12.91781444 +0000 UTC m=+6.062628063 container cleanup f5159dbddb5ba261954ba44ee8a73448fff8e40e35abda9563cc6b36ab1f4193 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-19T01:30:13.155 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:12 smithi116 bash[124243]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9 2024-03-19T01:30:13.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:12 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:12.895+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:13.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:12 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:12.895+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:13.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:12 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:12.895+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:13.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:12 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:12.895+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:13.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:12 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:12.895+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:13.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:12 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:12.895+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:43.916011+0000 front 2024-03-19T01:29:43.916188+0000 (oldest deadline 2024-03-19T01:30:09.215957+0000) 2024-03-19T01:30:13.536 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:13 smithi116 podman[124262]: 2024-03-19 01:30:13.177536295 +0000 UTC m=+0.635200468 container remove f5159dbddb5ba261954ba44ee8a73448fff8e40e35abda9563cc6b36ab1f4193 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-19T01:30:13.822 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:13 smithi116 podman[124353]: 2024-03-19 01:30:13.439374912 +0000 UTC m=+0.017491549 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:30:13.822 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:13 smithi116 podman[124353]: 2024-03-19 01:30:13.553225998 +0000 UTC m=+0.131342678 container create 2edf3f4a035e23a92275c000695b501cd3265d2e50397b5b322d31d6373d3a2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-19T01:30:13.822 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:13 smithi116 podman[124353]: 2024-03-19 01:30:13.747269746 +0000 UTC m=+0.325386398 container init 2edf3f4a035e23a92275c000695b501cd3265d2e50397b5b322d31d6373d3a2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-19T01:30:13.822 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:13 smithi116 podman[124353]: 2024-03-19 01:30:13.750676084 +0000 UTC m=+0.328792832 container start 2edf3f4a035e23a92275c000695b501cd3265d2e50397b5b322d31d6373d3a2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-19T01:30:13.822 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:13 smithi116 podman[124353]: 2024-03-19 01:30:13.822150102 +0000 UTC m=+0.400266756 container attach 2edf3f4a035e23a92275c000695b501cd3265d2e50397b5b322d31d6373d3a2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-19T01:30:13.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:13 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:13.628+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:13.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:13 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:13.628+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:13.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:13 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:13.628+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:13.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:13 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:13.628+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:13.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:13 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:13.628+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:13.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:13 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:13.628+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:14.155 INFO:journalctl@ceph.osd.9.smithi116.stdout:Mar 19 01:30:14 smithi116 podman[124353]: 2024-03-19 01:30:14.039614797 +0000 UTC m=+0.617731436 container died 2edf3f4a035e23a92275c000695b501cd3265d2e50397b5b322d31d6373d3a2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-9-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-19T01:30:14.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:13 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:13.856+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:14.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:13 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:13.856+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:14.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:13 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:13.856+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:14.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:13 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:13.856+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:14.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:13 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:13.856+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:14.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:13 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:13.856+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:43.916011+0000 front 2024-03-19T01:29:43.916188+0000 (oldest deadline 2024-03-19T01:30:09.215957+0000) 2024-03-19T01:30:14.610 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.9.service' 2024-03-19T01:30:14.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:30:14.651 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-19T01:30:14.651 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-19T01:30:14.651 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.10 2024-03-19T01:30:14.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:14 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:14.646+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:14.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:14 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:14.646+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:14.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:14 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:14.646+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:14.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:14 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:14.646+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:14.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:14 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:14.646+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:14.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:14 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:14.646+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:14.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:14 smithi156 systemd[1]: Stopping Ceph osd.10 for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:30:14.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:14 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:14.877+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:14.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:14 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:14.877+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:14.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:14 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:14.877+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:14.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:14 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:14.877+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:14.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:14 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:14.877+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:14.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:14 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:14.877+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:43.916011+0000 front 2024-03-19T01:29:43.916188+0000 (oldest deadline 2024-03-19T01:30:09.215957+0000) 2024-03-19T01:30:15.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:14 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:14.975+0000 7f8a532cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:30:15.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:14 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:14.975+0000 7f8a532cb700 -1 osd.10 90 *** Got signal Terminated *** 2024-03-19T01:30:15.474 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:14 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:14.975+0000 7f8a532cb700 -1 osd.10 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T01:30:15.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:15 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:15.636+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:15.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:15 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:15.636+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:15.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:15 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:15.636+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:15.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:15 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:15.636+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:15.899 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:15 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:15.636+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:15.900 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:15 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:15.636+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:16.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:15 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:15.832+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:16.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:15 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:15.832+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:16.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:15 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:15.832+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:16.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:15 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:15.832+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:16.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:15 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:15.832+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:16.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:15 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:15.832+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:43.916011+0000 front 2024-03-19T01:29:43.916188+0000 (oldest deadline 2024-03-19T01:30:09.215957+0000) 2024-03-19T01:30:17.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:16 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:16.665+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:17.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:16 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:16.665+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:17.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:16 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:16.665+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:17.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:16 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:16.665+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:17.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:16 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:16.665+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:17.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:16 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:16.665+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:17.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:16 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:16.811+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:17.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:16 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:16.811+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:17.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:16 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:16.811+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:17.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:16 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:16.811+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:17.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:16 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:16.811+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:17.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:16 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:16.811+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:43.916011+0000 front 2024-03-19T01:29:43.916188+0000 (oldest deadline 2024-03-19T01:30:09.215957+0000) 2024-03-19T01:30:18.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:17 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:17.673+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:18.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:17 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:17.673+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:18.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:17 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:17.673+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:18.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:17 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:17.673+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:18.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:17 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:17.673+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:18.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:17 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:17.673+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:18.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:17 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:17.673+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.301441+0000 front 2024-03-19T01:29:53.301347+0000 (oldest deadline 2024-03-19T01:30:16.801179+0000) 2024-03-19T01:30:18.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:17 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:17.801+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:18.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:17 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:17.802+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:18.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:17 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:17.802+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:18.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:17 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:17.802+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:18.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:17 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:17.802+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:18.225 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:17 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:17.802+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:43.916011+0000 front 2024-03-19T01:29:43.916188+0000 (oldest deadline 2024-03-19T01:30:09.215957+0000) 2024-03-19T01:30:18.225 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:17 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:17.802+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.316998+0000 front 2024-03-19T01:29:53.316950+0000 (oldest deadline 2024-03-19T01:30:16.817010+0000) 2024-03-19T01:30:19.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:18 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:18.654+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:19.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:18 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:18.654+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:19.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:18 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:18.654+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:19.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:18 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:18.654+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:19.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:18 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:18.654+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:19.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:18 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:18.654+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:19.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:18 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:18.654+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.301441+0000 front 2024-03-19T01:29:53.301347+0000 (oldest deadline 2024-03-19T01:30:16.801179+0000) 2024-03-19T01:30:19.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:18 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:18.753+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:19.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:18 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:18.753+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:19.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:18 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:18.753+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:19.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:18 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:18.753+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:19.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:18 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:18.753+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:19.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:18 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:18.753+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:43.916011+0000 front 2024-03-19T01:29:43.916188+0000 (oldest deadline 2024-03-19T01:30:09.215957+0000) 2024-03-19T01:30:19.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:18 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:18.753+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.316998+0000 front 2024-03-19T01:29:53.316950+0000 (oldest deadline 2024-03-19T01:30:16.817010+0000) 2024-03-19T01:30:19.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:19 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:19.710+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:06.708910+0000 front 2024-03-19T01:29:06.708846+0000 (oldest deadline 2024-03-19T01:29:32.009063+0000) 2024-03-19T01:30:19.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:19 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:19.710+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:14.810861+0000 front 2024-03-19T01:29:14.810795+0000 (oldest deadline 2024-03-19T01:29:40.011680+0000) 2024-03-19T01:30:19.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:19 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:19.710+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6814 osd.3 since back 2024-03-19T01:29:20.012239+0000 front 2024-03-19T01:29:20.012345+0000 (oldest deadline 2024-03-19T01:29:44.712227+0000) 2024-03-19T01:30:19.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:19 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:19.710+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:28.713638+0000 front 2024-03-19T01:29:28.713704+0000 (oldest deadline 2024-03-19T01:29:54.013696+0000) 2024-03-19T01:30:19.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:19 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:19.710+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:37.514916+0000 front 2024-03-19T01:29:37.514862+0000 (oldest deadline 2024-03-19T01:30:01.014962+0000) 2024-03-19T01:30:19.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:19 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:19.710+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:43.916011+0000 front 2024-03-19T01:29:43.916188+0000 (oldest deadline 2024-03-19T01:30:09.215957+0000) 2024-03-19T01:30:19.974 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:19 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10[91889]: 2024-03-19T01:30:19.710+0000 7f8a4b8b7700 -1 osd.10 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.316998+0000 front 2024-03-19T01:29:53.316950+0000 (oldest deadline 2024-03-19T01:30:16.817010+0000) 2024-03-19T01:30:20.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:19 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:19.693+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:20.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:19 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:19.693+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:20.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:19 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:19.693+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:20.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:19 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:19.693+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:20.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:19 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:19.693+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:20.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:19 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:19.693+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:20.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:19 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:19.693+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.301441+0000 front 2024-03-19T01:29:53.301347+0000 (oldest deadline 2024-03-19T01:30:16.801179+0000) 2024-03-19T01:30:20.875 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:20 smithi156 podman[103637]: 2024-03-19 01:30:20.599782845 +0000 UTC m=+5.754907484 container died 760672c2ee53fad7049ffb87efd601ba10b77642ff6c3f80f93e78811979e3c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, ceph=True) 2024-03-19T01:30:21.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:20 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:20.660+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:21.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:20 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:20.660+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:21.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:20 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:20.660+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:21.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:20 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:20.660+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:21.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:20 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:20.660+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:21.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:20 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:20.660+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:21.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:20 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:20.660+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.301441+0000 front 2024-03-19T01:29:53.301347+0000 (oldest deadline 2024-03-19T01:30:16.801179+0000) 2024-03-19T01:30:21.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:20 smithi156 podman[103637]: 2024-03-19 01:30:20.983247121 +0000 UTC m=+6.138371761 container cleanup 760672c2ee53fad7049ffb87efd601ba10b77642ff6c3f80f93e78811979e3c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-19T01:30:21.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:20 smithi156 bash[103637]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10 2024-03-19T01:30:21.538 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:21 smithi156 podman[103652]: 2024-03-19 01:30:21.225731927 +0000 UTC m=+0.621941052 container remove 760672c2ee53fad7049ffb87efd601ba10b77642ff6c3f80f93e78811979e3c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-19T01:30:21.821 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:21 smithi156 podman[103750]: 2024-03-19 01:30:21.441222867 +0000 UTC m=+0.018056284 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:30:21.821 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:21 smithi156 podman[103750]: 2024-03-19 01:30:21.55156119 +0000 UTC m=+0.128394600 container create 2010fbee08f51a819dac66e1b398cfb8f63fd663c990f201087123c4549a4bfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-19T01:30:21.821 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:21 smithi156 podman[103750]: 2024-03-19 01:30:21.762567045 +0000 UTC m=+0.339400475 container init 2010fbee08f51a819dac66e1b398cfb8f63fd663c990f201087123c4549a4bfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-19T01:30:21.821 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:21 smithi156 podman[103750]: 2024-03-19 01:30:21.76563674 +0000 UTC m=+0.342470161 container start 2010fbee08f51a819dac66e1b398cfb8f63fd663c990f201087123c4549a4bfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T01:30:21.821 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:21 smithi156 podman[103750]: 2024-03-19 01:30:21.820626146 +0000 UTC m=+0.397459590 container attach 2010fbee08f51a819dac66e1b398cfb8f63fd663c990f201087123c4549a4bfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-19T01:30:22.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:21 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:21.653+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:22.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:21 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:21.653+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:22.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:21 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:21.653+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:22.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:21 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:21.653+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:22.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:21 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:21.653+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:22.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:21 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:21.653+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:22.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:21 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:21.653+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.301441+0000 front 2024-03-19T01:29:53.301347+0000 (oldest deadline 2024-03-19T01:30:16.801179+0000) 2024-03-19T01:30:22.224 INFO:journalctl@ceph.osd.10.smithi156.stdout:Mar 19 01:30:22 smithi156 podman[103750]: 2024-03-19 01:30:22.045948739 +0000 UTC m=+0.622782156 container died 2010fbee08f51a819dac66e1b398cfb8f63fd663c990f201087123c4549a4bfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, ceph=True) 2024-03-19T01:30:22.644 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.10.service' 2024-03-19T01:30:22.685 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:30:22.685 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-19T01:30:22.685 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-19T01:30:22.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.11 2024-03-19T01:30:23.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:22 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:22.682+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:23.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:22 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:22.682+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:23.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:22 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:22.682+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:23.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:22 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:22.682+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:23.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:22 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:22.682+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:23.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:22 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:22.682+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:23.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:22 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:22.682+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.301441+0000 front 2024-03-19T01:29:53.301347+0000 (oldest deadline 2024-03-19T01:30:16.801179+0000) 2024-03-19T01:30:23.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:22 smithi162 systemd[1]: Stopping Ceph osd.11 for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:30:23.649 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:23 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:23.222+0000 7fb7b57ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:30:23.649 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:23 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:23.222+0000 7fb7b57ca700 -1 osd.11 90 *** Got signal Terminated *** 2024-03-19T01:30:23.649 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:23 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:23.222+0000 7fb7b57ca700 -1 osd.11 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-19T01:30:24.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:23 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:23.698+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:24.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:23 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:23.698+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:24.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:23 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:23.698+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:24.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:23 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:23.698+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:24.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:23 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:23.698+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:24.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:23 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:23.698+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:24.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:23 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:23.698+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.301441+0000 front 2024-03-19T01:29:53.301347+0000 (oldest deadline 2024-03-19T01:30:16.801179+0000) 2024-03-19T01:30:25.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:24 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:24.694+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:25.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:24 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:24.694+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:25.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:24 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:24.694+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:25.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:24 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:24.694+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:25.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:24 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:24.694+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:25.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:24 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:24.694+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:25.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:24 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:24.694+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.301441+0000 front 2024-03-19T01:29:53.301347+0000 (oldest deadline 2024-03-19T01:30:16.801179+0000) 2024-03-19T01:30:26.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:25 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:25.738+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:26.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:25 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:25.738+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:26.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:25 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:25.738+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:26.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:25 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:25.738+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:26.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:25 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:25.738+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:26.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:25 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:25.738+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:26.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:25 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:25.738+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.301441+0000 front 2024-03-19T01:29:53.301347+0000 (oldest deadline 2024-03-19T01:30:16.801179+0000) 2024-03-19T01:30:27.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:26 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:26.709+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:27.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:26 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:26.709+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:27.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:26 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:26.709+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:27.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:26 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:26.709+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:27.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:26 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:26.709+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:27.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:26 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:26.709+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:27.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:26 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:26.709+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.301441+0000 front 2024-03-19T01:29:53.301347+0000 (oldest deadline 2024-03-19T01:30:16.801179+0000) 2024-03-19T01:30:27.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:26 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:26.709+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6820 osd.8 since back 2024-03-19T01:30:02.602568+0000 front 2024-03-19T01:30:02.602612+0000 (oldest deadline 2024-03-19T01:30:26.102540+0000) 2024-03-19T01:30:28.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:27 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:27.715+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6806 osd.0 since back 2024-03-19T01:28:53.089949+0000 front 2024-03-19T01:28:53.089829+0000 (oldest deadline 2024-03-19T01:29:24.290455+0000) 2024-03-19T01:30:28.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:27 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:27.715+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6804 osd.1 since back 2024-03-19T01:29:07.191721+0000 front 2024-03-19T01:29:07.191702+0000 (oldest deadline 2024-03-19T01:29:29.992207+0000) 2024-03-19T01:30:28.149 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:27 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:27.715+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6804 osd.2 since back 2024-03-19T01:29:15.193705+0000 front 2024-03-19T01:29:15.193681+0000 (oldest deadline 2024-03-19T01:29:39.893772+0000) 2024-03-19T01:30:28.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:27 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:27.715+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6812 osd.4 since back 2024-03-19T01:29:30.196628+0000 front 2024-03-19T01:29:30.196703+0000 (oldest deadline 2024-03-19T01:29:52.496661+0000) 2024-03-19T01:30:28.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:27 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:27.715+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-03-19T01:29:38.898013+0000 front 2024-03-19T01:29:38.897884+0000 (oldest deadline 2024-03-19T01:30:03.598016+0000) 2024-03-19T01:30:28.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:27 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:27.715+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.116:6822 osd.6 since back 2024-03-19T01:29:47.599762+0000 front 2024-03-19T01:29:47.599815+0000 (oldest deadline 2024-03-19T01:30:11.600203+0000) 2024-03-19T01:30:28.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:27 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:27.715+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.156:6820 osd.7 since back 2024-03-19T01:29:53.301441+0000 front 2024-03-19T01:29:53.301347+0000 (oldest deadline 2024-03-19T01:30:16.801179+0000) 2024-03-19T01:30:28.150 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:27 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11[92501]: 2024-03-19T01:30:27.715+0000 7fb7b1dbe700 -1 osd.11 90 heartbeat_check: no reply from 172.21.15.162:6820 osd.8 since back 2024-03-19T01:30:02.602568+0000 front 2024-03-19T01:30:02.602612+0000 (oldest deadline 2024-03-19T01:30:26.102540+0000) 2024-03-19T01:30:29.090 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:28 smithi162 podman[104749]: 2024-03-19 01:30:28.823631953 +0000 UTC m=+5.943731286 container died 80f1a29a93356692168a8cdae1044aa4e90278aae2b541fe5ed621f32a7212eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-19T01:30:29.399 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:29 smithi162 podman[104749]: 2024-03-19 01:30:29.172024039 +0000 UTC m=+6.292123366 container cleanup 80f1a29a93356692168a8cdae1044aa4e90278aae2b541fe5ed621f32a7212eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-19T01:30:29.399 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:29 smithi162 bash[104749]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11 2024-03-19T01:30:29.765 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:29 smithi162 podman[104766]: 2024-03-19 01:30:29.439795828 +0000 UTC m=+0.615914317 container remove 80f1a29a93356692168a8cdae1044aa4e90278aae2b541fe5ed621f32a7212eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-19T01:30:30.059 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:29 smithi162 podman[104860]: 2024-03-19 01:30:29.765383462 +0000 UTC m=+0.098699594 container create b2f9674f0c4a09bcf9cbe6dc9da12de399fb01fa0d51cb852f1c0ec896936796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-19T01:30:30.060 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:29 smithi162 podman[104860]: 2024-03-19 01:30:29.684178536 +0000 UTC m=+0.017494683 image pull 2312c8e42d0069a2dbb2364bddb00264db4b800e7f18a7e7e4eb8f3ecab2ff1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639 2024-03-19T01:30:30.060 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:29 smithi162 podman[104860]: 2024-03-19 01:30:29.98457731 +0000 UTC m=+0.317893443 container init b2f9674f0c4a09bcf9cbe6dc9da12de399fb01fa0d51cb852f1c0ec896936796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-19T01:30:30.060 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:29 smithi162 podman[104860]: 2024-03-19 01:30:29.987817601 +0000 UTC m=+0.321133731 container start b2f9674f0c4a09bcf9cbe6dc9da12de399fb01fa0d51cb852f1c0ec896936796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-19T01:30:30.060 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:30 smithi162 podman[104860]: 2024-03-19 01:30:30.059564042 +0000 UTC m=+0.392880191 container attach b2f9674f0c4a09bcf9cbe6dc9da12de399fb01fa0d51cb852f1c0ec896936796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11-deactivate, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-19T01:30:30.399 INFO:journalctl@ceph.osd.11.smithi162.stdout:Mar 19 01:30:30 smithi162 podman[104860]: 2024-03-19 01:30:30.277720973 +0000 UTC m=+0.611037098 container died b2f9674f0c4a09bcf9cbe6dc9da12de399fb01fa0d51cb852f1c0ec896936796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-03-19T01:30:30.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@osd.11.service' 2024-03-19T01:30:30.873 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:30:30.873 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-19T01:30:30.873 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-19T01:30:30.873 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.a 2024-03-19T01:30:31.178 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:30:30 smithi116 systemd[1]: Stopping Ceph mds.a for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:30:31.655 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:30:31 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-a[104332]: 2024-03-19T01:30:31.177+0000 7f2779b68700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:30:31.655 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:30:31 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-a[104332]: 2024-03-19T01:30:31.177+0000 7f2779b68700 -1 mds.a *** got signal Terminated *** 2024-03-19T01:30:34.155 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:30:33 smithi116 podman[124472]: 2024-03-19 01:30:33.879177022 +0000 UTC m=+2.810396483 container died ab58a3562cb42f442fb893ada37619575e3ea0a2f202d61debbc0048510da8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-19T01:30:34.436 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:30:34 smithi116 podman[124472]: 2024-03-19 01:30:34.185127065 +0000 UTC m=+3.116346526 container cleanup ab58a3562cb42f442fb893ada37619575e3ea0a2f202d61debbc0048510da8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-a, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T01:30:34.436 INFO:journalctl@ceph.mds.a.smithi116.stdout:Mar 19 01:30:34 smithi116 bash[124472]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-a 2024-03-19T01:30:34.550 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.a.service' 2024-03-19T01:30:34.588 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:30:34.589 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-19T01:30:34.589 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-19T01:30:34.589 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.d 2024-03-19T01:30:34.905 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 01:30:34 smithi116 systemd[1]: Stopping Ceph mds.d for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:30:35.405 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 01:30:34 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-d[103376]: 2024-03-19T01:30:34.913+0000 7f0fc6573700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:30:35.405 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 01:30:34 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-d[103376]: 2024-03-19T01:30:34.913+0000 7f0fc6573700 -1 mds.d *** got signal Terminated *** 2024-03-19T01:30:36.731 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 01:30:36 smithi116 podman[124608]: 2024-03-19 01:30:36.459026444 +0000 UTC m=+1.645167199 container died d66a0cad8ef2e659edd959d578248de9a5a60db789d638175c7062b76bc5a97c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307) 2024-03-19T01:30:37.072 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 01:30:36 smithi116 podman[124608]: 2024-03-19 01:30:36.830232864 +0000 UTC m=+2.016373629 container cleanup d66a0cad8ef2e659edd959d578248de9a5a60db789d638175c7062b76bc5a97c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-d, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-19T01:30:37.072 INFO:journalctl@ceph.mds.d.smithi116.stdout:Mar 19 01:30:36 smithi116 bash[124608]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-d 2024-03-19T01:30:37.191 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.d.service' 2024-03-19T01:30:37.227 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:30:37.228 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-19T01:30:37.229 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-19T01:30:37.229 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.g 2024-03-19T01:30:37.405 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:30:37 smithi116 systemd[1]: Stopping Ceph mds.g for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:30:37.905 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:30:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-g[103976]: 2024-03-19T01:30:37.550+0000 7ff77dfbe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:30:37.905 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:30:37 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-g[103976]: 2024-03-19T01:30:37.550+0000 7ff77dfbe700 -1 mds.g *** got signal Terminated *** 2024-03-19T01:30:41.818 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:30:41 smithi116 podman[124732]: 2024-03-19 01:30:41.555492765 +0000 UTC m=+4.112866607 container died d1da46433e9cb2d1e57f304be45402d1a49c47822d03a8adee7962d05bbc0d14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-19T01:30:42.155 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:30:41 smithi116 podman[124732]: 2024-03-19 01:30:41.908710602 +0000 UTC m=+4.466084444 container cleanup d1da46433e9cb2d1e57f304be45402d1a49c47822d03a8adee7962d05bbc0d14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-g, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:30:42.155 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:30:41 smithi116 bash[124732]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-g 2024-03-19T01:30:42.155 INFO:journalctl@ceph.mds.g.smithi116.stdout:Mar 19 01:30:42 smithi116 podman[124747]: 2024-03-19 01:30:42.068189177 +0000 UTC m=+0.503075155 container remove d1da46433e9cb2d1e57f304be45402d1a49c47822d03a8adee7962d05bbc0d14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-g, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5) 2024-03-19T01:30:42.270 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.g.service' 2024-03-19T01:30:42.305 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:30:42.305 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-19T01:30:42.305 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-19T01:30:42.305 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.j 2024-03-19T01:30:42.604 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 01:30:42 smithi116 systemd[1]: Stopping Ceph mds.j for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:30:42.905 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 01:30:42 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-j[103025]: 2024-03-19T01:30:42.603+0000 7effb60e4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:30:42.905 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 01:30:42 smithi116 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-j[103025]: 2024-03-19T01:30:42.603+0000 7effb60e4700 -1 mds.j *** got signal Terminated *** 2024-03-19T01:30:45.713 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 01:30:45 smithi116 podman[124857]: 2024-03-19 01:30:45.434628625 +0000 UTC m=+2.915165509 container died cd92888c4567cd245e61110af2ccabc38940d753a73bdb4e38247b0752695d29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-19T01:30:46.071 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 01:30:45 smithi116 podman[124857]: 2024-03-19 01:30:45.78724438 +0000 UTC m=+3.267781269 container cleanup cd92888c4567cd245e61110af2ccabc38940d753a73bdb4e38247b0752695d29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-j, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-19T01:30:46.071 INFO:journalctl@ceph.mds.j.smithi116.stdout:Mar 19 01:30:45 smithi116 bash[124857]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-j 2024-03-19T01:30:46.220 DEBUG:teuthology.orchestra.run.smithi116:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.j.service' 2024-03-19T01:30:46.254 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:30:46.255 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-19T01:30:46.255 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-19T01:30:46.255 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.b 2024-03-19T01:30:46.550 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:30:46 smithi156 systemd[1]: Stopping Ceph mds.b for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:30:46.973 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:30:46 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b[97322]: 2024-03-19T01:30:46.549+0000 7f69d0f2e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:30:46.974 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:30:46 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b[97322]: 2024-03-19T01:30:46.549+0000 7f69d0f2e700 -1 mds.b *** got signal Terminated *** 2024-03-19T01:30:50.150 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:30:49 smithi156 podman[103867]: 2024-03-19 01:30:49.890548817 +0000 UTC m=+3.441026000 container died dfeadb187af2ffa2bbb3472d1679dbf281f0da6997143435336f69ba9cc9a961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-03-19T01:30:50.473 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:30:50 smithi156 podman[103867]: 2024-03-19 01:30:50.240866017 +0000 UTC m=+3.791343202 container cleanup dfeadb187af2ffa2bbb3472d1679dbf281f0da6997143435336f69ba9cc9a961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-19T01:30:50.474 INFO:journalctl@ceph.mds.b.smithi156.stdout:Mar 19 01:30:50 smithi156 bash[103867]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-b 2024-03-19T01:30:50.674 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.b.service' 2024-03-19T01:30:50.711 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:30:50.711 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-19T01:30:50.711 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-19T01:30:50.711 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.e 2024-03-19T01:30:51.028 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:30:50 smithi156 systemd[1]: Stopping Ceph mds.e for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:30:51.474 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:30:51 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-e[96690]: 2024-03-19T01:30:51.027+0000 7f36b8a9d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:30:51.474 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:30:51 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-e[96690]: 2024-03-19T01:30:51.027+0000 7f36b8a9d700 -1 mds.e *** got signal Terminated *** 2024-03-19T01:30:54.224 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:30:53 smithi156 podman[103989]: 2024-03-19 01:30:53.877322048 +0000 UTC m=+2.953213879 container died 09e1cff74bc46521e7c80bc70e2375d5487118c817a20a9129084e5241d8685b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:30:54.724 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:30:54 smithi156 podman[103989]: 2024-03-19 01:30:54.536839245 +0000 UTC m=+3.612731073 container cleanup 09e1cff74bc46521e7c80bc70e2375d5487118c817a20a9129084e5241d8685b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-19T01:30:54.724 INFO:journalctl@ceph.mds.e.smithi156.stdout:Mar 19 01:30:54 smithi156 bash[103989]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-e 2024-03-19T01:30:54.962 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.e.service' 2024-03-19T01:30:55.000 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:30:55.000 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-19T01:30:55.000 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-19T01:30:55.000 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.h 2024-03-19T01:30:55.223 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:30:55 smithi156 systemd[1]: Stopping Ceph mds.h for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:30:55.724 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:30:55 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-h[97011]: 2024-03-19T01:30:55.307+0000 7f2045417700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:30:55.724 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:30:55 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-h[97011]: 2024-03-19T01:30:55.307+0000 7f2045417700 -1 mds.h *** got signal Terminated *** 2024-03-19T01:30:58.190 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:30:57 smithi156 podman[104122]: 2024-03-19 01:30:57.89364907 +0000 UTC m=+2.676982383 container died 5c7c19489976d25ee75480dd3049ebeaa730b27905973591990e3defc5c39144 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-19T01:30:58.190 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:30:58 smithi156 podman[104122]: 2024-03-19 01:30:58.190041255 +0000 UTC m=+2.973374567 container cleanup 5c7c19489976d25ee75480dd3049ebeaa730b27905973591990e3defc5c39144 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-19T01:30:58.473 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:30:58 smithi156 bash[104122]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-h 2024-03-19T01:30:58.473 INFO:journalctl@ceph.mds.h.smithi156.stdout:Mar 19 01:30:58 smithi156 podman[104136]: 2024-03-19 01:30:58.432812154 +0000 UTC m=+0.538518652 container remove 5c7c19489976d25ee75480dd3049ebeaa730b27905973591990e3defc5c39144 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-h, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-19T01:30:58.556 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.h.service' 2024-03-19T01:30:58.591 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:30:58.591 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-19T01:30:58.591 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-19T01:30:58.592 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.k 2024-03-19T01:30:58.919 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:30:58 smithi156 systemd[1]: Stopping Ceph mds.k for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:30:59.223 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:30:58 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-k[96373]: 2024-03-19T01:30:58.918+0000 7fc11b49c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:30:59.224 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:30:58 smithi156 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-k[96373]: 2024-03-19T01:30:58.918+0000 7fc11b49c700 -1 mds.k *** got signal Terminated *** 2024-03-19T01:31:02.224 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:31:01 smithi156 podman[104246]: 2024-03-19 01:31:01.869406795 +0000 UTC m=+3.065524924 container died 128f16c9a1babe1a67640a1f0710cde7f1356d1a84ac146e0d1d351fcd3e6674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-19T01:31:02.602 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:31:02 smithi156 podman[104246]: 2024-03-19 01:31:02.326792564 +0000 UTC m=+3.522910681 container cleanup 128f16c9a1babe1a67640a1f0710cde7f1356d1a84ac146e0d1d351fcd3e6674 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-k, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:31:02.602 INFO:journalctl@ceph.mds.k.smithi156.stdout:Mar 19 01:31:02 smithi156 bash[104246]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-k 2024-03-19T01:31:02.750 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.k.service' 2024-03-19T01:31:02.785 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:31:02.785 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-19T01:31:02.785 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-19T01:31:02.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.c 2024-03-19T01:31:03.068 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:31:02 smithi162 systemd[1]: Stopping Ceph mds.c for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:31:03.330 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:31:03 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-c[97982]: 2024-03-19T01:31:03.067+0000 7f7139143700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:31:03.330 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:31:03 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-c[97982]: 2024-03-19T01:31:03.067+0000 7f7139143700 -1 mds.c *** got signal Terminated *** 2024-03-19T01:31:06.149 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:31:05 smithi162 podman[104967]: 2024-03-19 01:31:05.876285811 +0000 UTC m=+2.905780644 container died 000c09a15f6a4b40c28871e894612ca1e673c2fdc4810ee6129f207f6242f87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-19T01:31:06.494 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:31:06 smithi162 podman[104967]: 2024-03-19 01:31:06.284237316 +0000 UTC m=+3.313732128 container cleanup 000c09a15f6a4b40c28871e894612ca1e673c2fdc4810ee6129f207f6242f87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-19T01:31:06.494 INFO:journalctl@ceph.mds.c.smithi162.stdout:Mar 19 01:31:06 smithi162 bash[104967]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-c 2024-03-19T01:31:06.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.c.service' 2024-03-19T01:31:06.665 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:31:06.665 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-19T01:31:06.665 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-19T01:31:06.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.f 2024-03-19T01:31:07.149 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:31:06 smithi162 systemd[1]: Stopping Ceph mds.f for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:31:07.149 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:31:06 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-f[97676]: 2024-03-19T01:31:06.979+0000 7f1afaae7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:31:07.149 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:31:06 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-f[97676]: 2024-03-19T01:31:06.979+0000 7f1afaae7700 -1 mds.f *** got signal Terminated *** 2024-03-19T01:31:10.149 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:31:09 smithi162 podman[105100]: 2024-03-19 01:31:09.881481484 +0000 UTC m=+3.000709135 container died 53eec699f51fb00402eaaa0aa27a19be4f6666a3d7052c249263c07ca7e6dbea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-f, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T01:31:10.488 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:31:10 smithi162 podman[105100]: 2024-03-19 01:31:10.237850204 +0000 UTC m=+3.357077865 container cleanup 53eec699f51fb00402eaaa0aa27a19be4f6666a3d7052c249263c07ca7e6dbea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-f, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T01:31:10.489 INFO:journalctl@ceph.mds.f.smithi162.stdout:Mar 19 01:31:10 smithi162 bash[105100]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-f 2024-03-19T01:31:10.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.f.service' 2024-03-19T01:31:10.692 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:31:10.692 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-19T01:31:10.692 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-19T01:31:10.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.i 2024-03-19T01:31:11.149 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:31:10 smithi162 systemd[1]: Stopping Ceph mds.i for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:31:11.149 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:31:10 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-i[98298]: 2024-03-19T01:31:10.990+0000 7fb1ffc15700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:31:11.149 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:31:10 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-i[98298]: 2024-03-19T01:31:10.990+0000 7fb1ffc15700 -1 mds.i *** got signal Terminated *** 2024-03-19T01:31:14.125 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:31:13 smithi162 podman[105221]: 2024-03-19 01:31:13.877752473 +0000 UTC m=+2.976500835 container died d91e846e05c30fecd815f527e477acba3aa2037e5569e7d38879bd0437729db6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-i, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-19T01:31:14.399 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:31:14 smithi162 podman[105221]: 2024-03-19 01:31:14.223956576 +0000 UTC m=+3.322704937 container cleanup d91e846e05c30fecd815f527e477acba3aa2037e5569e7d38879bd0437729db6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-i, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240318.181317-b6ae249) 2024-03-19T01:31:14.399 INFO:journalctl@ceph.mds.i.smithi162.stdout:Mar 19 01:31:14 smithi162 bash[105221]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-i 2024-03-19T01:31:14.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.i.service' 2024-03-19T01:31:14.689 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:31:14.689 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-19T01:31:14.689 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-19T01:31:14.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.l 2024-03-19T01:31:15.011 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:31:14 smithi162 systemd[1]: Stopping Ceph mds.l for 35912c14-e58c-11ee-95c9-87774f69a715... 2024-03-19T01:31:15.399 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:31:15 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-l[97350]: 2024-03-19T01:31:15.010+0000 7fd92440f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-19T01:31:15.399 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:31:15 smithi162 ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-l[97350]: 2024-03-19T01:31:15.010+0000 7fd92440f700 -1 mds.l *** got signal Terminated *** 2024-03-19T01:31:18.149 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:31:17 smithi162 podman[105342]: 2024-03-19 01:31:17.879599421 +0000 UTC m=+2.969651333 container died e7c84b2d954c7ed2a87087912d7eff4eafb7311a415d5c5d8f46303dd34ed6ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-l, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-19T01:31:18.649 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:31:18 smithi162 podman[105342]: 2024-03-19 01:31:18.319201245 +0000 UTC m=+3.409253139 container cleanup e7c84b2d954c7ed2a87087912d7eff4eafb7311a415d5c5d8f46303dd34ed6ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-l, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-03-19T01:31:18.649 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:31:18 smithi162 bash[105342]: ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-l 2024-03-19T01:31:18.649 INFO:journalctl@ceph.mds.l.smithi162.stdout:Mar 19 01:31:18 smithi162 podman[105356]: 2024-03-19 01:31:18.470140689 +0000 UTC m=+0.583981693 container remove e7c84b2d954c7ed2a87087912d7eff4eafb7311a415d5c5d8f46303dd34ed6ad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c851f2a8983bc273fa71dd1a3e84be4b397f9c2099e180242b453d87e7c2b639, name=ceph-35912c14-e58c-11ee-95c9-87774f69a715-mds-l, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240318.181317-b6ae249, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-19T01:31:18.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-35912c14-e58c-11ee-95c9-87774f69a715@mds.l.service' 2024-03-19T01:31:18.717 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-19T01:31:18.717 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-19T01:31:18.718 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 35912c14-e58c-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-19T01:31:18.913 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:31:21.322 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 35912c14-e58c-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-19T01:31:21.514 INFO:teuthology.orchestra.run.smithi156.stdout:Deleting cluster with fsid: 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:31:24.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 35912c14-e58c-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-19T01:31:24.305 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:31:26.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T01:31:26.764 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T01:31:26.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-19T01:31:26.828 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-19T01:31:26.831 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250/remote/smithi116/crash 2024-03-19T01:31:26.832 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/crash -- . 2024-03-19T01:31:26.880 INFO:teuthology.orchestra.run.smithi116.stderr:tar: /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-19T01:31:26.881 INFO:teuthology.orchestra.run.smithi116.stderr:tar: Error is not recoverable: exiting now 2024-03-19T01:31:26.882 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250/remote/smithi156/crash 2024-03-19T01:31:26.883 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/crash -- . 2024-03-19T01:31:26.926 INFO:teuthology.orchestra.run.smithi156.stderr:tar: /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-19T01:31:26.926 INFO:teuthology.orchestra.run.smithi156.stderr:tar: Error is not recoverable: exiting now 2024-03-19T01:31:26.928 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250/remote/smithi162/crash 2024-03-19T01:31:26.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/crash -- . 2024-03-19T01:31:26.967 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /var/lib/ceph/35912c14-e58c-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-19T01:31:26.967 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now 2024-03-19T01:31:26.968 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-19T01:31:26.969 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-19T01:31:27.004 INFO:teuthology.orchestra.run.smithi116.stdout:2024-03-19T01:04:26.493904+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T01:31:27.004 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-19T01:31:27.005 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[SEC\]' /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-19T01:31:27.079 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]' /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-19T01:31:27.152 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[WRN\]' /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-19T01:31:27.224 INFO:teuthology.orchestra.run.smithi116.stdout:2024-03-19T01:04:26.493904+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-19T01:31:27.224 INFO:tasks.cephadm:Compressing logs... 2024-03-19T01:31:27.224 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T01:31:27.267 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T01:31:27.269 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-19T01:31:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-19T01:31:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:: No such file or directory 2024-03-19T01:31:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-19T01:31:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-19T01:31:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-19T01:31:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.6.log: /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-19T01:31:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-19T01:31:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.audit.log: /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-19T01:31:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-19T01:31:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-19T01:31:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mgr.x.log: 90.5% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-19T01:31:27.296 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-19T01:31:27.296 INFO:teuthology.orchestra.run.smithi162.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log: No such file or directory 2024-03-19T01:31:27.297 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-19T01:31:27.297 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-19T01:31:27.297 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-19T01:31:27.306 INFO:teuthology.orchestra.run.smithi156.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-19T01:31:27.306 INFO:teuthology.orchestra.run.smithi156.stderr:: No such file or directory 2024-03-19T01:31:27.306 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-19T01:31:27.306 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-19T01:31:27.306 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-19T01:31:27.306 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-19T01:31:27.306 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-19T01:31:27.306 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.b.log: /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.4.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-19T01:31:27.306 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-19T01:31:27.307 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-19T01:31:27.307 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-19T01:31:27.307 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-19T01:31:27.307 INFO:teuthology.orchestra.run.smithi156.stderr: 90.6% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.audit.log.gz/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.7.log: 2024-03-19T01:31:27.307 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-19T01:31:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.0.log: 94.6% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-19T01:31:27.311 INFO:teuthology.orchestra.run.smithi116.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-19T01:31:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.log 2024-03-19T01:31:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-19T01:31:27.311 INFO:teuthology.orchestra.run.smithi116.stderr: 82.7% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-19T01:31:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.log: 86.9% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-19T01:31:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-19T01:31:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-19T01:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-19T01:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.l.log: /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-19T01:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-19T01:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.i.log: /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-19T01:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mgr.z.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-19T01:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-19T01:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-19T01:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-19T01:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.f.log: 90.7% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-19T01:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.log 2024-03-19T01:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.cephadm.log: 82.4% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-19T01:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-19T01:31:27.313 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.log: 87.0% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-19T01:31:27.314 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-19T01:31:27.314 INFO:teuthology.orchestra.run.smithi162.stderr: 93.1% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.2.log: 2024-03-19T01:31:27.314 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-19T01:31:27.314 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-19T01:31:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.9.log: 91.0% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-19T01:31:27.319 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-19T01:31:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-19T01:31:27.322 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mgr.y.log: 93.0% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-19T01:31:27.323 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.log 2024-03-19T01:31:27.324 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.cephadm.log: 82.4% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-19T01:31:27.328 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-19T01:31:27.337 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-19T01:31:27.340 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.log: 86.9% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-19T01:31:27.345 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-19T01:31:27.350 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-19T01:31:27.377 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-19T01:31:27.418 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.a.log: 93.5% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-19T01:31:27.569 INFO:teuthology.orchestra.run.smithi162.stderr: 94.4% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-19T01:31:27.594 INFO:teuthology.orchestra.run.smithi116.stderr: 90.2% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-19T01:31:27.605 INFO:teuthology.orchestra.run.smithi156.stderr: 93.6% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-19T01:31:27.608 INFO:teuthology.orchestra.run.smithi162.stderr: 93.5% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-19T01:31:27.663 INFO:teuthology.orchestra.run.smithi156.stderr: 94.3% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-19T01:31:27.771 INFO:teuthology.orchestra.run.smithi162.stderr: 92.9% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-19T01:31:27.905 INFO:teuthology.orchestra.run.smithi156.stderr: 91.9% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-19T01:31:27.938 INFO:teuthology.orchestra.run.smithi162.stderr: 91.8% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-19T01:31:28.323 INFO:teuthology.orchestra.run.smithi116.stderr: 90.3% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-19T01:31:28.466 INFO:teuthology.orchestra.run.smithi162.stderr: 93.4% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-19T01:31:28.878 INFO:teuthology.orchestra.run.smithi156.stderr: 93.7% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-19T01:31:29.378 INFO:teuthology.orchestra.run.smithi116.stderr: 93.8% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-19T01:31:29.918 INFO:teuthology.orchestra.run.smithi156.stderr: 93.9% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-19T01:31:58.915 INFO:teuthology.orchestra.run.smithi116.stderr: 91.2% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-19T01:32:03.797 INFO:teuthology.orchestra.run.smithi162.stderr: 91.1% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-19T01:32:06.855 INFO:teuthology.orchestra.run.smithi156.stderr: 91.5% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-19T01:32:12.056 INFO:teuthology.orchestra.run.smithi162.stderr: 91.0% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-19T01:32:12.648 INFO:teuthology.orchestra.run.smithi156.stderr: 91.1% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-19T01:32:12.755 INFO:teuthology.orchestra.run.smithi116.stderr: 91.0% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-19T01:32:13.001 INFO:teuthology.orchestra.run.smithi162.stderr: 91.1% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-19T01:32:14.703 INFO:teuthology.orchestra.run.smithi162.stderr: 90.8% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-19T01:32:14.705 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-19T01:32:14.705 INFO:teuthology.orchestra.run.smithi162.stderr:real 0m47.421s 2024-03-19T01:32:14.705 INFO:teuthology.orchestra.run.smithi162.stderr:user 2m51.200s 2024-03-19T01:32:14.705 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m5.370s 2024-03-19T01:32:21.065 INFO:teuthology.orchestra.run.smithi116.stderr: 90.8% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-19T01:32:23.418 INFO:teuthology.orchestra.run.smithi156.stderr: 91.1% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-19T01:32:24.494 INFO:teuthology.orchestra.run.smithi156.stderr: 91.0% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-19T01:32:24.496 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-03-19T01:32:24.496 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m57.212s 2024-03-19T01:32:24.496 INFO:teuthology.orchestra.run.smithi156.stderr:user 3m16.669s 2024-03-19T01:32:24.496 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m5.943s 2024-03-19T01:32:33.736 INFO:teuthology.orchestra.run.smithi116.stderr: 90.7% -- replaced with /var/log/ceph/35912c14-e58c-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-19T01:32:33.737 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-03-19T01:32:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:real 1m6.459s 2024-03-19T01:32:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:user 3m14.305s 2024-03-19T01:32:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m6.002s 2024-03-19T01:32:33.738 INFO:tasks.cephadm:Archiving logs... 2024-03-19T01:32:33.738 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250/remote/smithi116/log 2024-03-19T01:32:33.740 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T01:33:09.825 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250/remote/smithi156/log 2024-03-19T01:33:09.826 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T01:33:44.377 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250/remote/smithi162/log 2024-03-19T01:33:44.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-19T01:34:14.876 INFO:tasks.cephadm:Removing cluster... 2024-03-19T01:34:14.877 DEBUG:teuthology.orchestra.run.smithi116:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 35912c14-e58c-11ee-95c9-87774f69a715 --force 2024-03-19T01:34:15.070 INFO:teuthology.orchestra.run.smithi116.stdout:Deleting cluster with fsid: 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:34:15.651 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 35912c14-e58c-11ee-95c9-87774f69a715 --force 2024-03-19T01:34:15.842 INFO:teuthology.orchestra.run.smithi156.stdout:Deleting cluster with fsid: 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:34:16.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 35912c14-e58c-11ee-95c9-87774f69a715 --force 2024-03-19T01:34:16.586 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: 35912c14-e58c-11ee-95c9-87774f69a715 2024-03-19T01:34:17.211 INFO:tasks.cephadm:Removing cephadm ... 2024-03-19T01:34:17.212 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-19T01:34:17.230 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-19T01:34:17.246 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-19T01:34:17.262 INFO:tasks.cephadm:Teardown complete 2024-03-19T01:34:17.262 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-19T01:34:17.274 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-19T01:34:17.275 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T01:34:17.277 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T01:34:17.288 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-19T01:34:17.398 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-19T01:34:17.408 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-19T01:34:17.408 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T01:34:17.411 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T01:34:17.414 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-19T01:34:17.426 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-03-19T01:34:17.426 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-03-19T01:34:17.427 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-03-19T01:34:17.430 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T01:34:17.430 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-03-19T01:34:17.431 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 13 +1610us[+1623us] +/- 177ms 2024-03-19T01:34:17.431 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -1052us[-1039us] +/- 78ms 2024-03-19T01:34:17.431 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +328us[ +328us] +/- 69ms 2024-03-19T01:34:17.431 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T01:34:17.433 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T01:34:17.433 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-03-19T01:34:17.433 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +1617us[+1617us] +/- 177ms 2024-03-19T01:34:17.433 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 -1045us[-1038us] +/- 78ms 2024-03-19T01:34:17.433 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 +332us[ +332us] +/- 69ms 2024-03-19T01:34:17.433 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T01:34:17.433 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-19T01:34:17.434 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-03-19T01:34:17.434 INFO:teuthology.orchestra.run.smithi116.stdout:^- hv01.front.sepia.ceph.com 2 6 377 11 +1931us[+1937us] +/- 177ms 2024-03-19T01:34:17.434 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 -727us[ -721us] +/- 78ms 2024-03-19T01:34:17.434 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 +635us[ +641us] +/- 69ms 2024-03-19T01:34:17.434 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-19T01:34:17.434 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-19T01:34:17.444 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-19T01:34:17.445 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-19T01:34:17.455 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T01:34:17.558 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T01:34:17.633 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-19T01:34:17.703 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T01:34:17.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:34:17.735 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T01:34:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:34:17.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-19T01:34:17.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:34:17.797 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-19T01:34:17.808 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-19T01:34:17.819 INFO:teuthology.task.internal:Duration was 4054.807583 seconds 2024-03-19T01:34:17.820 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-19T01:34:17.830 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-19T01:34:17.830 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T01:34:17.833 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T01:34:17.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-19T01:34:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T01:34:17.876 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T01:34:17.880 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-19T01:34:18.254 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-19T01:34:18.254 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-03-19T01:34:18.254 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T01:34:18.275 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2024-03-19T01:34:18.275 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T01:34:18.316 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-03-19T01:34:18.316 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-19T01:34:18.355 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-19T01:34:18.356 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T01:34:18.358 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T01:34:18.360 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-19T01:34:18.424 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-19T01:34:18.424 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T01:34:18.428 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T01:34:18.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-19T01:34:18.823 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-19T01:34:18.835 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-19T01:34:18.835 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T01:34:18.862 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T01:34:18.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-19T01:34:18.917 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-19T01:34:18.929 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T01:34:18.932 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T01:34:18.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-19T01:34:18.955 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-03-19T01:34:18.957 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2024-03-19T01:34:18.983 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-03-19T01:34:19.019 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T01:34:19.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:34:19.033 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T01:34:19.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:34:19.047 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-19T01:34:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-19T01:34:19.074 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-19T01:34:19.090 INFO:teuthology.task.internal:Transferring archived files... 2024-03-19T01:34:19.092 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250/remote/smithi116 2024-03-19T01:34:19.092 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T01:34:19.138 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250/remote/smithi156 2024-03-19T01:34:19.139 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T01:34:19.178 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250/remote/smithi162 2024-03-19T01:34:19.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-19T01:34:19.219 INFO:teuthology.task.internal:Removing archive directory... 2024-03-19T01:34:19.219 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T01:34:19.222 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T01:34:19.224 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-19T01:34:19.269 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-19T01:34:19.282 INFO:teuthology.task.internal:Not uploading archives. 2024-03-19T01:34:19.282 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-19T01:34:19.293 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-19T01:34:19.294 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T01:34:19.296 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T01:34:19.299 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-19T01:34:19.308 INFO:teuthology.orchestra.run.smithi116.stdout: 265963 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 01:34 /home/ubuntu/cephtest 2024-03-19T01:34:19.310 INFO:teuthology.orchestra.run.smithi156.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 01:34 /home/ubuntu/cephtest 2024-03-19T01:34:19.323 INFO:teuthology.orchestra.run.smithi162.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 19 01:34 /home/ubuntu/cephtest 2024-03-19T01:34:19.325 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-19T01:34:19.336 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-19T01:34:19.360 INFO:teuthology.nuke:Checking targets against current locks 2024-03-19T01:34:19.382 DEBUG:teuthology.nuke:shortname: smithi116 2024-03-19T01:34:19.382 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T01:34:19.405 DEBUG:teuthology.nuke:shortname: smithi156 2024-03-19T01:34:19.406 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T01:34:19.411 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 00:04:07.975846', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAd63bO977/b/mawCBs3CO6F2T2yWIBGBSz1MOF1jQiaRieZ1ZasJBXxWv49Hi0BC0cfbdv6zD/LV0Y0MCslZww='} 2024-03-19T01:34:19.429 DEBUG:teuthology.nuke:shortname: smithi162 2024-03-19T01:34:19.429 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-19T01:34:19.434 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 00:04:07.974081', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPU31mKDEia961VDaU8NKccsLQDlbNPuZu9H1aQo9Dr3JfCnKtUN+xBFuAvcLRcPWb4uUi+cjrNh/KoHUMbPtMU='} 2024-03-19T01:34:19.455 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609250', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-19 00:04:07.977102', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPjH2AFSIXZ2/ePqN38IWe8cxvPIf8Oz0/QA2tYVE5PDQiHKBamC85lViL2mrgcuXupFD2IlfNjE3UiheQ/HMRE='} 2024-03-19T01:34:19.470 INFO:teuthology.orchestra.console.smithi116:Power off 2024-03-19T01:34:19.471 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T01:34:19.498 DEBUG:teuthology.orchestra.console.smithi116:power off output: Chassis Power Control: Down/Off 2024-03-19T01:34:19.498 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:34:19.517 INFO:teuthology.orchestra.console.smithi156:Power off 2024-03-19T01:34:19.518 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T01:34:19.528 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-19T01:34:19.540 DEBUG:teuthology.orchestra.console.smithi156:power off output: Chassis Power Control: Down/Off 2024-03-19T01:34:19.540 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:34:19.556 INFO:teuthology.orchestra.console.smithi162:Power off 2024-03-19T01:34:19.556 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-19T01:34:19.566 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-03-19T01:34:19.588 DEBUG:teuthology.orchestra.console.smithi162:power off output: Chassis Power Control: Down/Off 2024-03-19T01:34:19.588 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:34:19.610 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-03-19T01:34:23.529 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:34:23.568 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:34:23.611 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:34:23.642 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-19T01:34:23.678 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-03-19T01:34:23.722 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-03-19T01:34:27.646 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:34:27.678 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:34:27.723 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:34:27.757 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is on 2024-03-19T01:34:27.789 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is on 2024-03-19T01:34:27.832 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2024-03-19T01:34:27.933 INFO:teuthology.orchestra.console.smithi162:Power off completed 2024-03-19T01:34:31.762 DEBUG:teuthology.orchestra.console.smithi116:pexpect command: ipmitool -H smithi116.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:34:31.789 DEBUG:teuthology.orchestra.console.smithi156:pexpect command: ipmitool -H smithi156.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-19T01:34:31.872 DEBUG:teuthology.orchestra.console.smithi116:check power output: Chassis Power is off 2024-03-19T01:34:31.898 DEBUG:teuthology.orchestra.console.smithi156:check power output: Chassis Power is off 2024-03-19T01:34:31.974 INFO:teuthology.orchestra.console.smithi116:Power off completed 2024-03-19T01:34:31.999 INFO:teuthology.orchestra.console.smithi156:Power off completed 2024-03-19T01:34:32.105 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} duration: 4054.807583093643 failure_reason: '"2024-03-19T01:04:26.493904+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-19T01:34:32.106 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-19T01:34:32.201 INFO:teuthology.run:FAIL